A two-dimensional code image recognition method, device and storage medium
Through the image positioning model trained by the multi-task cascade convolutional neural network model, the QR code area is determined and corrected, which solves the problem of low QR code recognition rate in complex backgrounds, and achieves higher recognition accuracy and efficiency.
Patent Information
- Application Number
- CN202010807076.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-08-12
- Publication Date
- 2024-11-29
- Estimated Expiration
- 2040-08-12
AI Technical Summary
In complex background, the recognition rate of QR codes is low and is greatly affected by lighting, shooting angle, distance and environment, making it difficult to accurately locate and identify the QR code area.
The image positioning model trained by the multi-task cascade convolutional neural network model is used to obtain the target image, determine the QR code area to be recognized, and correct it based on the position information of the position detection graph to improve the recognition rate of the QR code image.
It improves the recognition rate and positioning accuracy of QR code images, and can accurately locate and identify QR codes in complex environments, enhancing the applicability of the identification system.
Smart Images

Figure CN111950318B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of image recognition technology, and in particular to a method, device and storage medium for recognizing a two-dimensional code image. Background Art
[0002] In daily life, QR code has become one of the most convenient means of information dissemination because of its advantages such as large information capacity, strong fault tolerance and low cost. It is widely used in various fields such as e-commerce and product traceability.
[0003] A QR code is a black-and-white graphic that is distributed in a plane according to a certain pattern to record data symbol information. In practical applications, some actual scenarios, such as lighting, shooting angle, distance, environment, etc., will affect the final imaging effect of the QR code area, and thus will also affect the recognition of the QR code.
[0004] Therefore, how to efficiently and accurately locate the QR code area under complex backgrounds is particularly important to improve the recognition rate of the QR code. Summary of the invention
[0005] The embodiments of this specification provide a method, device and storage medium for recognizing a two-dimensional code image, which are used to improve the recognition rate of the two-dimensional code image.
[0006] To solve the above technical problems, the embodiments of this specification are implemented as follows:
[0007] The embodiment of this specification provides a method for recognizing a two-dimensional code image, including:
[0008] Get the target image;
[0009] Using the image positioning model, determining the to-be-recognized two-dimensional code region contained in the target image; the to-be-recognized two-dimensional code region contains the to-be-recognized two-dimensional code image; the image positioning model is obtained by training a multi-task cascade convolutional neural network model based on two-dimensional code image training samples;
[0010] Based on the image information of the to-be-identified two-dimensional code area, determining the position information of the position detection pattern in the to-be-identified two-dimensional code image;
[0011] Determine the correction parameters according to the position information and standard positioning information; the standard positioning information is the position information of the position detection pattern in the preset two-dimensional code image template;
[0012] Correcting the to-be-recognized two-dimensional code image according to the correction parameters to obtain a corrected two-dimensional code image;
[0013] The corrected two-dimensional code image is recognized.
[0014] The embodiment of this specification provides a two-dimensional code image recognition device, including:
[0015] one or more processors;
[0016] a storage device for storing one or more programs,
[0017] When the one or more programs are executed by the one or more processors, the one or more processors implement the above-mentioned two-dimensional code image recognition method.
[0018] An embodiment of the present specification provides a computer-readable storage medium on which a computer program is stored, characterized in that when the program is executed by a processor, the above-mentioned two-dimensional code image recognition method is implemented.
[0019] An embodiment of the present specification realizes the following beneficial effects: determining the area of the QR code to be identified in the target image through an image positioning model, and correcting the QR code image to be identified in the target image according to the position information of the position detection pattern in the area of the QR code to be identified, and then identifying the corrected QR code image, thereby improving the recognition rate of the QR code image.
[0020] In addition, the image localization model obtained by training the multi-task cascade convolutional neural network model according to the QR code image training samples is used to locate the QR code area to be identified in the target image, which can improve the positioning accuracy and speed. BRIEF DESCRIPTION OF THE DRAWINGS
[0021] In order to more clearly illustrate the embodiments of this specification or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.
[0022] Figure 1 A schematic diagram of the structure of a two-dimensional code template provided in an embodiment of this specification;
[0023] Figure 2 A schematic diagram of a two-dimensional code image recognition method provided in an embodiment of this specification in an actual application scenario;
[0024] Figure 3 A schematic diagram of a flow chart of a method for recognizing a two-dimensional code image provided in an embodiment of this specification;
[0025] Figure 4 A schematic diagram of a flow chart of a method for recognizing a two-dimensional code image provided in an embodiment of this specification;
[0026] Figure 5 A flowchart of a method for training a two-dimensional code image positioning model provided in an embodiment of this specification;
[0027] Figure 6 A schematic diagram of a two-dimensional code image annotation information provided in an embodiment of this specification;
[0028] Figure 7 The embodiments of this specification provide corresponding to Figure 3 A structural schematic diagram of a two-dimensional code image recognition device;
[0029] Figure 8 The embodiments of this specification provide corresponding to Figure 5 A structural schematic diagram of a training device for a two-dimensional code image positioning model. DETAILED DESCRIPTION
[0030] In order to make the purpose, technical solutions and advantages of one or more embodiments of this specification clearer, the technical solutions of one or more embodiments of this specification will be clearly and completely described below in combination with the specific embodiments of this specification and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on the embodiments in this description, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of one or more embodiments of this specification.
[0031] The technical solutions provided by the embodiments of this specification are described in detail below in conjunction with the accompanying drawings.
[0032] In practical applications, QR codes are usually generated based on preset QR code templates. Figure 1 A schematic diagram of the structure of a two-dimensional code template provided in the embodiment of this specification, such as Figure 1 As shown, the two-dimensional code template may generally include: a position detection pattern 101, a correction pattern 102, an auxiliary positioning pattern 103, format information 104, data and error correction code words 105, and all or part of the version information 106.
[0033] Among them, the position detection pattern 101 is used for positioning the two-dimensional code. For each two-dimensional code, the position is fixed, but the size may vary. The black and white rectangular blocks in the position detection pattern 101 can be used for image processing detection. In the two-dimensional code template, a position detection pattern separator can also be set near the position detection pattern to separate the position detection pattern from other parts of the two-dimensional code.
[0034] Correction pattern 102: mainly used for correcting the shape of the QR code, especially when the QR code is printed on an uneven surface, or is distorted when photographed; in actual applications, the number of correction patterns may vary depending on the size of the QR code.
[0035] Auxiliary positioning pattern 103: It is usually located between two position detection patterns and is composed of black and white grids. These small black and white grids are similar to coordinate axes. They define a grid on the QR code and can assist in the positioning of the QR code.
[0036] Format information 104: indicates the error correction level of the two-dimensional code, which can be divided into L, M, Q, and H.
[0037] Data and error correction codeword 105: Use black and white binary grids to encode content. Usually, 8 grids can encode one byte. The error correction codeword is used to correct errors caused by damage to the QR code.
[0038] Version information 106: that is, the specification of the QR code. Usually, each version symbol has 4 more modules on each side than the previous version.
[0039] In practical applications, the QR code template can also be set according to actual needs. For example, the QR code template can be set to various shapes such as circle, triangle, etc., and the content it contains can also be set according to actual needs. This specification does not make specific limitations on this in the embodiments.
[0040] In the prior art, when a user uses a terminal to identify a QR code image, in order to ensure that the QR code in the image can be identified, the terminal usually needs to be brought close to or aimed at the QR code image. In addition, when the environment in which the QR code image is located is relatively complex, for example, when the QR code image is not directly in front of the terminal, the QR code may not be recognized due to angle problems; for example, when the QR code image is far away from the terminal, the image presented in the terminal is not sufficient for the terminal to recognize the QR code in the image, etc., all of which may cause the QR code to be unrecognizable. In order to solve the defects in the prior art, this solution provides the following embodiments:
[0041] Figure 2 The following is a schematic diagram of a two-dimensional code image recognition method provided in an embodiment of this specification in an actual application scenario. The two-dimensional code image recognition method provided in an embodiment of this specification can be applied not only to a scenario where the target image contains one two-dimensional code image, but also to a scenario where the target image contains multiple two-dimensional code images.
[0042] like Figure 2As shown, it is assumed that a number of items 2 are stored in a shelf 1, each item 2 is provided with a QR code image 3 representing the item information, and a camera 4 collects images of the items in the shelf 1, and counts the items in the shelf by identifying the QR code image in the image.
[0043] Normally, the captured object image 5 may include multiple QR code images, and due to the different relative positions of the object 2 and the camera 4, the styles of the multiple QR code images presented in the object image 5 will also be different. For example, the QR code image farther away from the camera is displayed smaller in the object image, and due to the shooting angle, the QR code displayed in the object image will have a certain degree of deformation.
[0044] In the embodiment of the present specification, an image positioning model 6 is used to determine multiple two-dimensional code areas 7 contained in the object image, and each two-dimensional code area 7 is corrected to obtain a corrected two-dimensional code image 8. The corrected two-dimensional code image 8 is then recognized to obtain a recognition result 9, which can effectively recognize the two-dimensional code image in the image and improve the recognition rate of the two-dimensional code.
[0045] It should be noted that the above content is only an exemplary description of the scenarios in which the methods in the embodiments of this specification can be applied. In actual applications, the methods in the embodiments of this specification can be applied to a variety of scenarios, such as scanning code payment, identifying image information received by the terminal, etc. The specific application scenarios are not limited in the embodiments of this specification.
[0046] Next, a two-dimensional code image recognition method provided in the embodiment of the specification is specifically described in conjunction with the accompanying drawings:
[0047] Figure 3 The following is a flow chart of a method for recognizing a two-dimensional code image provided by an embodiment of the present specification. From a program perspective, the execution subject of the process can be a program installed on an application server or an application client.
[0048] like Figure 3 As shown, the process may include the following steps:
[0049] Step 302: Acquire a target image.
[0050] In practical applications, the execution subject for identifying the QR code can obtain the target image, and the execution subject can be various electronic devices that support QR code recognition, including but not limited to smart phones, tablet computers, computers, etc. The target image can be an image collected by the execution subject, or an image locally saved by the execution subject. The method for obtaining the target image is not limited in the embodiments of this specification.
[0051] In order to reduce the influence of interference information on the two-dimensional code image recognition, the acquired target image may also be preprocessed, for example, grayscale adjustment, denoising and the like may be performed on the target image.
[0052] Step 304: using the image localization model, determine the area of the QR code to be identified contained in the target image; the area of the QR code to be identified contains the QR code image to be identified; the image localization model is obtained by training a multi-task cascade convolutional neural network model based on the QR code image training samples.
[0053] In the embodiments of the present specification, a pre-trained image positioning model can be used to locate the QR code area of the acquired target image, and the QR code area to be identified can be determined from the target image.
[0054] Among them, the multi-task cascade convolutional neural network model can be composed of multiple convolutional neural network layers. Through the collaborative work of multiple network layers, the fast and accurate positioning of the QR code area in the target image can be achieved, thereby ensuring the efficiency and accuracy of positioning.
[0055] In practical applications, the multi-task cascade convolutional neural network model may include a Pnet (Proposal Network) layer, an Rnet (Refine Network) layer, and an Onet (Output Network) layer; the Pnet layer is used to obtain as many QR code areas in the image to be identified as possible; the Rnet layer is used to screen and / or adjust the QR code areas obtained by the Pnet layer to improve the accuracy of QR code area positioning; Onet is used to further optimize the output result of the Rnet layer and mark the QR code area to be identified in the image to be identified. Specifically, the QR code area to be identified can be selected in the form of a positioning box.
[0056] Step 306: Based on the image information of the to-be-identified two-dimensional code region, determine the position information of the position detection pattern in the to-be-identified two-dimensional code image.
[0057] In the embodiments of this specification, the to-be-recognized two-dimensional code region determined by the image positioning model is generally a region of the target image that contains the to-be-recognized two-dimensional code image, wherein the two-dimensional code image is generally generated based on a two-dimensional code image template, such as Figure 1 As shown, the QR code image template may include a position detection pattern for positioning the QR code, and the position detection pattern may be located at the upper left, upper right, and lower left of the template, respectively. In actual applications, the specific position, quantity, and set shape of the position detection pattern can be set according to actual needs, and are not specifically limited in the embodiments of this specification.
[0058] Step 308: Determine correction parameters according to the position information and standard positioning information; the standard positioning information is the position information of the position detection pattern in the preset two-dimensional code image template.
[0059] The preset QR code template can be a template for generating a QR code image in a target image. In actual applications, the size, shape, style and other features of the QR code image template are usually within a range of high recognition rate. It can be understood that a QR code image generated using a QR code image template can usually be accurately recognized without any deformation.
[0060] In the embodiment of the present specification, the position detection pattern in the template will also exist in the two-dimensional code image in the target image, and the relative position of the position detection pattern in the two-dimensional code image is unchanged. Therefore, the mapping relationship between the two-dimensional code image in the target image and the preset two-dimensional code image template can be determined based on the position information of the position detection pattern in the two-dimensional code in the target image and the position information of the position detection pattern in the preset two-dimensional code image template. Through this mapping relationship, the two-dimensional code image in the target image can have the same size, shape, style and other features as the preset two-dimensional code image template. In the embodiment of the present specification, the mapping relationship can be represented by a correction parameter.
[0061] Step 310: Correct the two-dimensional code image to be identified according to the correction parameters to obtain a corrected two-dimensional code image.
[0062] In the embodiments of this specification, the correction parameters can be used to correct the QR code image to be identified in the target image, so that the QR code image to be identified in the target image can be restored to the size, shape, style, etc. of the preset QR code image template as much as possible to improve the recognition rate of the QR code.
[0063] Step 312: Recognize the corrected two-dimensional code image.
[0064] It should be understood that the order of some steps in the methods described in one or more embodiments of this specification can be interchanged according to actual needs, or some steps can be omitted or deleted.
[0065] Figure 3 The method in the invention determines the to-be-recognized two-dimensional code region in the target image through the image positioning model, and corrects the to-be-recognized two-dimensional code image in the target image according to the position information of the position detection pattern in the to-be-recognized two-dimensional code region, and then recognizes the corrected two-dimensional code image, thereby improving the recognition rate of the two-dimensional code image.
[0066] In addition, the image localization model obtained by training the multi-task cascade convolutional neural network model according to the QR code image training samples is used to locate the QR code area to be identified in the target image, which can improve the positioning accuracy and speed.
[0067] based on Figure 3 The method, the examples of this specification also provide some specific implementation plans of the method, which are described below.
[0068] Optionally, in step 304, determining the to-be-recognized two-dimensional code region contained in the target image by using the image positioning model may specifically include:
[0069] Inputting the target image into the image positioning model to obtain at least one candidate QR code region;
[0070] Determining the confidence level of each candidate QR code region;
[0071] The candidate two-dimensional code region with the greatest confidence is determined as the two-dimensional code region to be identified.
[0072] In the process of determining the QR code region to be identified by using the image positioning model in the embodiments of this specification, the image positioning model can locate at least one candidate QR code region for each QR code region of the target image, and can also calculate the confidence of each candidate QR code region, and determine the candidate QR code region with the largest confidence as the QR code region in the target image. The confidence can be understood as the probability that the candidate QR code region is a QR code region that meets the requirements.
[0073] In practical applications, the target image is input into the image positioning model to obtain a target image marked with a positioning frame. The area in the positioning frame can be understood as the QR code area to be identified.
[0074] Considering that there may be multiple QR codes to be identified in the target image, there will also be multiple QR code areas to be identified. Determining the QR code area only by the confidence of the candidate QR code area may not accurately locate the QR code area in the target image. Therefore, in order to improve the accuracy of QR code area positioning, determining the candidate QR code area with the largest confidence as the QR code area may specifically include:
[0075] Determining an intersection-over-union ratio between the candidate two-dimensional code regions according to the confidence level;
[0076] Divide the candidate two-dimensional code regions whose intersection-over-union ratio is greater than a preset intersection-over-union ratio threshold into the same set to obtain multiple sets;
[0077] For each of the sets, the candidate two-dimensional code region with the largest confidence is determined as the two-dimensional code region to be identified.
[0078] In the embodiments of this specification, the intersection-and-union ratio can be understood as the degree of overlap of two candidate two-dimensional code regions, and the specific intersection-and-union ratio can be equal to the quotient of the intersection area of the two candidate two-dimensional code regions and the union area of the two candidate two-dimensional code regions. In this embodiment, the larger the intersection-and-union ratio of the two candidate two-dimensional code regions, the higher the degree of overlap of the two candidate two-dimensional code regions.
[0079] In the process of locating the target image, the image positioning model may mark multiple positioning frames for the same QR code area to be identified in order to mark all the QR code areas to be identified in the target image as much as possible, and each positioning frame represents a candidate QR code area. In practical applications, when multiple candidate QR code areas are gathered in one position, it can be said that there is a QR code area to be identified at the position. In order to improve the accuracy of positioning, some overlapping candidate QR code areas need to be deleted. Therefore, in the embodiments of this specification, the intersection-over-union ratio between candidate QR code areas can be used to delete candidate QR code areas with a high degree of overlap.
[0080] In order to determine the candidate two-dimensional code region corresponding to each two-dimensional code region in the target image, the candidate two-dimensional code regions can be grouped according to the intersection-and-union ratio, and the candidate two-dimensional code regions with an intersection-and-union ratio greater than a preset intersection-and-union ratio threshold value can be divided into the same set. Since the intersection-and-union ratio of the candidate two-dimensional code regions represents the overlap of the candidate two-dimensional code regions to a certain extent, when the overlap of two candidate two-dimensional code regions is high, it can be considered that the two candidate two-dimensional code regions correspond to the same two-dimensional code. Therefore, the candidate two-dimensional code regions with an intersection-and-union ratio greater than the preset intersection-and-union ratio threshold value can be divided into the same set, which can be understood as the candidate two-dimensional code regions in the same set corresponding to the same two-dimensional code in the target image.
[0081] After the grouping sets are determined, the candidate two-dimensional code region with the greatest confidence in each set can be determined as the two-dimensional code region to be identified in the target image, and then multiple two-dimensional code regions to be identified in the target image can be accurately marked.
[0082] In the embodiments of this specification, in order to ensure that the to-be-recognized two-dimensional code regions of different sizes in the target image can be located, the target image can also be scaled according to a preset rule based on the target image to obtain a series of target images, and the series of target images can be input into the image positioning model to locate the two-dimensional code region. Specifically, before the target image is input into the image positioning model, the target image can also be scaled according to a preset rule to obtain a plurality of detection images; and the detection images are input into the image positioning model.
[0083] The image positioning model in the embodiment of this specification can be obtained by training based on a multi-task cascade convolutional neural network model, which can include a convolution layer, a connection layer, and an output layer. As another implementation, step 304 of the embodiment of this specification uses the image positioning model to determine the area of the two-dimensional code to be identified contained in the target image, which can specifically include:
[0084] Scaling the target image according to a preset rule to obtain a plurality of detection images;
[0085] Inputting the detection image into a convolutional layer in the image positioning model to obtain a first number of first candidate QR code regions;
[0086] intercepting the target image according to the first candidate two-dimensional code region to obtain a first auxiliary image;
[0087] Inputting the first auxiliary image into a connection layer in the image positioning model to obtain a second number of second candidate two-dimensional code regions; the second number is smaller than the first number;
[0088] Intercepting the target image according to the second candidate two-dimensional code area to obtain a second auxiliary image;
[0089] The second auxiliary image is input into the output layer of the image positioning model to obtain the two-dimensional code area to be identified.
[0090] The step of inputting the detection image into a convolutional layer in the image positioning model to obtain a first number of first candidate two-dimensional code regions may specifically include:
[0091] Inputting the detection image into the convolution layer according to a first preset size to perform a convolution operation, and obtaining a plurality of first candidate QR code regions of the detection image whose confidence is greater than a preset confidence;
[0092] Non-maximum suppression processing is performed on the plurality of first candidate two-dimensional code regions to obtain the first number of first candidate two-dimensional code regions.
[0093] The step of intercepting the target image according to the first candidate two-dimensional code region to obtain the first auxiliary image may specifically include:
[0094] Determine the size correspondence between the detection image and the target image according to the preset rule and the convolution operation of the convolution layer;
[0095] Determine, according to the size correspondence, an image region corresponding to the first candidate two-dimensional code region in the target image;
[0096] The image area corresponding to the first candidate two-dimensional code area in the target image is intercepted to obtain the first auxiliary image.
[0097] In the embodiments of this specification, the convolution layer in the image positioning model can be used to mark out possible QR code areas in the target image as much as possible, and then the connection layer and the output layer can be used to select and / or adjust the candidate QR code areas, and finally an accurate positioning result of the QR code image can be obtained. As a specific embodiment, the convolution layer can also be understood as the aforementioned Pnet layer, the connection layer can be understood as the aforementioned Rnet layer, and the output layer can be understood as the aforementioned Onet layer. The image positioning model in the embodiments of this specification uses a multi-layer cascade method to process the target image, which can also improve the processing speed.
[0098] In practical applications, the image localization model can adopt a lightweight framework so that it can be applied to various terminals, such as mobile phones, tablets, barcode scanners, etc., and it is a comprehensive model that can complete multiple tasks such as classification and regression.
[0099] Considering the influence of factors such as the external environment in actual applications, the two-dimensional code to be identified in the target image may exist in various forms, such as deformation, bending, blur, etc. In order to enable the image positioning model to accurately locate the two-dimensional codes in various forms, the two-dimensional code image training samples described in the embodiments of this specification may include:
[0100] A sample image is obtained by capturing an image of an object containing a two-dimensional code image using preset parameters, wherein the preset parameters include at least one of a preset tilt angle, a preset light intensity, a preset exposure, a preset shutter time, and a preset aperture size.
[0101] The tilt angle can be understood as the angle between the plane where the captured image is located and the plane where the lens of the image acquisition device is located, or it can be understood as the angle between the central axis of the captured image and the central axis of the lens. The captured image is an image within the shooting range of the lens. For example, when a user uses a mobile terminal to scan and identify a code, when the lens of the mobile terminal is directly above the QR code image, the tilt angle at this time can be considered to be 0 degrees; assuming that the position of the code image is fixed, the terminal is moved to rotate around the QR code image. During the movement, there will be a certain angle between the lens and the QR code image. As the degree of rotation increases, the angle between the plane where the lens is located and the plane where the QR code image is located will also increase. It can be understood that the tilt angle becomes larger. When the central axis of the lens is perpendicular to the central axis of the QR code image, the tilt angle at this time can be considered to be 90 degrees.
[0102] In practical applications, a sample acquisition device can be used to acquire sample images at different tilt angles. The sample acquisition device may include a lens module, a lens bracket, a connecting rod, a sample bracket, and a sample containing a two-dimensional code image. The lens bracket is used to fix the lens module, the sample bracket is used to fix the sample, and the connecting rod is used to connect the lens bracket and the sample bracket. A rotating component with an angle scale may be provided at the connection between the connecting rod and the lens bracket and / or at the connection between the connecting rod and the sample bracket. The angle between the lens module and the sample may be changed by the rotating component, so that sample images at different tilt angles may be acquired.
[0103] In the embodiments of this specification, a training sample with a large tilt angle can be used to train a neural network model to obtain an image positioning model, thereby improving the scope of application of the image positioning model, so that the image positioning model can locate images that cannot be located by conventional image positioning tools, and can also improve the recognition rate of QR code images to a certain extent. For example, sample images with tilt angles ranging from a first preset angle to a second preset angle can be selected for collection, wherein the first preset angle can be 0 degrees and the second preset angle can be 80 degrees. It should be noted that the preset angle can be set according to actual needs and is not specifically limited in the embodiments of this specification.
[0104] Considering that in actual applications, images may be in environments with different brightness levels, in the embodiments of this specification, images under different light intensities may also be collected as sample images.
[0105] The light intensity may be the light intensity in the environment where the image is located, which may indicate the strength of the light, and may be measured by the energy of visible light received per unit area, which may be referred to as illumination, and the unit is lux (Lux or Lx). Generally, the light intensity may be measured using a illuminance meter. In summer, under direct sunlight, the light intensity may reach 60,000 to 100,000 lx, outdoors without the sun, it may be 10,000 to 10,000 lx, indoors in summer, it may be 100 to 550 lx, and at night under a full moon, it may be 0.2 lx. An incandescent lamp may emit approximately 12.561x of light per watt.
[0106] In the embodiments of the present specification, images within a preset light intensity range can be collected as sample images. For example, sample images are collected by placing the image in an environment with a preset light intensity range of 0.2 lx to 60,000 lx.
[0107] In the embodiments of this specification, images with different exposures can also be collected as training samples, where the exposure can be represented by an exposure value, which is related to the shutter time, aperture size, etc. of the lens. Generally, the lower the shutter speed, the larger the aperture, the larger the light transmission time and area, that is, the larger the exposure value. In practical applications, when the exposure is large or small, it will affect the clarity of the image and the positioning of the two-dimensional code image. In the embodiments of this specification, some images with large or small exposures can be used as training samples to improve the generalization ability of the image positioning model, reduce the impact of exposure on image positioning, and then the two-dimensional code area can be positioned for more images with different exposures.
[0108] In practical applications, sample collection can be completed by using one of the preset conditions of a preset tilt angle, a preset light intensity, a preset exposure, a preset shutter time, and a preset aperture size, or by using a combination of multiple preset conditions, for example, an image collected under the conditions of a tilt angle of a degree, a light intensity of b1x, and an exposure of c EV (Exposure Value) is used as a sample image. It should be noted that in practical applications, the preset parameters can be set according to actual needs, and the specific parameter settings are not limited in the embodiments of this specification, as long as they can meet the needs.
[0109] In order to obtain more abundant training samples, the two-dimensional code image training samples described in the embodiments of this specification may include:
[0110] At least one of an image obtained by collecting a two-dimensional code image on paper, an image obtained by collecting a two-dimensional code image displayed on a screen of an electronic product, and an image obtained by collecting a two-dimensional code image after being plastic-sealed.
[0111] Considering the diversity of the forms of two-dimensional code images in practical applications, the training samples in the embodiments of this specification may be images obtained by collecting paper versions of two-dimensional code images. For example, some sellers print their payment codes on paper, and buyers can pay the sellers the corresponding amount by scanning the payment codes, etc. In addition, as the paper images are used for a longer period of time, they may be worn to varying degrees and their clarity will decrease. Therefore, in the embodiments of this specification, paper images of different clarity may also be collected as training samples so that the image positioning model can be applied to more scenarios.
[0112] In practical applications, in order to reduce the wear and tear on paper QR code images, paper QR code images are usually sealed with a plastic transparent film. In the embodiments of this specification, the sealed QR code images can also be collected as training samples. For example, the sealed QR code images can be collected from different angles, and QR code images with creases in the sealing film can also be collected, etc.
[0113] In actual applications, the QR code image may also be directly displayed on the screen of the electronic terminal. The clarity of the QR code image is related to the screen resolution, the terminal's own screen brightness, and the intensity of the external environment light in which the screen is located. In the embodiments of this specification, QR code images on the screens of electronic terminals in different states can also be collected as training samples.
[0114] In order to obtain more abundant training samples, in the embodiments of this specification, training samples can also be collected by using images in an existing image database, or by using a web crawler to crawl images on the network.
[0115] In the embodiment of this specification, after determining the to-be-recognized two-dimensional code area contained in the target image in step 304, the following may also be included:
[0116] Determine whether the confidence level of the to-be-identified two-dimensional code area is greater than or equal to a preset confidence level.
[0117] In practical applications, the confidence of a QR code area can also reflect, to a certain extent, the probability that the QR code image in the QR code area can be recognized. When the confidence of the QR code area is low, it can also indicate that the probability that the QR code image in the QR code area can be recognized is low. In order to reduce resource waste, in the embodiments of this specification, subsequent correction and recognition processing can be performed on the QR code area with a confidence greater than or equal to a preset confidence, and subsequent processing can be directly terminated for the QR code area with a confidence less than the preset confidence.
[0118] In the embodiment of this specification, in step 306, determining the position information of the position detection pattern in the image of the two-dimensional code to be identified based on the image information of the two-dimensional code area to be identified may specifically include:
[0119] Determining the position detection pattern based on the image information of the to-be-identified two-dimensional code area;
[0120] According to the position detection pattern, the position coordinates of the key points are determined; the key points include at least one of the center point of the position detection pattern and the corner point of the position detection pattern.
[0121] For the convenience of calculation, in the embodiments of this specification, the position information of the points of the position detection pattern can be used to represent the position information of the position detection pattern. Specifically, the center point, corner point, etc. can be used as key points to represent the position information of the position detection pattern. Among them, the corner point can be understood as the intersection of two adjacent sides in the position detection pattern.
[0122] In the embodiments of the present specification, the position detection pattern may be a regular polygon, and specifically may include a rectangle.
[0123] In the embodiment of this specification, in step 308, determining the correction parameters according to the position information and the standard positioning information may specifically include:
[0124] Arrange the position coordinates of the key points in a preset order to obtain a key point position matrix;
[0125] Arranging the position coordinates of the standard key points in the preset two-dimensional code image template according to the preset order to obtain a standard key point position matrix;
[0126] A correction matrix is obtained according to the key point position matrix and the standard key point position matrix; the correction matrix is used to represent the mapping relationship between the key point position matrix and the standard key point position matrix.
[0127] In order to accurately locate the information in a two-dimensional plane image, at least three points are usually selected to locate the plane. Therefore, the two-dimensional code image in the embodiment of this specification may include at least three key points. In actual applications, the three key points are usually not located on the same straight line.
[0128] The preset order may refer to the order of key points at different positions. Figure 1 Taking the two-dimensional code image template shown in as an example, assuming that the two-dimensional code image in this embodiment is generated based on the template, position detection patterns are set in the upper left corner, upper right corner, and lower left corner of the two-dimensional code image respectively; the distance between the corner points in two adjacent position detection patterns is calculated, and the corner point with the largest distance is determined as the key point, that is, the key point of the upper left corner position detection pattern is the corner point of the upper left corner of the position detection pattern, the key point of the upper right corner position detection pattern is the corner point of the upper right corner of the position detection pattern, and the key point of the lower left corner position detection pattern is the corner point of the lower left corner of the position detection pattern. Corner point; accordingly, the upper left corner point of the upper left corner position detection pattern in the two-dimensional code image template can be used to determine the standard key point of the position detection image, the upper right corner point of the upper right corner position detection pattern can be used to determine the standard key point of the position detection image, and the lower left corner point of the lower left corner position detection pattern can be used to determine the standard key point of the position detection image; further, the coordinates of the key points and standard key points obtained are arranged in a preset order, such as in the order of upper left corner, upper right corner, and lower left corner, to obtain a key point position matrix and a standard key point position matrix.
[0129] In the embodiment of this specification, the correction matrix is obtained according to the key point position matrix and the standard key point position matrix, which may specifically include:
[0130] According to the formula [PQ]T = [RS], the correction parameter T is obtained;
[0131] Among them, R is the horizontal coordinate in the position coordinates of the standard key point; Q is the vertical coordinate in the position coordinates of the standard key point; P is the horizontal coordinate in the position coordinates of the key point; Q is the vertical coordinate in the position coordinates of the key point; [PQ] is the key point position matrix composed of the position coordinates of the key points; [RS] is the standard key point position matrix composed of the position coordinates of the standard key points.
[0132] In practical applications, the QR code image can be understood as being obtained by scaling, translating, rotating, and other transformations based on the standard QR code image obtained from the template. In order to achieve correct alignment of the QR code area, the transformation matrix can be obtained according to the affine transformation formula.
[0133] Still taking the above two-dimensional code image in which the upper left corner, the upper right corner, and the lower left corner are respectively provided with position detection graphics as an example, in this embodiment, the key points and the standard key points may have the following linear relationship:
[0134] Among them, (r1, s1), (r2, s2), (r3, s3) are the standard key point coordinates corresponding to the position detection patterns of the upper left corner, upper right corner, and lower left corner in the standard positioning pattern; (p1, q1), (p2, q2), (p3, q3) are the key point coordinates corresponding to the position detection patterns of the upper left corner, upper right corner, and lower left corner in the two-dimensional code image;
[0135] Expressing it in matrix form, we can get
[0136]
[0137] Further, the relationship between the key point position matrix and the standard key point position matrix can be obtained:
[0138] [RS 1] = [PQ 1] T, where T is the correction matrix.
[0139] In practical applications, for the convenience of calculation, one dimension is usually added to the two-dimensional coordinate relationship [PQ]T=[RS] to obtain [RS 1]=[PQ 1]T. Therefore, the correction matrix T can be obtained by the above process.
[0140] It should be noted that the above content is only an example to more clearly illustrate the content of this embodiment. In practical applications, the setting of key points, the number of key points, and the order of key points in the matrix can all be set according to actual needs, and are not specifically limited in the embodiments of this specification.
[0141] In the embodiment of the present specification, the correction matrix may include an affine transformation matrix, which may include at least one of a translation matrix, a rotation matrix, a scaling matrix, and a shearing matrix.
[0142] In order to adapt to the correction of more forms of two-dimensional code images, the embodiments of this specification can also use perspective transformation to correct the two-dimensional code image.
[0143] In the embodiment of this specification, step 310 corrects the to-be-recognized two-dimensional code image according to the correction parameters to obtain a corrected two-dimensional code image, which may specifically include:
[0144] Determine the position coordinates of the key points;
[0145] Based on the correction matrix, obtaining the position coordinates of the corrected key points corresponding to the position coordinates of the key points;
[0146] Assigning pixels of the key points to the corrected key points;
[0147] Grayscale interpolation operation is performed based on the pixels of the corrected key points to obtain a corrected two-dimensional code image.
[0148] Since the two-dimensional code image to be identified is generated by a two-dimensional code image template, the relative position of the position detection pattern is fixed, so the correction matrix can be used to correct the image to be identified to obtain a correction image that is similar to the two-dimensional code image template, and the pixels at the key points can be used as a reference to perform grayscale interpolation operations to obtain the corrected two-dimensional code image. Among them, the grayscale interpolation operation can use algorithms such as forward mapping method, backward mapping method, nearest neighbor interpolation, cubic convolution method, and bilinear interpolation method.
[0149] In the embodiments of this specification, the correction matrix can be a translation matrix, a rotation matrix, a scaling matrix, a shearing matrix, or a combination of multiple matrices. For example, for a two-dimensional code image to be identified with an inclined angle, a rotation matrix can be used to correct it. If the size of the image is small, it can be corrected by using a rotation matrix and then enlarged by using a scaling matrix to make its size close to the template size. In this process, the correction matrix can be understood as a combination of a rotation matrix and a scaling matrix.
[0150] In the embodiments of this specification, any pixel point in the image to be identified including the key point can be selected as a point to be corrected, and the correction matrix can be used to correct it, thereby completing the correction of the image to be identified. The pixel at any pixel point in the image to be identified can also be mapped to the corresponding pixel of the corrected image to complete the correction of the image. In order to improve the accuracy of the correction, the above-mentioned grayscale interpolation operation can be used to map the pixels.
[0151] In order to more clearly illustrate the solution in the embodiment of this specification, based on the above content, the embodiment of this specification also provides an overall flow chart of the above method. Figure 4 A schematic diagram of a process for recognizing a two-dimensional code image provided in an embodiment of this specification is shown in FIG. Figure 4 As shown, the process may include:
[0152] Step 402: Acquire a target image;
[0153] Step 404: preprocessing the target image to obtain an image pyramid of the target image;
[0154] Step 406: using the obtained image pyramid of the target image as a series of detection images;
[0155] Step 408: input the obtained detection image into the image positioning model to obtain a target image marked with a positioning frame, wherein the positioning frame is used to frame the QR code area to be identified in the target image;
[0156] Step 410: Determine whether the confidence level of the to-be-identified two-dimensional code region is greater than or equal to a preset confidence level;
[0157] Step 412: If the confidence level of the to-be-identified two-dimensional code region is less than a preset confidence level, then the identification of the to-be-identified two-dimensional code region is terminated;
[0158] Step 414: if the confidence level of the to-be-identified two-dimensional code region is greater than or equal to a preset confidence level, determining a correspondence between key points used for positioning in the to-be-identified two-dimensional code region and standard key points;
[0159] Step 416: determining a correction matrix according to the corresponding relationship; the correction matrix is used to correct the image of the two-dimensional code to be identified in the area of the two-dimensional code to be identified into an image close to the style of the two-dimensional code image template;
[0160] Step 418: Recognize the corrected two-dimensional code image.
[0161] The present specification also provides a method for training a two-dimensional code image positioning model, which can be applied to Figure 3 In the recognition method of the two-dimensional code image shown.
[0162] Figure 5 A flowchart of a training method for a two-dimensional code image positioning model provided in an embodiment of this specification is provided. The image positioning model in step 304 is obtained by training a multi-task cascade convolutional neural network model based on a two-dimensional code image training sample, which may specifically include:
[0163] Step 502: Obtain a two-dimensional code image training sample;
[0164] Step 504: annotate the two-dimensional code image training sample to obtain annotated samples; the annotation information in the annotated samples includes at least one of the position information of the position detection pattern of the two-dimensional code area in the two-dimensional code image training sample and the position information of the positioning frame of the two-dimensional code area;
[0165] Figure 6 A schematic diagram of a two-dimensional code image annotation information provided in an embodiment of this specification, such as Figure 6 As shown, the position information of the position detection figure 601 can be represented in the form of key points 602, and specifically, can also be represented by the position coordinates of the key points 602. Among them, the key points 602 can be the corner points of the position detection figure, and specifically, Figure 6 In the figure, the key point of the upper-left corner position detection figure is represented by the coordinates of the upper-left corner point of the upper-left corner position detection figure, the key point of the upper-right corner position detection figure is represented by the coordinates of the upper-right corner point of the upper-right corner position detection figure, and the key point of the lower-right corner position detection figure is represented by the coordinates of the lower-right corner point of the lower-right corner position detection figure.
[0166] The positioning frame 603 is used to frame the QR code area in the image. The position information of the positioning frame 603 can be represented in the form of a calibration point 604. Figure 6 The middle positioning frame 603 is a rectangle, which can be represented by four calibration points 604, and specifically, can also be represented by the calibration coordinates of the calibration points.
[0167] In actual applications, there is usually a small blank area around the QR code image, which is used to distinguish the QR code image from the environment in which the QR code image is located. The calibration point 604 in the embodiment of this specification can be located in the blank area. Specifically, the point in the blank area that is a preset distance away from the key point 602 in the position detection image along a preset direction can be selected as the calibration point 604. A positioning frame 603 can also be obtained based on the calibration point 604, and the QR code image is included in the positioning frame 603.
[0168] For example, suppose Figure 6 Assuming that the coordinates of the key point of the upper left corner position detection image are (x1, y1), the point with coordinates (x1-a, y1+a) in the blank area can be used as a calibration coordinate point of the positioning frame; similarly, assuming that the coordinates of the key point of the upper right corner position detection image are (x2, y2), the point with coordinates (x1-a, y1+a) in the blank area can be used as a calibration coordinate point of the positioning frame. 2+a, y2+a) is taken as the second calibration coordinate point of the positioning frame; assuming that the coordinates of the key point of the lower left corner position detection image are (x3, y3), the point with coordinates (x3-a, y3-a) located in the blank area can be taken as the third calibration coordinate point of the positioning frame; here it is assumed that the positioning frame is a rectangle, and the fourth calibration coordinate point of the positioning frame can be determined based on the three known calibration coordinate points, and then the positioning frame can be represented by the determined four calibration coordinate points.
[0169] In practical applications, the blank area around the QR code image can also be determined based on the grayscale value. For example, the QR code image is usually composed of black and white pixels according to certain rules, while the blank area is usually composed of one type of pixel. The pixel difference between the two can be used to identify the blank area, and then a positioning frame can be set in the blank area. The positioning frame can also be based on the boundary of the QR code image, and the positioning frame is composed of points at a preset distance from the boundary.
[0170] It should be noted that Figure 6 It is only a schematic diagram of the annotation information. In actual applications, the shape of the two-dimensional code image can be a variety of styles such as rectangle, circle, polygon, etc., the shape of the positioning frame and the number of positioning frame calibration points can also be multiple, and the shape, style, and number of the position detection image can also be various. These can be set according to actual needs, and this specification embodiment does not specifically limit this. Step 506: Input the annotated sample into the multi-task cascade convolutional neural network model to obtain the training result;
[0171] Step 508: Determine whether the training result meets the preset requirements and obtain a determination result;
[0172] Step 510: If the judgment result indicates that the training result meets the preset requirements, the training of the multi-task cascade convolutional neural network model is terminated to obtain the image positioning model.
[0173] The method in this embodiment uses training samples to train a multi-task cascade convolutional neural network model to obtain a two-dimensional code image positioning model that meets the requirements, so as to realize the positioning of the two-dimensional code area to be identified in the target image.
[0174] Moreover, in this embodiment, a positioning frame can be set in the blank area around the two-dimensional code image. With the help of the self-learning ability of the computer model, the obtained two-dimensional code image positioning model can identify the two-dimensional code area from the overall layout characteristics of the two-dimensional code image, and use the positioning frame for calibration, so that the positioning of the two-dimensional code area can be completed before the position detection pattern is recognized. In principle, the scheme in this embodiment, by using 4 positioning points for calibration, the training model is used to locate the two-dimensional code. In fact, it draws on the principle of human visual recognition of two-dimensional codes. It can be roughly understood that an image with black and white patchy images and blank gaps around is usually a two-dimensional code. In this embodiment, the positioning of the two-dimensional code area is not completed by using the position detection pattern in the image. It can be understood that the positioning of the two-dimensional code area can be completed before the position detection pattern in the image can be identified in the embodiment of this specification, so the speed of positioning the two-dimensional code area can be increased, and the recognition efficiency of the two-dimensional code image can be further improved. As described above, the two-dimensional code image training samples in the embodiments of the present specification may include: using preset parameters to perform image acquisition on an object containing a two-dimensional code image to generate a training sample, wherein the preset parameters include at least one of a preset tilt angle, a preset light intensity, a preset exposure, a preset shutter time, and a preset aperture size.
[0175] In practical applications, the preset parameters can be set according to actual needs. In order to enable the trained QR code image positioning model to locate QR code images in various scenarios, the preset parameters can be set to a wider range, and the ratio can be preset to obtain images within the range.
[0176] For example, the preset tilt angle interval is [0, 80] degrees. Generally, the image positioning success rate between 0 and 30 degrees is higher. In order to enable the QR code image positioning model to cope with a wider tilt angle, a larger number of training samples between 30 and 80 degrees can be selected. For example, the number of samples with a tilt angle in the range of [0, 80] degrees can be selected in a ratio of 1:1.5:2:3:3.5:3.5:4:4.5 with a step of 10 degrees.
[0177] In practical applications, the preset light intensity can be set in the range of ISO 50 to 3200; the preset exposure can be set in the range of -4 to +4EV; the preset shutter time can be set in the range of 1 / 4000 to 30 seconds; the preset aperture size can be set in the range of F2.8 to F16. It should be noted that the specific data of the preset parameters can be set as needed and are not specifically limited in this embodiment.
[0178] Moreover, in the prior art, the recognition probability of a two-dimensional code image with an inclination angle of 0 to 45 degrees, a light sensitivity of 200 to 2600, an exposure of -2 to +2, a shutter time of 1 / 3000 to 1 second, and an aperture factor of F8 to F16 is usually high, while most of the two-dimensional code images taken under characteristic conditions such as a larger inclination angle, higher or lower light sensitivity and exposure, shutter time, and aperture size are difficult or impossible to recognize. By adopting the training method in the embodiment of this specification, samples with a wider feature range are selected for model training, so that the model can locate the two-dimensional code area that cannot be successfully located or has a lower probability of being successfully located in the existing technology, and the recognition rate of the two-dimensional code can also be improved.
[0179] In the embodiment of this specification, in step 504, the training samples are labeled to obtain labeled samples, which may specifically include:
[0180] Manually annotating the two-dimensional code image training sample to obtain a first annotated sample; the first annotated sample includes a positioning frame, and the positioning frame is used to indicate the two-dimensional code area in the two-dimensional code image training sample;
[0181] Obtaining a selection box; the selection box is used to frame an image in the first annotated sample;
[0182] Placing the selection box at a plurality of different positions in the first annotated sample to obtain a plurality of corresponding second annotated samples;
[0183] Calculating the overlap between the selection box and the positioning box in each of the second annotated samples;
[0184] Determine the second labeled sample whose overlap degree is within a first preset range as a positive sample;
[0185] Determine the second labeled sample whose overlap degree is within a second preset range as a negative sample;
[0186] Determine the second annotated sample whose overlap is within the third preset range as an intermediate sample; wherein the overlap within the first preset range is greater than the overlap within the third preset range, and the overlap within the third preset range is greater than the overlap within the second preset range;
[0187] The positive samples, negative samples and intermediate samples are used to train a neural network model.
[0188] In practical applications, you can also use annotation tools to annotate images, such as the image annotation tool labellmg.
[0189] In the embodiments of this specification, a combination of a labeling tool and manual labeling may be used to obtain a labeled sample. Specifically, the collected sample image may be labeled using a labeling tool and manual labeling according to a preset ratio.
[0190] In the embodiment of the present specification, the multi-task cascade convolutional neural network model may include a convolutional layer, a connection layer and an output layer. In step 506, the labeled sample is input into the multi-task cascade convolutional neural network model to obtain a training result, which may specifically include:
[0191] Inputting the labeled samples into a convolutional layer according to a first preset size to obtain a first training result;
[0192] Determine position information of a positioning frame in the first training result, and correct the positioning frame using a regression strategy to obtain a corrected positioning frame;
[0193] Based on the corrected positioning frame, correct the first training result to obtain a connection layer training sample;
[0194] Inputting the connection layer training sample into the connection layer to obtain a second training result;
[0195] Determine the position information of the key points in the second training result; the key points include at least one of the center point of the positioning figure and the corner point of the positioning figure;
[0196] The output layer is trained based on the position information of the key points.
[0197] In practical applications, the labeled samples can also be used to train the convolution layer, connection layer and output layer simultaneously or non-simultaneously to obtain a QR code image positioning model that meets the preset performance.
[0198] Based on the same idea, the embodiments of this specification also provide a device corresponding to the above identification method. Figure 7 The embodiments of this specification provide corresponding to Figure 3 A schematic diagram of the structure of a two-dimensional code image recognition device. Figure 7 As shown, the device may include:
[0199] The target acquisition module 702 is used to acquire a target image;
[0200] The positioning module 704 is used to determine the to-be-recognized two-dimensional code region contained in the target image by using an image positioning model; the to-be-recognized two-dimensional code region contains the to-be-recognized two-dimensional code image; the image positioning model is obtained by training a multi-task cascade convolutional neural network model based on two-dimensional code image training samples;
[0201] A position information determination module 706, configured to determine the position information of the position detection pattern in the image of the to-be-identified two-dimensional code based on the image information of the to-be-identified two-dimensional code region;
[0202] A correction parameter determination module 708 is used to determine the correction parameters according to the position information and standard positioning information; the standard positioning information is the position information of the position detection pattern in the preset two-dimensional code image template;
[0203] A correction module 710, configured to correct the to-be-recognized two-dimensional code image according to the correction parameters to obtain a corrected two-dimensional code image;
[0204] The recognition module 712 is used to recognize the corrected two-dimensional code image.
[0205] Based on the same idea, the embodiments of this specification also provide a device corresponding to the above training method. Figure 8 The embodiments of this specification provide corresponding to Figure 5 A structural diagram of a training device for a two-dimensional code image positioning model. Figure 8 As shown, the device may include:
[0206] The sample acquisition module 802 is used to acquire a two-dimensional code image training sample;
[0207] The sample annotation module 804 is used to annotate the two-dimensional code image training sample to obtain annotated samples; the annotation information in the annotated samples includes at least one of the position information of the positioning pattern of the two-dimensional code area in the two-dimensional code image training sample and the position information of the positioning frame of the two-dimensional code area;
[0208] A model training module 806 is used to input the labeled samples into a multi-task cascade convolutional neural network model to obtain a training result;
[0209] A result judgment module 808 is used to judge whether the training result meets the preset requirements and obtain a judgment result;
[0210] The model determination module 810 is used to terminate the training of the multi-task cascade convolutional neural network model and obtain the image positioning model if the judgment result indicates that the training result meets the preset requirements.
[0211] Based on the same idea, the embodiments of this specification also provide a device corresponding to the above method.
[0212] A two-dimensional code image recognition device provided in an embodiment of this specification may include:
[0213] one or more processors;
[0214] a storage device for storing one or more programs,
[0215] When the one or more programs are executed by the one or more processors, the one or more processors implement the above-mentioned two-dimensional code image recognition method.
[0216] Based on the same idea, the embodiments of this specification also provide a computer-readable storage medium corresponding to the above method, on which a computer program is stored, and when the program is processed and executed, the method of two-dimensional code image recognition as described above can be implemented.
[0217] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for devices and media, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.
[0218] In the 1990s, improvements to a technology could be clearly distinguished as hardware improvements (for example, improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to the method flow). However, with the development of technology, many improvements to the method flow today can be regarded as direct improvements to the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved method flow into the hardware circuit. Therefore, it cannot be said that an improvement in a method flow cannot be implemented using a hardware entity module. For example, a programmable logic device (PLD) (such as a field programmable gate array (FPGA)) is such an integrated circuit whose logical function is determined by the user's programming of the device. Designers can "integrate" a digital system on a PLD by programming it themselves, without having to ask a chip manufacturer to design and produce a dedicated integrated circuit chip. Moreover, nowadays, instead of manually making integrated circuit chips, this kind of programming is mostly implemented by "logic compiler" software, which is similar to the software compiler used when developing and writing programs, and the original code before compilation must also be written in a specific programming language, which is called hardware description language (HDL). There is not only one HDL, but many kinds, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used ones are VHDL (Very-High-Speed Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also know that it is only necessary to program the method flow slightly in the above-mentioned hardware description languages and program it into the integrated circuit, and then it is easy to obtain the hardware circuit that implements the logic method flow.
[0219] The controller can be implemented in any appropriate manner, for example, the controller can take the form of a microprocessor or processor and a computer-readable medium storing a computer-readable program code (such as software or firmware) that can be executed by the (micro)processor, a logic gate, a switch, an application-specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320. The memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art also know that in addition to implementing the controller in a purely computer-readable program code manner, the controller can be implemented in the form of a logic gate, a switch, an application-specific integrated circuit, a programmable logic controller, and an embedded microcontroller by logically programming the method steps. Therefore, this controller can be considered as a hardware component, and the devices included therein for implementing various functions can also be regarded as structures within the hardware component. Or even, the devices for implementing various functions can be regarded as both software modules for implementing the method and structures within the hardware component.
[0220] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.
[0221] For the convenience of description, the above device is described in various units according to their functions. Of course, when implementing the present application, the functions of each unit can be implemented in the same or multiple software and / or hardware.
[0222] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0223] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0224] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0225] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.
[0226] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.
[0227] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.
[0228] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.
[0229] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.
[0230] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.
[0231] The present application may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.
[0232] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.
Claims
1. A method for recognizing a two-dimensional code image, characterized in that: include: Get the target image; Using an image positioning model, determining a two-dimensional code area to be identified contained in the target image; the two-dimensional code area to be identified contains a two-dimensional code image to be identified; the image positioning model is obtained by training a multi-task cascade convolutional neural network model according to a two-dimensional code image training sample; the image positioning model includes a multi-layer cascaded convolutional layer, a connection layer and an output layer, and using the image positioning model to determine the two-dimensional code area to be identified contained in the target image, including: Scaling the target image according to a preset rule to obtain a plurality of detection images; Inputting the detection image into a convolutional layer in the image positioning model to obtain a first number of first candidate QR code regions; intercepting the target image according to the first candidate two-dimensional code region to obtain a first auxiliary image; Inputting the first auxiliary image into a connection layer in the image positioning model to obtain a second number of second candidate two-dimensional code regions; the second number is smaller than the first number; Intercepting the target image according to the second candidate two-dimensional code area to obtain a second auxiliary image; Inputting the second auxiliary image into the output layer of the image positioning model to obtain the two-dimensional code area to be identified; Based on the image information of the to-be-identified two-dimensional code area, determining the position information of the position detection pattern in the to-be-identified two-dimensional code image; Determine the correction parameters according to the position information and standard positioning information; the standard positioning information is the position information of the position detection pattern in the preset two-dimensional code image template; Correcting the to-be-recognized two-dimensional code image according to the correction parameters to obtain a corrected two-dimensional code image; The corrected two-dimensional code image is recognized.
2. The method according to claim 1, characterized in that: The method of using the image positioning model to determine the area of the two-dimensional code to be identified contained in the target image specifically includes: Inputting the target image into the image positioning model to obtain at least one candidate QR code region; Determining the confidence level of each candidate QR code region; The candidate two-dimensional code region with the greatest confidence is determined as the two-dimensional code region to be identified.
3. The method according to claim 2, characterized in that: The step of determining the candidate two-dimensional code region with the greatest confidence as the two-dimensional code region to be identified specifically includes: Determining an intersection-over-union ratio between the candidate two-dimensional code regions according to the confidence level; Divide the candidate two-dimensional code regions whose intersection-over-union ratio is greater than a preset intersection-over-union ratio threshold into the same set to obtain multiple sets; For each of the sets, the candidate two-dimensional code region with the largest confidence is determined as the two-dimensional code region to be identified.
4. The method according to claim 1, characterized in that: The two-dimensional code image training samples include: A sample image is obtained by capturing an image of an object containing a two-dimensional code image using preset parameters, wherein the preset parameters include at least one of a preset tilt angle, a preset light intensity, a preset exposure, a preset shutter time, and a preset aperture size.
5. The method according to claim 1, characterized in that: The two-dimensional code image training samples include: At least one of an image obtained by collecting a two-dimensional code image on paper, an image obtained by collecting a two-dimensional code image displayed on a screen of an electronic product, and an image obtained by collecting a two-dimensional code image after being plastic-sealed.
6. The method according to claim 1, characterized in that: After determining the to-be-recognized two-dimensional code area contained in the target image, the method further includes: Determine whether the confidence level of the to-be-identified two-dimensional code area is greater than or equal to a preset confidence level.
7. The method according to claim 1, characterized in that: The determining, based on the image information of the to-be-identified two-dimensional code region, the position information of the position detection pattern in the to-be-identified two-dimensional code image specifically includes: Determining the position detection pattern based on the image information of the to-be-identified two-dimensional code area; According to the position detection pattern, the position coordinates of the key points are determined; the key points include at least one of the center point of the position detection pattern and the corner point of the position detection pattern.
8. The method according to claim 7, characterized in that: The determining of the correction parameters according to the position information and the standard positioning information specifically includes: Arrange the position coordinates of the key points in a preset order to obtain a key point position matrix; Arranging the position coordinates of the standard key points in the preset two-dimensional code image template according to the preset order to obtain a standard key point position matrix; A correction matrix is obtained according to the key point position matrix and the standard key point position matrix; the correction matrix is used to represent the mapping relationship between the key point position matrix and the standard key point position matrix.
9. The method according to claim 8, characterized in that: The obtaining of a correction matrix according to the key point position matrix and the standard key point position matrix specifically includes: According to the formula [PQ]T = [RS], the correction matrix T is obtained; Among them, R is the horizontal coordinate in the position coordinates of the standard key point; Q is the vertical coordinate in the position coordinates of the standard key point; P is the horizontal coordinate in the position coordinates of the key point; Q is the vertical coordinate in the position coordinates of the key point; [PQ] is the key point position matrix; [RS] is the standard key point position matrix.
10. The method according to claim 9, characterized in that: The step of correcting the to-be-recognized two-dimensional code image according to the correction parameters to obtain a corrected two-dimensional code image specifically includes: Determine the position coordinates of the key points; Based on the correction matrix, obtaining the position coordinates of the corrected key points corresponding to the position coordinates of the key points; Assigning pixels of the key points to the corrected key points; Grayscale interpolation operation is performed based on the pixels of the corrected key points to obtain a corrected two-dimensional code image.
11. The method according to claim 1, characterized in that: The image positioning model is obtained by training a multi-task cascade convolutional neural network model based on the two-dimensional code image training samples, specifically including: Obtain QR code image training samples; Annotating the two-dimensional code image training sample to obtain annotated samples; the annotation information in the annotated samples includes at least one of position information of a position detection pattern of a two-dimensional code area in the two-dimensional code image sample and position information of a positioning frame of the two-dimensional code area; Inputting the labeled samples into the multi-task cascade convolutional neural network model to obtain training results; Determine whether the training result meets the preset requirements and obtain a determination result; If the judgment result indicates that the training result meets the preset requirements, the training of the multi-task cascade convolutional neural network model is terminated to obtain the image positioning model.
12. A two-dimensional code image recognition device, characterized in that: The device comprises: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 11.
13. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 11 is implemented.
Citation Information
Patent Citations
License plate detection method based on multi-task cascade convolutional neural network
CN110033002A
Two-dimensional code identification method, device and equipment
CN110046529A
