Bar code detection and identification method based on end-to-end key point detection

Abstract

The invention discloses a bar code detection and identification method based on end-to-end key point detection, and the method comprises the following steps: 1, collecting images containing various types of bar codes from a plurality of real scenes, constructing a comprehensive and diversified data set, and marking and dividing the data set; 2, constructing a Ground Truth for the model, wherein the Ground Truth needs to comprise the following three kinds of information: key point position information, angular point angle information and bar code classification information; 3, designing and realizing an encoder and decoder model which integrates the functions of feature extraction, key point positioning, classification and discrimination, angle prediction and the like, automatically processing the input image and identifying the bar code in the image; 4, performing post-processing on corner information identified by the neural network, including association of different bar code corners, and ensuring complete identification of the bar codes; step 5, performing performance evaluation on the complete bar code detection system by using an independent verification set, and testing the performance of the model in various aspects, including accuracy, PCK (Performance of Key Kepoints) and correction rate; the method can effectively improve the bar code detection accuracy in changeable and complex scenes, and is especially suitable for modern logistics, retail and other industries needing efficient bar code processing.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision and barcode recognition and detection, and in particular relates to a barcode detection and recognition method based on end-to-end key point detection. Background Art

[0002] In modern logistics, retail, warehousing and other fields, barcode technology plays a core role, improving operational efficiency and achieving a high degree of automation. With the rapid development of information technology, higher requirements are placed on barcode detection technology, especially high-precision and high-efficiency recognition capabilities in complex environments. Traditional barcode scanning equipment and methods often encounter difficulties when processing high-density and different types of barcodes, especially at improper angles.

[0003] In order to overcome these limitations, deep learning technology has been introduced into the field of barcode detection. Using deep neural networks, especially convolutional neural networks (CNN), features can be directly extracted from images and barcodes can be recognized. This method not only improves the recognition accuracy, but also enhances the system's ability to adapt to complex environments. Deep learning models can handle barcode images of various deformations, scales, and damage, significantly improving the robustness of barcode recognition technology.

[0004] Existing deep learning barcode detection algorithms usually adopt a multi-stage processing flow, first determining the specific location of the barcode through target detection, and then correcting the barcode's posture through rotation. Although this method is effective, the multi-stage processing significantly increases the overall time consumption of barcode detection. To address this problem, this method proposes an end-to-end barcode detection solution that identifies the corner points of the barcode by integrating key point detection and directly corrects the barcode's posture. This integrated processing method not only greatly shortens the detection time, but also significantly improves the system's robustness and accuracy for barcode recognition by accurately locating the key points of the barcode. Summary of the invention

[0005] The present invention designs a barcode detection and recognition method based on end-to-end key point detection, including key point detection, angle prediction and barcode type classification. First, a heat map is generated through Gaussian circles to identify and locate the corner points of the barcode in the image; then, the direction of the barcode corner points relative to the center is calculated using a vector map to provide data support for angle prediction; finally, different types of barcodes such as one-dimensional codes, QR codes and DM codes are distinguished through different output channels. This integrated method not only improves the accuracy of barcode detection, but also greatly enhances the system's adaptability to different barcode types and complex scenarios.

[0006] To achieve the above purpose, the technical solution adopted by the present invention is as follows:

[0007] A barcode detection and recognition method based on end-to-end key point detection comprises the following steps:

[0008] Step 1: Collect images containing various types of barcodes from a variety of real scenes, build a comprehensive and diverse dataset, and annotate and divide the dataset;

[0009] Step 2: Build the ground truth for the model, which needs to include the following three types of information: key point location information, barcode classification information, and corner angle information;

[0010] Step 3, design and implement an encoder-decoder model that integrates the functions of feature extraction, key point location, classification discrimination and angle prediction to automatically process the input image and recognize the barcode in it;

[0011] Step 4, post-processing the corner point information identified by the neural network, including the association of corner points of different barcodes, to ensure complete recognition of the barcode;

[0012] Step 5: Use an independent validation set to evaluate the performance of the complete barcode detection system and test the performance of the model in various aspects, including PCK (Percentage of Correct Keypoints) and correction rate;

[0013] Preferably, in step 1, images containing various types of barcodes are collected from a variety of real scenes to construct a comprehensive and diverse data set, and the specific steps of annotating and dividing the data set are as follows:

[0014] Step 11: Collect various barcode images in logistics, supermarket retail, warehousing, and public areas. The code systems include one-dimensional codes, QR codes, DM codes, and include special scene images such as reflective and incomplete codes.

[0015] Step 12: annotate the key points of the barcode for the collected data set in the format of (X, Y, is_loss), where X represents the horizontal coordinate of the corner point, Y represents the vertical coordinate of the corner point, and is_loss represents whether the corner point is incomplete;

[0016] Step 13: Divide the labeled data set into training set, validation set, and test set in a ratio of 7:2:1.

[0017] Preferably, in step 2, the ground truth label (Ground Truth) for the model needs to include the following three types of information: key point position information, corner point angle information, and barcode classification information. The specific steps are as follows:

[0018] Step 21, for each corner point of the barcode, based on the (X, Y) described in step 12 as the center of the circle, a Gaussian Circle is used to generate a heat map as key point position information;

[0019] Step 22, by calculating the vector direction of each corner point relative to the center of the barcode, a corresponding direction vector diagram is generated as the corner point angle information;

[0020] Step 23, generating heat maps and vector maps for the one-dimensional code, QR code, and DM code according to the method described in step 21 and step 22, respectively, as three different channels, to obtain barcode classification information;

[0021] In step 24, the heat map and vector map obtained in step 23 are horizontally spliced ​​to obtain a 12-dimensional matrix vector as the final true label (GroundTruth).

[0022] Preferably, in step 22, the vector direction of each corner point relative to the center of the barcode is calculated to generate a corresponding vector graph as the corner point angle information. The specific steps are as follows:

[0023] In order to solve the sudden change problem of 0° and 360° in actual prediction, the angle information is converted into corresponding sin and cos values, that is, two values ​​from -1 to 1 are predicted, and a unique angle value is determined by the sin and cos values. The specific steps are as follows:

[0024] Step 221, according to the key point position heat map obtained in step 21, calculate the angle of each pixel point on the key point Gaussian circle pointing to the center of the barcode, ranging from 0° to 360°;

[0025] Step 222, according to the specific angle value obtained in step 221, the corresponding sin value and cos value are calculated, and the sin value and cos value are assigned to the corresponding pixel point position on the Gaussian circle described in step 21 to obtain a vector graph of sin and cos;

[0026] Step 223, horizontally concatenate the sin and cos vector graphs described in step 222 to obtain a 2-dimensional vector matrix as the final direction vector graph.

[0027] Preferably, in step 3, the specific steps of designing and implementing an encoder-decoder model to integrate the functions of feature extraction, key point positioning, classification discrimination and angle prediction are as follows:

[0028] Step 31, convert the 1280*1280 input image into a 1280*1280*3 RGB vector matrix, and further convert it into a 256*256*1 grayscale vector matrix to reduce the input feature information;

[0029] Step 32, design a three-layer downsampling convolutional neural network and a three-layer upsampling transposed convolutional neural network, and fuse the downsampling output with the upsampling output through skip connections to obtain global features. The output layer activation function uses Sigmoid and Tanh. For the corner point position information output range of 0 to 1, Sigmoid is used. For the angle information output range of -1 to 1, Tanh is used.

[0030] Step 33, input the grayscale vector matrix described in step 31 into the neural network output to obtain a 256*256*12 vector matrix;

[0031] Preferably, in step 4, the corner point information identified by the neural network is post-processed, including the association of corner points of different barcodes, and the specific steps of ensuring complete identification of the barcode include:

[0032] Step 41, calculating a candidate set of intersection points of all predicted corner points on the extended line of the predicted angle direction;

[0033] Step 42, selecting the predicted corner points that meet the conditions from the intersection candidate set described in step 41;

[0034] Step 43, performing intersection calculation based on the predicted corner points described in step 42, removing duplicate predicted corner point sets, and obtaining final matching corner points.

[0035] Preferably, in step 41, the step of calculating a candidate set of intersection points of all predicted corner points on the extended line of the predicted angle direction mainly includes:

[0036] Step 411, for all corner points P i and P j Combine them two by two and calculate their intersection I ij ;

[0037] Step 412, screening candidate intersection points I ij , the following two conditions are met:

[0038] a. Intersection I ij Cannot be in the negative direction of corner prediction;

[0039] b. Intersection I ij To two corner points P i and P j The distance d i and d j , satisfying |d i -d j |≤Δ d , where Δ d is the distance threshold.

[0040] Preferably, in step 42, the specific steps of selecting the predicted corner points that meet the conditions from the intersection candidate set in step 41 are:

[0041] Step 421, calculate each corner point P k With center point I ij The angle θ k ;

[0042] Step 422, screening corner points that meet the angle threshold, satisfying the following conditions:

[0043] a.Angle θ k Should be consistent with the predicted angle θ pred Roughly the same, satisfying |θ k -θ pred |≤Δ θ , where Δ θ is the angle threshold.

[0044] Step 423: compare the corner points meeting the angle threshold with the center point I ij Calculate the distance and take the distance d k The four or fewer nearest corner points are used as matching corner points:

[0045]

[0046] Preferably, in step 5, the performance of the complete barcode detection system is evaluated using an independent validation set to test the performance of the model in various aspects, including PCK (Percentage of Correct Keypoints) and correction rate. The specific steps are:

[0047] Step 51, define the specific PCK formula as:

[0048]

[0049] Step 52, define a specific correction rate formula as:

[0050]

[0051] Step 52, using the training set and validation set in step 1 to train the model defined in step 3, and using the test set in step 1 to perform PCK (Percentage of Correct Keypoints) and correction rate evaluation on the trained barcode detection model.

[0052] Based on the same inventive concept, the present invention provides a computer system, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the computer program is loaded into the processor, the steps of the barcode detection method based on end-to-end key point detection are implemented.

[0053] The beneficial effects of the present invention are:

[0054] The present invention proposes a barcode detection and recognition method based on end-to-end key point detection, which has significant beneficial effects. First, the method generates corner points through Gaussian heat maps, calculates angles through vector maps, and performs multi-channel classification, so that the barcode detection system can quickly and accurately identify different types of barcodes in complex scenarios, including one-dimensional codes, QR codes, and DM codes. This end-to-end design does not rely on traditional image preprocessing steps, greatly improves the adaptability and robustness of the system, and can maintain a high detection rate under conditions of poor lighting, large angle changes, blurred or damaged barcodes. Secondly, the present invention integrates corner point detection, barcode classification, and direction prediction into a single neural network, which greatly simplifies the barcode detection process. In addition, a corner point post-processing association method is proposed, which greatly improves the detection rate of barcodes. In general, the present invention effectively solves the shortcomings of traditional barcode detection methods in dynamic logistics and retail environments, and provides efficient and reliable technical support for modern logistics and retail industries. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 Schematic diagram of the process of the invention.

[0056] Figure 2 Schematic diagram of the construction of the true label (GroundTruth). DETAILED DESCRIPTION

[0057] In order to make the purpose, technical solution and advantages of the present invention more clear, the technical solution of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the present invention.

[0058] like Figure 1-Figure 2 As shown, a barcode detection and recognition method based on end-to-end key point detection includes the following steps:

[0059] Step 1: Collect images containing various types of barcodes from a variety of real scenes, build a comprehensive and diverse dataset, and annotate and divide the dataset.

[0060] Step 11, collect various barcode images in logistics, supermarket retail, warehousing, and public areas. The code systems include one-dimensional codes, QR codes, DM codes, and include special scene images such as reflective and incomplete codes.

[0061] Step 12: annotate the collected data set in the format of (X, Y, is_loss), where X represents the horizontal coordinate of the corner point, Y represents the vertical coordinate of the corner point, and is_loss represents whether the corner point is incomplete. Four consecutive (X, Y, is_loss) are the four corner points of the same barcode, and they are annotated clockwise from the upper left corner.

[0062] Step 13: Divide the labeled data set into training set, validation set, and test set in a ratio of 7:2:1.

[0063] In this embodiment, a total of 1500 pictures were collected and annotated using the makesense.ai label annotation tool. According to the ratio of 7:2, 1050 pictures were taken as the training set, 300 pictures were taken as the validation set, and 150 pictures were taken as the test set.

[0064] Step 2: Build the ground truth for the model, which needs to include the following three types of information: key point location information, barcode classification information, and corner point angle information.

[0065] Step 21, as Figure 2 As shown in the example figure in the upper left corner, for each corner point of the barcode, based on (X, Y) as the center of the circle described in step 12, a Gaussian Circle is used to generate a heat map as the key point location information.

[0066] Step 22, such as Figure 2 As shown in the example in the upper right corner, by calculating the angle of each corner point relative to the center of the barcode and calculating the sin and cos values ​​of the angle, we get the sin and cos vector graphs, and then splice them horizontally to get the corresponding direction vector graph as the corner point angle information.

[0067] Step 23, such as Figure 2 As shown in the example in the lower right corner, a heat map and a vector map are generated for a one-dimensional code, a QR code, and a DM code according to the method described in steps 21 and 22, respectively, as three different channels to obtain barcode classification information.

[0068] In step 24, the heat map and vector map obtained in step 23 are horizontally spliced ​​to obtain a 12-dimensional matrix vector as the final true label (GroundTruth).

[0069] Step 3: Design and implement an encoder-decoder model that integrates feature extraction, key point localization, classification discrimination, and angle prediction.

[0070] Step 31, convert the 1280*1280 input image into a 256*256*1 vector matrix. The original image is a 1280*1280*3 RGB image, which is converted into a 256*256*1 grayscale image.

[0071] Step 32, design a three-layer downsampling convolutional neural network and a three-layer upsampling transposed convolutional neural network, and fuse the downsampling output with the upsampling output through skip connections to obtain global features. The output layer activation function uses Sigmoid and Tanh. For the corner position information output range of 0 to 1, Sigmoid is used, and for the angle information output range of -1 to 1, Tanh is used. The first three layers of convolutional neural networks are feature extraction networks, which are used to extract the features of the target barcode in the training set image and obtain multi-scale features; the last three layers of transposed convolutional neural networks are upsampling networks, which are used to increase the dimension of the feature map and output feature maps that meet the input size.

[0072] Step 33, input the grayscale vector matrix described in step 31 into the neural network output to obtain a 256*256*12 vector matrix.

[0073] The vector matrix is ​​256*256*12. The first four feature maps represent the prediction map of the position information of the diagonal corner points of the one-dimensional code. The last two feature maps are the prediction maps of sin and cos respectively. The last eight feature maps represent the feature maps of QR code and DM code respectively.

[0074] Step 4: Post-process the corner point information identified by the neural network, including the association of different barcode corner points to ensure complete barcode recognition.

[0075] Step 41, Figure 2 Taking the one-dimensional code in the real picture as an example, calculate the candidate set of the intersection of all predicted corner points of the one-dimensional code on the extended line of the predicted angle. The intersection point cannot be in the negative direction of the predicted angle, and the distance difference between the intersection point and the two angles cannot exceed 50 pixels.

[0076] Step 42: select the predicted corner points that meet the conditions in the candidate set described in step 41, and calculate each corner point P k With center point I ij The angle θ k , filter the matches of |θ k -θ pred ∣≤15° corner points, compare the corner points that meet the angle threshold with the center point I ij Calculate the distance and take the distance d k The four or fewer nearest corner points are used as matching corner points:

[0077]

[0078] Step 43: for all candidate intersection points I ij The matching corner point set is processed to remove duplicate corner point sets, and finally the four matching corner points are obtained as the final matching corner points. The four corner points are connected in sequence to obtain a one-dimensional barcode.

[0079] Step 5: Use 150 images to evaluate the performance of the complete barcode detection model and test the performance of the model in various aspects, including PCK (Percentage of Correct Keypoints) and correction rate.

[0080] Step 51, define the specific PCK formula as:

[0081]

[0082] Step 52, define a specific correction rate formula as:

[0083]

[0084] Step 53, use 1050 training set images and 300 validation set images to train the model defined in step 3, and use 150 test set images to evaluate the PCK (Percentage of Correct Keypoints) and correction rate of the trained barcode detection model.

Claims

1. The present invention discloses a barcode detection and recognition method based on end-to-end key point detection, characterized in that: The following steps are involved: Step 1: Collect images containing various types of barcodes from a variety of real scenes, build a comprehensive and diverse dataset, and annotate and divide the dataset; Step 2: Build the ground truth for the model, which needs to include the following three types of information: key point location information, barcode classification information, and corner angle information; Step 3, design and implement an encoder-decoder model that integrates the functions of feature extraction, key point location, classification discrimination and angle prediction to automatically process the input image and recognize the barcode in it; Step 4, post-processing the corner point information identified by the neural network, including the association of corner points of different barcodes, to ensure complete recognition of the barcode; In step 5, an independent validation set is used to evaluate the performance of the complete barcode detection system, testing the performance of the model in various aspects, including PCK (Percentage of Correct KeyPoints) and correction rate.

2. The barcode detection and recognition method based on end-to-end key point detection according to claim 1, characterized in that: In step 1, images containing various types of barcodes are collected from a variety of real scenes to build a comprehensive and diverse dataset, and the specific steps of annotating and dividing the dataset are as follows: Step 11: Collect various barcode images in logistics, supermarket retail, warehousing, and public areas. The code systems include one-dimensional codes, QR codes, DM codes, and include special scene images such as reflective and incomplete codes. Step 12: annotate the key points of the barcode for the collected data set in the format of (X, Y, is_loss), where X represents the horizontal coordinate of the corner point, Y represents the vertical coordinate of the corner point, and is_loss represents whether the corner point is incomplete; Step 13: Divide the labeled data set into training set, validation set, and test set in a ratio of 7:2:

1.

3. The barcode detection and recognition method based on end-to-end key point detection according to claim 1, characterized in that: In step 2, the following three types of information need to be included in the real label for the model: key point location information, corner point angle information, and barcode classification information. The specific steps are as follows: Step 21, for each corner point of the barcode, based on the (X, Y) described in step 12 as the center of the circle, a Gaussian Circle is used to generate a heat map as the key point position information; Step 22, by calculating the vector direction of each corner point relative to the center of the barcode, a corresponding direction vector diagram is generated as the corner point angle information; Step 23, generating heat maps and vector maps for the one-dimensional code, QR code, and DM code according to the method described in step 21 and step 22, respectively, as three different channels, to obtain barcode classification information; In step 24, the heat map and vector map obtained in step 23 are horizontally spliced ​​to obtain a 12-dimensional matrix vector as the final ground truth.

4. The barcode detection and recognition method based on end-to-end key point detection according to claim 3, characterized in that: In step 22, the vector direction of each corner point relative to the center of the barcode is calculated to generate a corresponding vector diagram as the corner point angle information. The specific steps are as follows: In order to solve the sudden change problem of 0° and 360° in actual prediction, the angle information is converted into corresponding sin and cos values, that is, two values ​​from -1 to 1 are predicted, and a unique angle value is determined by the sin and cos values. The specific steps are as follows: Step 221, according to the key point position heat map obtained in step 21, calculate the angle of each pixel point on the key point Gaussian circle pointing to the center of the barcode, ranging from 0° to 360°; Step 222, according to the specific angle value obtained in step 221, the corresponding sin value and cos value are calculated, and the sin value and cos value are assigned to the corresponding pixel point position on the Gaussian circle described in step 21 to obtain a vector graph of sin and cos; Step 223, horizontally concatenate the sin and cos vector graphs described in step 222 to obtain a 2-dimensional vector matrix as the final direction vector graph.

5. The barcode detection and recognition method based on end-to-end key point detection according to claim 1, characterized in that: In step 3, the specific steps of designing and implementing an encoder-decoder model to integrate the functions of feature extraction, key point positioning, classification discrimination and angle prediction are as follows: Step 31, convert the 1280*1280 input image into a 1280*1280*3 RGB vector matrix, and further convert it into a 256*256*1 grayscale vector matrix to reduce the input feature information; Step 32, design a three-layer downsampling convolutional neural network and a three-layer upsampling transposed convolutional neural network, and fuse the downsampling output with the upsampling output through skip connections to obtain global features. The output layer activation function uses Sigmoid and Tanh. For the corner point position information output range of 0 to 1, Sigmoid is used. For the angle information output range of -1 to 1, Tanh is used. Step 33, input the grayscale vector matrix described in step 31 into the neural network output to obtain a 256*256*12 vector matrix.

6. The barcode detection and recognition method based on end-to-end key point detection according to claim 1, characterized in that: In step 4, the corner point information identified by the neural network is post-processed, including the association of different barcode corner points, and the specific steps of ensuring the complete identification of the barcode include: Step 41, calculating a candidate set of intersection points of all predicted corner points on the extended line of the predicted angle direction; Step 42, selecting the predicted corner points that meet the conditions from the intersection candidate set described in step 41; Step 43, performing intersection calculation based on the predicted corner points described in step 42, removing duplicate predicted corner point sets, and obtaining final matching corner points.

7. The barcode detection and recognition method based on end-to-end key point detection according to claim 6, characterized in that: In step 41, the step of calculating the candidate set of intersection points of all predicted corner points on the extended line of the predicted angle direction mainly includes: Step 411, for all corner points P i and P j Combine them two by two and calculate their intersection I ij ; Step 412, screening candidate intersection points I ij , the following two conditions are met: a. Intersection I ij It cannot be in the negative direction of the corner prediction; b. Intersection I ij To two corner points P i and P j The distance d i and d j , satisfying |d i -d j ∣≤Δ d , where Δ d is the distance threshold.

8. The barcode detection and recognition method based on end-to-end key point detection according to claim 6, characterized in that: In step 42, the specific steps of selecting the predicted corner points that meet the conditions from the intersection candidate set in step 41 are as follows: Step 421, calculate each corner point P k With center point I ij The angle θ k ; Step 422, screening corner points that meet the angle threshold, satisfying the following conditions: a.Angle θ k Should be consistent with the predicted angle θ pred Roughly the same, satisfying |θ k -θ pred ∣≤Δ θ , where Δ θ is the angle threshold. Step 423: compare the corner points meeting the angle threshold with the center point I ij Calculate the distance and take the distance d k The four or fewer nearest corner points are used as matching corner points:

9. The barcode detection and recognition method based on end-to-end key point detection according to claim 1, characterized in that: In step 5, an independent validation set is used to evaluate the performance of the complete barcode detection system, and the specific steps of testing the performance of the model in various aspects, including PCK (Percentage of Correct Keypoints) and correction rate, are as follows: Step 51, define the specific PCK formula as: Step 52, define a specific correction rate formula as: Step 52, using the training set and validation set in step 1 to train the model defined in step 3, and using the test set in step 1 to perform PCK (Percentage of Correct Keypoints) and correction rate evaluation on the trained barcode detection model.

10. A computer system comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the computer program is loaded into the processor, the steps of the barcode detection method based on end-to-end key point detection according to any one of claims 1 to 9 are implemented.

Images