Graphical code positioning method and device, computer device and computer readable storage medium

Abstract

The application relates to a graphic code positioning method and device, computer equipment and a computer readable storage medium. The method comprises the following steps: inputting a to-be-processed image into a first positioning model to obtain position information of a candidate code area in the to-be-processed image; mapping the position information of the candidate code area in the to-be-processed image into position information of the candidate code area on an intermediate feature map, extracting code area features corresponding to the candidate code area from the intermediate feature map according to the position information of the candidate code area on the intermediate feature map; screening a target code area from the candidate code areas according to the code area features corresponding to the candidate code areas; the target code area is a candidate code area comprising a graphic code; inputting the code area features of the target code area into a second positioning model to obtain target position information of the target code area; the second positioning model is used for determining the position information of each vertex in the target code area according to the code area features of the target code area, and the position information is taken as the target position information. By adopting the application, the positioning accuracy of the graphic code can be improved.

Description

Technical Field

[0001] This application relates to the field of image recognition technology, and in particular to a graphic code positioning method, apparatus, computer device, and computer-readable storage medium. Background Technology

[0002] In industrial settings, barcodes are primarily used for product traceability, equipment management, inventory control, quality control, and automated production. Through rapid scanning and data reading, they improve production efficiency, reduce error rates, optimize resource utilization, and enhance the transparency and controllability of the production process. Accurate and rapid positioning of barcodes and other graphic codes is crucial for improving the success rate of the entire decoding process.

[0003] In traditional technologies, deep learning-based target detection algorithms can be used to locate graphic codes. However, this method is often limited by the complexity of industrial scenarios and is easily affected by interference factors such as component texture, lighting, and image blurring, resulting in low accuracy in locating graphic codes. Summary of the Invention

[0004] Therefore, it is necessary to provide a graphic code positioning method, apparatus, computer equipment, computer-readable storage medium, and computer program product to address the above-mentioned technical problems, which can improve the positioning accuracy of graphic codes.

[0005] Firstly, this application provides a method for locating graphic codes, including:

[0006] The image to be processed is input into the first localization model to obtain the location information of the candidate code region in the image to be processed; the first localization model is used to extract intermediate feature maps from the image to be processed, and determine the location information of the candidate code region from the image to be processed based on the intermediate feature maps;

[0007] The position information of the candidate code region in the image to be processed is mapped to the position information of the candidate code region on the intermediate feature map, and the code region feature corresponding to the candidate code region is extracted from the intermediate feature map based on the position information of the candidate code region on the intermediate feature map.

[0008] Based on the code region characteristics corresponding to each candidate code region, a target code region is selected from each candidate code region; the target code region is a candidate code region that includes graphic codes.

[0009] The code region features of the target code region are input into the second positioning model to obtain the target location information of the target code region; the second positioning model is used to determine the position information of each vertex in the target code region based on the code region features of the target code region, as the target location information.

[0010] Secondly, this application provides a graphic code positioning device, comprising:

[0011] The first positioning module is used to input the image to be processed into the first positioning model to obtain the position information of the candidate code region in the image to be processed; the first positioning model is used to extract an intermediate feature map from the image to be processed, and determine the position information of the candidate code region from the image to be processed based on the intermediate feature map;

[0012] An extraction module is used to map the position information of the candidate code region in the image to be processed to the position information of the candidate code region on the intermediate feature map, and to extract the code region features corresponding to the candidate code region from the intermediate feature map based on the position information of the candidate code region on the intermediate feature map.

[0013] The filtering module is used to filter out target code regions from each candidate code region according to the code region characteristics corresponding to each candidate code region; the target code region is a candidate code region including graphic codes;

[0014] The second positioning module is used to input the code region features of the target code region into the second positioning model to obtain the target position information of the target code region; the second positioning model is used to determine the position information of each vertex in the target code region based on the code region features of the target code region, as the target position information.

[0015] Thirdly, this application provides a computer device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps in the method described above.

[0016] Fourthly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in the above-described method.

[0017] Fifthly, this application provides a computer program product comprising a computer program that, when executed by a processor, implements the steps of the method described above.

[0018] The aforementioned graphic code positioning method, apparatus, computer equipment, computer-readable storage medium, and computer program product, in which the first positioning model initially determines the position information of candidate code areas based on intermediate feature maps, achieves coarse positioning of possible areas of graphic codes, and narrows the scope of subsequent processing; the code area features extracted from the intermediate feature maps through the position information conversion of candidate code areas are reused in target code area recognition and the second positioning model, avoiding the extraction of duplicate features and improving the overall graphic code positioning efficiency; according to the code area features corresponding to each candidate code area, the target code area including graphic codes can be screened from each candidate code area, eliminating interference from non-barcode areas; the second positioning model performs fine positioning of the position information of each vertex in the target code area based on the reused code area features. The graphic code positioning process from coarse to fine improves the positioning accuracy of graphic codes and realizes a multi-stage, progressive graphic code positioning process of coarse positioning, screening, and fine positioning. It can effectively resist the influence of interference factors such as component texture, lighting changes, and imaging blur in industrial scenarios, and solves the problem of low positioning accuracy caused by the above-mentioned interference factors in traditional deep learning-based target detection algorithms, thus achieving high-precision positioning of graphic codes. Attached Figure Description

[0019] Figure 1 An application environment diagram of a graphic code positioning method provided in this application embodiment;

[0020] Figure 2 A flowchart illustrating a graphic code positioning method provided in an embodiment of this application;

[0021] Figure 3 A logic diagram of a graphic code positioning method provided in an embodiment of this application;

[0022] Figure 4 A structural block diagram of a graphic code positioning device provided in an embodiment of this application;

[0023] Figure 5 An internal structural diagram of a computer device provided in an embodiment of this application;

[0024] Figure 6 An internal structural diagram of another computer device provided in an embodiment of this application;

[0025] Figure 7 This is an internal structural diagram of a computer-readable storage medium provided in an embodiment of this application. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0027] In related technologies, image code localization methods are often limited by the complexity of industrial scenarios, the texture of components, lighting, and image blurring, resulting in inaccurate localization and poor generalization performance. With the rapid development of deep learning in object detection, applying deep learning to image code detection in industrial scenarios can significantly improve detection performance and stability, providing more accurate location information for subsequent decoding. However, these deep learning-based image code localization methods suffer from the following problems:

[0028] 1. In industrial scenarios, graphic code positioning is easily affected by external factors such as lighting, which affects the positioning accuracy. Traditional methods often require fine-tuning of parameters to extract features to adapt to different scenarios, and different scenarios require different supporting parameters, resulting in poor generalization performance.

[0029] 2. In traditional deep model-based industrial applications, image code localization often relies on a single-stage localization model, which frequently results in insufficient localization accuracy, leading to subsequent decoding failures. Furthermore, false detections often occur during localization, causing the downstream decoding module to continue running, impacting overall detection time and efficiency.

[0030] 3. In existing multi-stage localization schemes, multiple models often run in series, and each stage needs to re-extract features from the image, resulting in low efficiency.

[0031] Therefore, it is necessary to propose a multi-stage image code localization method for industrial scenarios based on deep learning, which is characterized by high accuracy, low false detection rate, and high efficiency.

[0032] The graphic code positioning method provided in this application embodiment can be applied to, for example... Figure 1 In the application environment shown, terminal 102 communicates with server 104 via a communication network. A data storage system can store the data that server 104 needs to process. The data storage system can be integrated onto server 104 or located in the cloud or on other network servers. Terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, smart in-vehicle devices, etc. Portable wearable devices can include smartwatches, smart bracelets, head-mounted devices, etc. Server 104 can be implemented using a standalone server or a server cluster consisting of multiple servers.

[0033] like Figure 2 As shown, this application provides a graphic code positioning method, which is applied to... Figure 1The method will be illustrated using terminal 102 or server 104 as examples. It is understood that the computer device may include at least one of a terminal and a server. The method includes the following steps:

[0034] S202. Input the image to be processed into the first localization model to obtain the position information of the candidate code region in the image to be processed.

[0035] In practical applications, graphic codes can include one-dimensional codes and two-dimensional codes. One-dimensional codes can include barcodes, while two-dimensional codes can include Quick Response Code (QR Code), Data Matrix Code (DM Code), etc.

[0036] The images to be processed may include original images containing potential graphic codes in industrial scenes, such as images of production materials and equipment labels taken by industrial cameras, which may contain noise in industrial scenes such as uneven lighting, component texture interference, and image blurring.

[0037] The first localization model can be a deep learning model, such as a convolutional neural network (CNN) model. This first localization model can be used for coarse localization of graphic codes, initially locating regions that might be graphic codes from a complex image to be processed, thus narrowing down the scope for subsequent fine-tuning. For example, the first localization model may include an input layer, an output layer, and hidden layers, with the hidden layers including convolutional layers, pooling layers, and fully connected layers.

[0038] Among them, the candidate code area is the region that the first localization model identifies as potentially containing a graphic code. In other words, the candidate code area may be the region containing a graphic code or it may be an interference region that does not contain a graphic code.

[0039] The first localization model is used to extract intermediate feature maps from the image to be processed, and to determine the location information of the candidate code region from the image to be processed based on the intermediate feature maps.

[0040] The intermediate feature map is the feature representation output by the intermediate feature layer during the feature extraction process of the first localization model. The intermediate feature map can include a multi-channel two-dimensional matrix. The intermediate feature map retains the key features of the image to be processed. Furthermore, since the intermediate feature map is generated by the first localization model through a downsampling operation of a convolutional neural network, its resolution is lower than that of the image to be processed. For example, the intermediate feature map can be the output feature corresponding to an 8x downsampling operation performed on the backbone network of the first localization model.

[0041] The location information of the candidate code region may include its spatial coordinates in the image to be processed. For example, the location information may include the positions of each vertex of the candidate code region. Further, if the candidate code region is rectangular, its location may include the positions of its four corner points. In a specific implementation, the image to be processed is input to a first localization model, which outputs multiple sets of localization information. Each set of localization information represents a candidate code region, i.e., a region that may be a graphic code. The multiple sets of localization information may include n sets of 4... An array of 2, where each array represents the four corner points of a candidate code region in a two-dimensional plane, serving as the position information of the candidate code region. Each row of the array corresponds to the x-axis and y-axis coordinates of a corner point in the two-dimensional plane, and each column of the array corresponds to the four corner points of the candidate code region (top left, top right, bottom left, and bottom right).

[0042] In the specific implementation, the image to be processed is input into the first localization model. The first localization model extracts features from the image through a convolutional neural network, and finally outputs two feature maps. One feature map is used to predict the center point position of the graphic code, and the other feature map is used to predict the x and y coordinate offsets of the four corner points of the graphic code relative to the center point. Then, based on these two feature maps, the first localization model predicts the positions of the four corner points of each candidate code region, i.e., n four-corner offsets. An array of 2. The first localization model represents the candidate code area by predicting the four corner points. Compared with traditional detection schemes (such as those using rectangular boxes or rotated rectangles), it has more flexible degrees of freedom. Moreover, by selecting four corner points for prediction, there is no need to perform the step of straightening the graphic code, which reduces the complexity of the overall localization process.

[0043] During the training of the first localization model, Gaussian focal loss and smooth L1 loss can be used. Gaussian focal loss introduces a Gaussian distribution to model the uncertainty of the target position. Its core function is to dynamically adjust sample weights to focus on difficult-to-classify samples. For example, for easily classifiable background or low-confidence samples, the loss weight is reduced using a decay factor; for difficult-to-classify foreground or high-confidence samples, a higher loss weight is retained to strengthen the model's learning of key targets. Simultaneously, using a Gaussian distribution to describe the probability distribution of the target position improves the robustness of the model's localization in interference scenarios. Smooth L1 loss is a combination of L1 loss (mean absolute error MAE) and L2 loss (mean squared error MSE). When the error between the predicted and true values ​​is small, L2 loss is used to make the loss function smooth and its derivative continuous near zero; when the error between the predicted and true values ​​is small, L1 loss is used to reduce the excessive influence of outliers on the loss function, reduce the model's sensitivity to extreme values, and improve the model's localization accuracy.

[0044] S204. Map the position information of the candidate code region in the image to be processed to the position information of the candidate code region on the intermediate feature map, and extract the code region features corresponding to the candidate code region from the intermediate feature map based on the position information of the candidate code region on the intermediate feature map.

[0045] In practice, since the intermediate feature maps are generated by the first localization model through a series of convolution and pooling operations, their resolution is smaller than that of the image to be processed. For example, if the resolution of the image to be processed is 1024... 1024, the resolution of the intermediate feature map is 32. 32. Therefore, based on the resolution ratio between the image to be processed and the intermediate feature map, the positional information of the candidate code region in the image to be processed can be converted into the positional information of the candidate code region on the intermediate feature map. For example, assuming the resolution of the intermediate feature map is 1 / 16 of the image to be processed, the positional information of the candidate code region in the image to be processed includes the positions of the four corner points of the candidate code region. Assuming the position of the top left corner point is (x1, y1), the position of this corner point mapped to on the intermediate feature map is (1 / 16...). x1, 1 / 16 Similarly, the positions of other corner points mapped to the intermediate feature map can be obtained, thus obtaining the position information of the candidate code region on the intermediate feature map. Then, based on the position information of the candidate code region on the intermediate feature map, the code region features corresponding to the candidate code region can be extracted from the intermediate feature map through rounding or interpolation operations.

[0046] Among them, the code region features are the features corresponding to the candidate code regions in the intermediate feature map.

[0047] In one embodiment, the positional information of the n candidate code regions output by the first localization model can be mapped onto the intermediate feature layer of the first localization model, so as to map the positional information of the candidate code regions in the image to be processed to the positional information of the candidate code regions on the intermediate feature map; then, the corresponding features are extracted by interpolation to form n code region features, and these features are reused in subsequent stages (target code region selection, fine localization). In summary, the intermediate features of the first localization model (convolutional neural network) can be extracted as code region features by mapping interpolation based on the positional information of the candidate code regions.

[0048] S206. Based on the characteristics of each candidate code region, select the target code region from each candidate code region.

[0049] In practice, the features of each code region can be analyzed by preset discrimination rules (such as feature matching threshold) or classification models (such as lightweight convolutional neural networks or fully connected classifiers) to determine whether they conform to the typical feature patterns of graphic codes, thereby selecting the target code region from each candidate code region.

[0050] In one embodiment, the code features corresponding to each candidate code area can be classified one by one, and the corresponding category can be output, including QR codes such as Quick Response Code (QR Code) and Data Matrix Code (DM Code), as well as various graphic code types such as one-dimensional codes and background categories. Candidate code areas belonging to the background category can be directly filtered, as the background category does not belong to any graphic code type.

[0051] Specifically, based on the characteristics of each candidate code region, a 1*m feature map can be output, where m represents the number of graphic code categories, including the number of code systems to be classified and a background class. The background class represents a candidate code region that is part of the background and will not participate in subsequent processes. Then, the target code region can be selected from each candidate code region based on the 1*m feature map. The addition of the background class can effectively remove false detection boxes in the background. Compared to adding a background class directly in the coarse localization stage of the first localization model, this method can more accurately exclude background regions by classifying based on the coarse localization results. Since this method predicts individual codes, it can also control the size of the input features to make some blurry small codes more accurately classified after magnification.

[0052] The target code area is the candidate code area that includes graphic codes. In other words, the target code area can be the candidate code area containing any type of graphic code, excluding background interference candidate code areas that do not belong to graphic codes.

[0053] S208. Input the code region features of the target code region into the second positioning model to obtain the target location information of the target code region.

[0054] The second localization model is used to determine the position information of each vertex in the target code region based on the code region features, and uses this as the target position information. Since the second localization model can reuse intermediate features extracted by the first localization model as input, replacing the traditional input method of the original image, it allows the second localization model to use a model with shallower network depth, fewer parameters, and less computation, thereby improving the overall efficiency of image code localization.

[0055] For example, the second positioning model can output m 4s based on the code region characteristics of the target code region. An array of 2 (where m is the number of recognized graphic codes), a single array can represent the four corner positions of the graphic code in a two-dimensional plane.

[0056] The target location information includes the location information of each vertex in the target code area. The location information of each vertex in the target code area can be the two-dimensional coordinates (x, y) of the four corner points of the target code area (such as the upper left corner, upper right corner, lower left corner, and lower right corner) in the image coordinate system, which is used to completely describe the geometric boundary of the graphic code.

[0057] The second localization model can be a deep learning model. Based on the characteristics of the target code region, it accurately outputs the position information of each vertex in the target code region, achieving fine localization of the graphic code. Thus, the first localization model quickly filters candidate code regions that may contain graphic codes from the image to be processed, achieving the region detection task; the second localization model is used to accurately predict vertex coordinates within the candidate code regions, achieving the keypoint detection task.

[0058] In one embodiment, the code region features of the target code region are input into a second localization model. This model outputs four feature maps, which are used to extract the four corner positions of the target code region. Then, the Deep Spatial Network Transform (DSNT) algorithm is used to extract the final four corner positions of the target code region. DSNT is a deep learning algorithm for accurate regression of keypoint coordinates. This method eliminates complex post-localization calculations, such as Non-Maximum Suppression (NMS) operations. Traditional feature map-based prediction methods typically use the argmax function to obtain the maximum point on the feature map as the final coordinate point, resulting in integer coordinates that cannot accurately describe the true location information, leading to accuracy errors. By using the DSNT algorithm, the output four corner positions are weighted to obtain floating-point localization results, making the image code localization more accurate.

[0059] As can be seen, in this embodiment, the first positioning model initially determines the position information of the candidate code area based on the intermediate feature map, realizing coarse positioning of the possible area of ​​the graphic code and narrowing the scope of subsequent processing; the code area features extracted from the intermediate feature map through the position information of the candidate code area are reused in the target code area recognition and the second positioning model, avoiding the extraction of repeated features and improving the overall graphic code positioning efficiency; according to the code area features corresponding to each candidate code area, the target code area including the graphic code can be screened from each candidate code area, eliminating the interference of non-barcode areas; the second positioning model performs fine positioning of the position information of each vertex in the target code area based on the reused code area features. The graphic code positioning process from coarse to fine improves the positioning accuracy of the graphic code and realizes a multi-stage, progressive graphic code positioning process of coarse positioning, screening, and fine positioning. It can effectively resist the influence of interference factors such as component texture, lighting changes, and imaging blur in industrial scenarios, and solve the problem of low positioning accuracy caused by the above-mentioned interference factors in traditional deep learning-based target detection algorithms, thus achieving high-precision positioning of graphic codes.

[0060] In some embodiments, mapping the position information of the candidate code region in the image to be processed to the position information of the candidate code region on the intermediate feature map includes: determining the mapping ratio between the image to be processed and the intermediate feature map based on the downsampling factor used to extract the intermediate feature map from the image to be processed according to the first localization model; and mapping the position information of the candidate code region in the image to be processed to the position information of the candidate code region on the intermediate feature map according to the mapping ratio.

[0061] The downsampling factor can be a scaling parameter used by the first localization model (convolutional neural network) to scale the original image during the extraction of intermediate feature maps. Since convolutional neural networks can progressively compress image size (reduce the number of pixels) through convolutional and pooling layers, the downsampling factor can be the size ratio between the original image and the intermediate feature map. For example, if the downsampling factor is 16, it means that one pixel in the intermediate feature map corresponds to 16 pixels in the image to be processed. A 16-pixel region. The mapping ratio can be the reciprocal of the downsampling factor; for example, if the downsampling factor is 16, the mapping ratio can be 1 / 16.

[0062] In practice, based on the mapping ratio, the position information of the candidate code region in the image to be processed is mapped to the position information of the candidate code region on the intermediate feature map. This can be achieved by scaling the spatial coordinates proportionally. For example, assuming the mapping ratio is r, and the top-left corner coordinates of the candidate code region in the image to be processed are (x1, y1) and the bottom-right corner coordinates are (x2, y2), then the corresponding top-left corner coordinates of the candidate code region on the intermediate feature map are (a1, b1) and the bottom-right corner coordinates are (a2, a2), where a1 = x1. r, b1=y1 r, a2 = x2 r, b2=y2 r.

[0063] As can be seen, in this embodiment, the mapping ratio is dynamically calculated by the downsampling factor of the first positioning model. The mapping ratio based on the actual downsampling factor can accurately align the spatial positions of the image to be processed and the intermediate feature map, so that the position of the candidate code area on the image to be processed can be accurately mapped to the position on the intermediate feature map. This allows the code area features corresponding to the candidate code area to be extracted from the intermediate feature map efficiently and accurately for feature reuse, thereby improving the accuracy and efficiency of graphic code positioning.

[0064] In some embodiments, the position information of the candidate code region on the intermediate feature map includes the spatial coordinates of each vertex in the candidate code region on the intermediate feature map; extracting the code region features corresponding to the candidate code region from the intermediate feature map based on the position information of the candidate code region on the intermediate feature map includes: determining the surrounding pixels of each vertex in the candidate code region on the intermediate feature map based on the spatial coordinates of each vertex in the candidate code region on the intermediate feature map; and determining the code region features corresponding to the candidate code region based on the feature values ​​of the surrounding pixels.

[0065] Among them, the spatial coordinates of each vertex in the candidate code region on the intermediate feature map are floating-point spatial coordinates obtained after mapping transformation, such as (10.2, 8.7). These coordinates may fall in the pixel gaps of the intermediate feature map, rather than strictly integer pixel points, such as (10, 8), (11, 9), etc. Therefore, feature values ​​need to be supplemented by interpolation.

[0066] Among them, the surrounding pixels of each vertex in the candidate code region on the intermediate feature map are integer pixels. For example, for floating-point spatial coordinates (10.2, 8.7), the surrounding pixels can be (10, 8), (10, 9), (11, 8), (11, 9), etc., which serve as the basis for interpolation calculation.

[0067] In practical applications, since the pixel positions in the intermediate feature map are defined by integer coordinates (i.e., the effective coordinates on the feature map are discrete integers), while the position information of the candidate code region on the intermediate feature map obtained through mapping is in floating-point spatial coordinates, it cannot be directly mapped to the integer pixel positions in the feature map. Directly rounding down would lead to feature region localization errors. Therefore, when the spatial coordinates of each vertex in the candidate code region on the intermediate feature map are floating-point spatial coordinates, an interpolation algorithm can be used to interpolate based on the known feature values ​​of the surrounding integer pixels to calculate the feature values ​​of each vertex in the candidate code region, and then the code region features corresponding to the candidate code region can be determined based on the feature values ​​of each vertex in the candidate code region. Optionally, the interpolation algorithm may include a bilinear interpolation algorithm.

[0068] For example, suppose any vertex A in the candidate code region has spatial coordinates (x=10.2, y=8.7) on the intermediate feature map, and the four integer pixels surrounding it are: B1=(10,8), with feature value f1; B2=(10,9), with feature value f2; B3=(11,8), with feature value f3; and B4=(11,9), with feature value f4. Then, the final feature value f of the position of vertex A can be calculated using bilinear interpolation. Specifically, interpolation is first performed in the x-direction to calculate the feature value f(x=10.2,y=8) = w1 corresponding to the coordinate point (10.2,8). f1 + w3 f3, and the eigenvalue f(x=10.2,y=9) = w2 corresponding to the coordinate point (10.2,9). f2 + w4 f4, where w1, w2, w3, and w4 are distance weights determined by the distances between vertex A and its surrounding pixels; the greater the distance, the smaller the weight. Then, interpolation is performed in the y-direction to calculate the final feature value f(x=10.2, y=8.7) = w5 for vertex A (10.2, 8.7). f(x=10.2,y=8)+w6 f(x=10.2,y=9), where w5 and w6 are distance weights, determined by the distances between vertex A(10.2, 8.7) and coordinate points (10.2,8) and (10.2,9), respectively. The larger the distance, the smaller the weight. Similarly, the final feature values ​​corresponding to each vertex of the candidate code region in the intermediate feature map can be calculated. Thus, the final region range corresponding to the candidate code region can be located from the intermediate feature map, and the code region features corresponding to the candidate code region can be extracted based on the final region range corresponding to the candidate code region in the intermediate feature map.

[0069] As can be seen, in this embodiment, the feature values ​​of surrounding integer pixels can be used to estimate the code region features corresponding to the candidate code region, making the extracted code region features more refined and continuous, avoiding feature loss due to coordinate discretization, and providing higher quality feature input for subsequent fine localization.

[0070] In some embodiments, selecting target code regions from candidate code regions based on the code region features corresponding to each candidate code region includes: inputting the code region features corresponding to the candidate code regions into a classification model to obtain the probability distribution of the region categories corresponding to the candidate code regions; the probability distribution includes the predicted probability that the candidate code regions belong to each region category; determining the region category with the highest predicted probability as the region category of the candidate code region; and determining the candidate code regions whose region categories belong to the graphic code category from among the candidate code regions as the target code regions.

[0071] In practice, code region features can be iteratively input into the classification model one by one to obtain the probability distribution of the region category corresponding to each code region feature. The region category can include different code types such as Quick Response Code (QRCode), Data Matrix Code (DM Code), and one-dimensional barcodes, as well as a background category. The background category is used to determine whether the candidate code region is a graphic code. If it is not a graphic code (belonging to the background category), the subsequent process for the current candidate code region ends, and the next code region feature is processed. If it is determined to be a graphic code (not belonging to the background category), this code region feature is used as the code region feature of the target code region and subsequently input into the second localization model to complete the graphic code localization process.

[0072] In this probability distribution, each predicted probability represents the likelihood that a candidate code region belongs to a certain region category. A higher predicted probability indicates a greater likelihood that the candidate code region belongs to that region category. Therefore, the region category corresponding to the highest predicted probability can be selected from the probability distribution as the final region category for that candidate code region. By traversing all candidate code regions and checking the region category of each candidate code region, only candidate code regions with the region category of graphic code are retained and identified as target code regions.

[0073] Optionally, the classification model may include a lightweight deep learning model, such as a classification network based on fully connected layers and convolutional layers. In practical applications, since the classification model uses the features obtained by mapping and interpolating the candidate code regions identified by the first localization model to the intermediate feature layer as input, the network parameters before the intermediate feature layer of the first localization model can be fixed (frozen) during model training.

[0074] As can be seen, in this embodiment, by outputting the probability distribution of the regional categories of each candidate code area through the classification model, and determining the regional category of each candidate code area based on the predicted probability in the probability distribution, the interference of non-graphic code areas can be eliminated, and only the effective area containing the real barcode can be retained, providing accurate targets for subsequent fine positioning, reducing the false detection rate of graphic code positioning, and improving the efficiency and accuracy of graphic code positioning.

[0075] In some embodiments, the code region features of the target code region are input into a second localization model to obtain the target location information of the target code region, including: inputting the code region features of the target code region into the second localization model; determining the feature map of each vertex in the target code region based on the code region features of the target code region using the second localization model; the feature value of each pixel in the feature map is used to characterize the probability that the pixel belongs to the corresponding vertex; for the feature map of any vertex, normalizing the feature values ​​of each pixel in the feature map of the vertex using the second localization model to obtain the normalized feature value of each pixel; and determining the position information of the vertex based on the normalized feature value of each pixel and the spatial coordinates of each pixel; and determining the target location information of the target code region based on the position information of each vertex using the second localization model.

[0076] In a specific implementation, the second localization model can be a deep learning model based on the Deep Spatial Network Transform (DSNT) algorithm.

[0077] In its implementation, the second localization model determines the feature map of each vertex in the target code region based on the code region characteristics. This feature map can be a heatmap in the DSNT algorithm, which is a two-dimensional feature matrix generated by the second localization model for each vertex. The feature value of each pixel in the feature map of each vertex represents the probability that the pixel belongs to the corresponding vertex (such as the top left vertex or the bottom right vertex). The higher the value, the greater the probability that the pixel is a vertex.

[0078] Then, the feature values ​​of each pixel in the feature map of the vertex are normalized by the second localization model. This can be done by using the spatial Softmax function to normalize the feature values ​​of each pixel in the feature map of the vertex, converting the feature value of each pixel in the feature map of the vertex into a normalized probability value, so that the sum of the normalized probability values ​​of all pixels is equal to 1. This normalized probability value is the normalized feature value.

[0079] Then, for any vertex's feature map, the normalized feature value of each pixel in the feature map is multiplied point-to-point with the spatial coordinates of each pixel to obtain the vertex's position information. Specifically, the spatial coordinates (x, y) of each pixel in the feature map can be converted into two X and Y arrays with values ​​ranging from [-1, 1]. These two X and Y arrays have the same size as the normalized feature map. Then, for any vertex, the normalized feature map corresponding to that vertex is multiplied point-to-point with the X array and then added to obtain the vertex's x-coordinate; simultaneously, the normalized feature map corresponding to that vertex is multiplied point-to-point with the Y array, and the results of each point-to-point multiplication are added to obtain the vertex's y-coordinate; based on the vertex's x and y coordinates, the vertex's position information can be obtained. Similarly, the position information of each vertex can be determined.

[0080] As can be seen, in this embodiment, by generating feature maps of vertices and using the feature values ​​of pixels to characterize the probability that a pixel belongs to the corresponding vertex, the model can focus more on high-probability potential vertex regions, effectively filter background noise and interference from non-vertex regions, and improve feature utilization efficiency. Furthermore, by normalizing the feature values ​​and then obtaining the floating-point position information of vertices based on the normalized feature values ​​of each pixel and the spatial coordinates of each pixel, the accuracy limitation of traditional discrete pixel positioning can be overcome, thus improving the accuracy of image code positioning.

[0081] In some embodiments, inputting the code region features of the target code region into the second localization model includes: inputting the code region features of the target code region into a feature enhancement model to obtain enhanced code region features of the target code region; the feature enhancement model is used to filter the noise of the code region features and enhance the details of the code region features to obtain enhanced code region features; and inputting the enhanced code region features of the target code region into the second localization model.

[0082] The feature enhancement model is used to optimize feature quality. Its input is the code region features corresponding to the target code region, and its output is the enhanced code region features of the same size. In practical applications, since the feature enhancement model uses features obtained by mapping and interpolating the candidate code regions identified by the first localization model to the intermediate feature layer as input, the network parameters before the intermediate feature layer of the first localization model can be fixed (frozen) during training. Similarly, the network parameters of the feature enhancement model also need to be fixed during training the second localization model.

[0083] The noise filtering process may include reducing interference information in the code area features (such as noise caused by imaging blur or uneven illumination) through smoothing and noise reduction algorithms (such as Gaussian filtering, mean filtering or adaptive filtering); the detail enhancement process may include edge enhancement, contrast enhancement or sharpening algorithms to enhance key details in the code area features (such as barcode stripe edges, vertex texture abrupt changes, etc.).

[0084] As can be seen, in this embodiment, the feature enhancement model enhances the code features of the target code area, including filtering out some interference such as blurred lighting, and restores higher quality features which are then fed into the second positioning model, thereby further improving the accuracy of graphic code positioning.

[0085] For the convenience of those skilled in the art, Figure 3 An exemplary logic diagram of a graphic code positioning method is provided. For example... Figure 3 As shown, after the input image (i.e. the image to be processed) passes through the coarse localization module (including the first localization model), the coarse localization module will output multiple sets of localization information. Each set represents a candidate code area, that is, a region that may be a graphic code. Then, the position of each candidate code area is mapped to the intermediate feature layer of the coarse localization network, and the corresponding features are extracted by interpolation for feature reuse.

[0086] Next, the candidate code region features are cyclically input into the classification module (including the classification model). The classification module outputs the corresponding category information, such as different code systems like QR codes, DM codes, and one-dimensional codes, as well as the background class. The background class is used to distinguish candidate code regions that are not graphic codes. If the classification module determines that the current candidate code region is not a graphic code, it will end the subsequent process for that candidate code region and continue processing the features of the next candidate code region. If the classification module determines that the current candidate code region is a graphic code, it will send the candidate code region features to the super-resolution module (including the feature enhancement model) for feature enhancement. After that, the output of the super-resolution module will be input into the fine localization module (including the second localization model). After the fine localization module completes its processing, it will output the final result as the code region, completing the localization process.

[0087] As can be seen, the aforementioned graphic code localization method, based on deep learning algorithms, decomposes the graphic code localization task in industrial scenarios into four stages: coarse localization, classification, super-resolution, and fine localization. Different tasks and focuses are assigned to these stages, allowing them to collaborate and simplifying the complex task. Furthermore, addressing the challenges of graphic code detection in industrial scenarios, the coarse-to-fine detection process improves detection accuracy, the inclusion of a classification model reduces the probability of false detections, and the use of feature reuse—reusing intermediate features extracted by the coarse localization network in the first stage as input for subsequent stages instead of traditional image input—allows subsequent models to use shallower networks with fewer parameters and lower computational cost, thus improving overall detection efficiency. In addition, the introduction of a super-resolution model to enhance intermediate features further improves detection accuracy.

[0088] It should be understood that although the steps in the flowcharts of the above embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps.

[0089] Based on the same inventive concept, this application also provides a graphic code positioning device. The solution provided by this device is similar to the solution described in the above method. Therefore, the specific limitations of one or more graphic code positioning device embodiments provided below can be found in the limitations of the graphic code positioning method above, and will not be repeated here.

[0090] like Figure 4 As shown, this application embodiment provides a graphic code positioning device 400, including:

[0091] The first positioning module 402 is used to input the image to be processed into the first positioning model to obtain the position information of the candidate code region in the image to be processed; the first positioning model is used to extract an intermediate feature map from the image to be processed, and determine the position information of the candidate code region from the image to be processed based on the intermediate feature map;

[0092] The extraction module 404 is used to map the position information of the candidate code region in the image to be processed to the position information of the candidate code region on the intermediate feature map, and to extract the code region feature corresponding to the candidate code region from the intermediate feature map according to the position information of the candidate code region on the intermediate feature map.

[0093] The filtering module 406 is used to filter out target code areas from each candidate code area according to the code area characteristics corresponding to each candidate code area; the target code area is a candidate code area including graphic codes.

[0094] The second positioning module 408 is used to input the code region features of the target code region into the second positioning model to obtain the target position information of the target code region; the second positioning model is used to determine the position information of each vertex in the target code region according to the code region features of the target code region, as the target position information.

[0095] In some embodiments, in mapping the position information of the candidate code region in the image to be processed to the position information of the candidate code region on the intermediate feature map, the extraction module 404 is specifically configured to: determine the mapping ratio between the image to be processed and the intermediate feature map by extracting the downsampling factor of the intermediate feature map from the image to be processed according to the first localization model; and map the position information of the candidate code region in the image to be processed to the position information of the candidate code region on the intermediate feature map according to the mapping ratio.

[0096] In some embodiments, the position information of the candidate code region on the intermediate feature map includes the spatial coordinates of each vertex in the candidate code region on the intermediate feature map; in terms of extracting the code region features corresponding to the candidate code region from the intermediate feature map based on the position information of the candidate code region on the intermediate feature map, the extraction module 404 is specifically used to: determine the surrounding pixels of each vertex in the candidate code region on the intermediate feature map based on the spatial coordinates of each vertex in the candidate code region on the intermediate feature map; and determine the code region features corresponding to the candidate code region based on the feature values ​​of the surrounding pixels.

[0097] In some embodiments, in selecting target code regions from the candidate code regions based on the code region features corresponding to each candidate code region, the filtering module 406 is specifically configured to input the code region features corresponding to the candidate code regions into a classification model to obtain a probability distribution of the region categories corresponding to the candidate code regions; the probability distribution includes the predicted probability that the candidate code regions belong to each region category; the region category with the highest predicted probability is determined as the region category of the candidate code region; and candidate code regions whose region categories belong to the graphic code category are determined from the candidate code regions as the target code regions.

[0098] In some embodiments, in inputting the code region features of the target code region into the second positioning model to obtain the target location information of the target code region, the second positioning module 408 is specifically configured to: input the code region features of the target code region into the second positioning model; determine the feature map of each vertex in the target code region based on the code region features of the target code region using the second positioning model; the feature value of each pixel in the feature map is used to characterize the probability that the pixel belongs to the corresponding vertex; for the feature map of any vertex, normalize the feature values ​​of each pixel in the feature map of the vertex using the second positioning model to obtain the normalized feature value of each pixel; and determine the location information of the vertex based on the normalized feature value of each pixel and the spatial coordinates of each pixel; and determine the target location information of the target code region based on the location information of each vertex using the second positioning model.

[0099] In some embodiments, in inputting the code region features of the target code region into the second positioning model, the second positioning module 408 is specifically configured to: input the code region features of the target code region into a feature enhancement model to obtain enhanced code region features of the target code region; the feature enhancement model is configured to filter the noise of the code region features and enhance the details of the code region features to obtain the enhanced code region features; and input the enhanced code region features of the target code region into the second positioning model.

[0100] Each module in the aforementioned graphic code positioning device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0101] In some embodiments, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 5As shown, the computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores image data. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. When the computer program is executed by the processor, it implements the steps in the aforementioned graphic code positioning method.

[0102] In some embodiments, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 6 As shown, the computer device includes a processor, memory, input / output interface, communication interface, display unit, and input device. The processor, memory, and input / output interface are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interface. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The input / output interface is used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When the computer program is executed by the processor, it performs the steps in the aforementioned graphic code positioning method. The display unit of the computer device forms a visually visible image and can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen; the input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs or touchpads set on the casing of the computer device, or external keyboards, touchpads or mice, etc.

[0103] Those skilled in the art will understand that Figure 5 or Figure 6The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0104] In some embodiments, a computer device is provided, the computer device including a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the steps in the above method embodiments.

[0105] In some embodiments, such as Figure 7 The diagram shows the internal structure of a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps described in the above-described method embodiments.

[0106] In some embodiments, a computer program product is provided, which includes a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0107] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0108] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0109] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0110] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for locating images using graphic codes, characterized in that, include: The image to be processed is input into the first localization model to obtain the position information of the candidate code region in the image to be processed; The first localization model is used to extract intermediate feature maps from the image to be processed, and to determine the location information of the candidate code region from the image to be processed based on the intermediate feature maps; The position information of the candidate code region in the image to be processed is mapped to the position information of the candidate code region on the intermediate feature map. Based on the spatial coordinates of each vertex in the candidate code region on the intermediate feature map, the surrounding pixels of each vertex in the candidate code region on the intermediate feature map are determined. Based on the feature values ​​of the surrounding pixels, the code region feature corresponding to the candidate code region is determined. The code region features corresponding to the candidate code region are input into the classification model to obtain the probability distribution of the region category corresponding to the candidate code region; the probability distribution includes the predicted probability that the candidate code region belongs to each region category; the region category with the highest predicted probability is determined as the region category of the candidate code region; from each candidate code region, the candidate code region whose region category belongs to the graphic code category is determined as the target code region; The code region features of the target code region are input into the second positioning model to obtain the target location information of the target code region; The second positioning model is used to determine the position information of each vertex in the target code region based on the code region characteristics of the target code region, and use it as the target position information.

2. The method according to claim 1, characterized in that, The step of mapping the position information of the candidate code region in the image to be processed to the position information of the candidate code region on the intermediate feature map includes: Based on the downsampling factor used by the first localization model to extract the intermediate feature map from the image to be processed, the mapping ratio between the image to be processed and the intermediate feature map is determined. According to the mapping ratio, the position information of the candidate code region in the image to be processed is mapped to the position information of the candidate code region on the intermediate feature map.

3. The method according to claim 1, characterized in that, The step of inputting the code region features of the target code region into the second positioning model to obtain the target location information of the target code region includes: The code region features of the target code region are input into the second localization model. The second localization model determines the feature map of each vertex in the target code region based on the code region features of the target code region. The feature value of each pixel in the feature map is used to characterize the probability that the pixel belongs to the corresponding vertex. For any vertex feature map, the feature values ​​of each pixel in the vertex feature map are normalized by the second localization model to obtain the normalized feature values ​​of each pixel, and the position information of the vertex is determined based on the normalized feature values ​​of each pixel and the spatial coordinates of each pixel. The second positioning model determines the target location information of the target code area based on the position information of each vertex.

4. The method according to claim 3, characterized in that, The step of inputting the code region features of the target code region into the second localization model includes: The code region features of the target code region are input into the feature enhancement model to obtain the enhanced code region features of the target code region; the feature enhancement model is used to filter the noise of the code region features and enhance the details of the code region features to obtain the enhanced code region features. The enhanced code region features of the target code region are input into the second localization model.

5. A graphic code positioning device, characterized in that, include: The first positioning module is used to input the image to be processed into the first positioning model to obtain the position information of the candidate code region in the image to be processed; The first localization model is used to extract intermediate feature maps from the image to be processed, and to determine the location information of the candidate code region from the image to be processed based on the intermediate feature maps; The extraction module is used to map the position information of the candidate code region in the image to be processed to the position information of the candidate code region on the intermediate feature map, and to determine the surrounding pixels of each vertex in the candidate code region on the intermediate feature map based on the spatial coordinates of each vertex in the candidate code region on the intermediate feature map; and to determine the code region feature corresponding to the candidate code region based on the feature values ​​of the surrounding pixels. The filtering module is used to input the code region features corresponding to the candidate code region into the classification model to obtain the probability distribution of the region category corresponding to the candidate code region; the probability distribution includes the predicted probability of the candidate code region belonging to each region category; the region category with the highest predicted probability is determined as the region category of the candidate code region; and candidate code regions whose region category belongs to the graphic code category are determined from each candidate code region as target code regions. The second positioning module is used to input the code region features of the target code region into the second positioning model to obtain the target location information of the target code region. The second positioning model is used to determine the position information of each vertex in the target code region based on the code region characteristics of the target code region, and use it as the target position information.

6. The apparatus according to claim 5, characterized in that, In mapping the position information of the candidate code region in the image to be processed to the position information of the candidate code region on the intermediate feature map, the extraction module is specifically used to extract the downsampling factor used by the intermediate feature map from the image to be processed according to the first localization model, and determine the mapping ratio between the image to be processed and the intermediate feature map. According to the mapping ratio, the position information of the candidate code region in the image to be processed is mapped to the position information of the candidate code region on the intermediate feature map.

7. A computer device, the computer device comprising a memory and a processor, the memory storing a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 4.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 4.

Images