Two-dimensional code detection method and device, model training method and device, equipment and storage medium

Abstract

Embodiments of the application disclose a two-dimensional code detection method and device, a model training method and device, equipment and a storage medium. The method comprises: obtaining a two-dimensional code region of a to-be-detected image according to a preset first detection positioning model; obtaining a square region of a back-to-word type in the two-dimensional code in the to-be-detected image according to a preset second detection positioning model; inputting the square region of the back-to-word type into a preset first classification model to obtain a first classification result of the square of the back-to-word type in the two-dimensional code; obtaining a square region of a non-back-to-word type in the two-dimensional code, and inputting the square region of the non-back-to-word type into a preset second classification model to obtain a second classification result of the square of the non-back-to-word type in the two-dimensional code, thereby avoiding the technical problem that slight defects on the two-dimensional code are not easy to distinguish, and greatly improving the precision and efficiency of two-dimensional code detection in a small circuit breaker.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, and in particular to a two-dimensional code detection method, a model training method, an apparatus, a device and a storage medium. BACKGROUND

[0002] In the production process of a miniature circuit breaker, a two-dimensional code (which may contain characters below) is usually sprayed on the surface of the device to record product information, so as to facilitate inquiry and traceability. However, the two-dimensional code on the surface of some miniature circuit breakers on the production line may have defects such as incomplete spraying and uneven density, so it is necessary to identify miniature circuit breakers with two-dimensional code printing quality defects from the production line to facilitate quick repair.

[0003] At present, there are two directions for two-dimensional code printing quality defect detection algorithms: one direction is based on traditional image algorithms, which can realize manual positioning of two-dimensional codes and judgment of whether defects exist, but in actual industrial production, due to the variety of two-dimensional code printing quality defects and the high requirement for algorithm running time, traditional image algorithms have problems in applicability and efficiency; the other direction is based on deep learning, such as Chinese patents CN 111524119 A and CN 114139564 A. In patent CN 111524119 A, GoogLeNet network model is mainly used to determine whether a two-dimensional code has defects, but no method for positioning the two-dimensional code is given, and GoogLeNet classification algorithm appeared earlier, so its precision and efficiency are relatively low; and in patent CN 114139564 A, only a target detection positioning model (VGG-SSD) is used for two-dimensional code positioning and classification, which results in low classification accuracy, especially for slight defects, and VGG-SSD appeared earlier, resulting in low precision and efficiency of two-dimensional code detection. SUMMARY

[0004] In view of the deficiencies of the prior art, the present application provides a two-dimensional code detection method, a model training method, an apparatus, a device and a storage medium, aiming to solve the technical problem of low precision and efficiency of two-dimensional code detection in images in the prior art.

[0005] In a first aspect, an embodiment of the present application provides a two-dimensional code detection method, which includes:

[0006] obtaining a two-dimensional code region of a to-be-detected image according to a preset first detection positioning model;

[0007] obtaining a Hengzi type square region of the two-dimensional code in the to-be-detected image according to a preset second detection positioning model;

[0008] input the square region of the back-shaped square into a preset first classification model to obtain a first classification result of the back-shaped square in the two-dimensional code;

[0009] obtain a non-back-shaped square region of the two-dimensional code, and input the non-back-shaped square region into a preset second classification model to obtain a second classification result of the non-back-shaped square in the two-dimensional code.

[0010] In a second aspect, an embodiment of the present application provides a model training method, which comprises:

[0011] obtaining a sample image set collected;

[0012] annotating each sample image in the sample image set to obtain an annotated sample image set;

[0013] preprocessing the annotated sample image set to obtain a first training set and a verification set;

[0014] training a two-dimensional code image detection network according to the first training set and the verification set, wherein the detection network comprises a first detection positioning model, a second detection positioning model, a first classification model and a second classification model; the detection positioning category of the first detection positioning model comprises a two-dimensional code of an image, the detection category of the second detection positioning model comprises a back-shaped square of the two-dimensional code, the classification category of the first classification model comprises the back-shaped square, and the classification category of the second classification model comprises a non-back-shaped square in the two-dimensional code.

[0015] In a third aspect, an embodiment of the present application provides a two-dimensional code detection device, which comprises:

[0016] a first obtaining unit configured to obtain a two-dimensional code region of a to-be-detected image according to a preset first detection positioning model;

[0017] a second obtaining unit configured to obtain a back-shaped square region of the two-dimensional code in the to-be-detected image according to a preset second detection positioning model;

[0018] a first input unit configured to input the back-shaped square region into a preset first classification model to obtain a first classification result of the back-shaped square in the two-dimensional code;

[0019] a second input unit configured to obtain a non-back-shaped square region of the two-dimensional code, and input the non-back-shaped square region into a preset second classification model to obtain a second classification result of the non-back-shaped square in the two-dimensional code.

[0020] In a fourth aspect, an embodiment of the present application provides a model training device, which comprises:

[0021] The third acquisition unit is used to acquire the collected sample image set;

[0022] The annotation unit is used to annotate each sample image in the sample image set to obtain an annotated sample image set;

[0023] The preprocessing unit is used to preprocess the labeled sample image set to obtain a first training set and a validation set;

[0024] The training unit is used to train a detection network for QR code images based on the first training set and the validation set, wherein the detection network includes a first detection and localization model, a second detection and localization model, a first classification model, and a second classification model; the detection and localization category of the first detection and localization model includes the QR code in the image, the detection category of the second detection and localization model includes the squares in the QR code, the classification category of the first classification model includes the squares in the QR code, and the classification category of the second classification model includes the non-squares in the QR code.

[0025] Fifthly, embodiments of the present invention also provide an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the QR code detection method or model training method as described above.

[0026] Sixthly, embodiments of the present invention also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the above-described QR code detection method or model training method.

[0027] This invention provides a QR code detection method, model training method, apparatus, device, and storage medium. It employs a first detection and localization model to acquire the QR code region of the image to be detected, and a second detection and localization model to acquire the square region of the QR code. Finally, a first classification model and a second classification model are used to classify the square region and non-square region of the QR code, respectively, to determine whether the QR code in the image to be detected is defective. This invention, by employing multiple detection and localization models and classification models to detect the QR code in the image to be detected, avoids the technical problem of being unable to distinguish minor defects on the QR code, greatly improving the accuracy and efficiency of QR code detection. Attached Figure Description

[0028] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0029] Figure 1 An application scenario schematic diagram provided for an embodiment of the present application is shown in FIG. 1.

[0030] Figure 2 A flowchart of a two-dimensional code detection method provided for an embodiment of the present application is shown in FIG. 2.

[0031] Figure 3 A flowchart of a two-dimensional code detection method provided for an embodiment of the present application is shown in FIG. 3.

[0032] Figure 4 Another flowchart of a two-dimensional code detection method provided for an embodiment of the present application is shown in FIG. 4.

[0033] Figure 5 A schematic block diagram of generating a feature pyramid provided for an embodiment of the present application is shown in FIG. 5.

[0034] Figure 6 Another flowchart of a two-dimensional code detection method provided for an embodiment of the present application is shown in FIG. 6.

[0035] Figure 7 A flowchart of a model training method provided for an embodiment of the present application is shown in FIG. 7.

[0036] Figure 8 A small circuit breaker image after labeling provided for an embodiment of the present application is shown in FIG. 8.

[0037] Figure 9 An image after two-dimensional code labeling of a square block provided for an embodiment of the present application is shown in FIG. 9.

[0038] Figure 10 A flowchart of a model training method provided for an embodiment of the present application is shown in FIG. 10.

[0039] Figure 11 Another flowchart of a model training method provided for an embodiment of the present application is shown in FIG. 11.

[0040] Figure 12 A schematic block diagram of a two-dimensional code detection device provided for an embodiment of the present application is shown in FIG. 12.

[0041] Figure 13 A schematic block diagram of a model training device provided for an embodiment of the present application is shown in FIG. 13.

[0042] Figure 14 A schematic block diagram of an electronic device provided for an embodiment of the present application is shown in FIG. 14. DETAILED DESCRIPTION

[0043] With reference to the drawings and the embodiments of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described. Obviously, the described embodiments are some of the embodiments of the present application, but not all of the embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts are within the scope of the present application.

[0044] It should be understood that the terms "comprising" and "including" as used in the specification and the appended claims indicate the presence of the described features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0045] It should also be understood that the terms used in the present application specification are only for the purpose of describing particular embodiments and are not intended to limit the present application. As used in the present application specification and the appended claims, the singular forms "a", "an" and "the" are intended to include the plural forms unless the context clearly indicates otherwise.

[0046] It should be further understood that the term "and / or" used in the present application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations thereof, and includes these combinations.

[0047] Please refer to Figure 1 , the system architecture 100 can include terminal devices 101, 102, 103, a network 104, and a server 105. The network 104 is used to provide a communication link medium between the terminal devices 101, 102, 103 and the server 105. The network 104 can include various connection types, such as wired, wireless communication links, or optical fiber cables, etc.

[0048] A user can use the terminal devices 101, 102, 103 to interact with the server 105 through the network 104 to receive or send messages, etc. Various communication client applications can be installed on the terminal devices 101, 102, 103, such as web browser applications, search applications, instant messaging tools, etc.

[0049] The terminal devices 101, 102, 103 can be various electronic devices with display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, and desktop computers, etc.

[0050] The server 105 can be a server that provides various services, such as a background server that provides support for the pages displayed on the terminal devices 101, 102, 103.

[0051] It should be noted that the two-dimensional code detection method provided by the embodiments of the present application can be deployed in a terminal without GPU acceleration such as a CPU terminal or a mobile terminal, wherein the terminal without GPU acceleration such as the CPU terminal or the mobile terminal can be used to execute the terminal device 101, 102, 103 mentioned in the embodiments, and accordingly, the two-dimensional code detection apparatus is generally configured in the terminal device 101, 102, 103.

[0052] It should be understood that Figure 1 The number of terminal devices, networks and servers in the above-mentioned embodiments is only illustrative. Any number of terminal devices, networks and servers can be provided according to the needs of implementation.

[0053] Please refer to Figure 2 , Figure 2 The specific implementation of the two-dimensional code detection method is shown.

[0054] It should be noted that the method of the present application is not limited to the flow sequence shown in Figure 2 The method includes the following steps:

[0055] As shown in Figure 2 The method includes the following steps S110-S140.

[0056] S110, obtaining a two-dimensional code region of a to-be-detected image according to a preset first detection positioning model;

[0057] S120, obtaining a Hengzi type square region of a two-dimensional code in the to-be-detected image according to a preset second detection positioning model;

[0058] S130, inputting the Hengzi type square region into a preset first classification model to obtain a first classification result of the Hengzi type square in the two-dimensional code;

[0059] S140, obtaining a non-Hengzi type square region of the two-dimensional code, and inputting the non-Hengzi type square region into a preset second classification model to obtain a second classification result of the non-Hengzi type square in the two-dimensional code.

[0060] The first classification result is data information about whether the square of the character of a Chinese character is defective, and the second classification result is data information about whether the area of the square of the character of a Chinese character on the two-dimensional code is defective. The first detection positioning model is mainly used for positioning detection of the two-dimensional code of the to-be-detected image to obtain the two-dimensional code area of the to-be-detected image. The second detection positioning model is mainly used for positioning detection of the square of the character of a Chinese character of the two-dimensional code to obtain the square of the character of a Chinese character area in the two-dimensional code. The first classification model is mainly used for classifying the square of the character of a Chinese character of the two-dimensional code to determine whether the square of the character of a Chinese character is defective. The second classification model is mainly used for classifying the area other than the square of the character of a Chinese character in the two-dimensional code to determine whether the area other than the square of the character of a Chinese character of the two-dimensional code is defective.

[0061] Specifically, the first detection positioning model, the second detection positioning model, the first classification model, and the second classification model can all be lightweight models in this embodiment, and can all be deployed on a CPU end or a mobile end or other terminal without GPU acceleration, so as to realize quality detection of the two-dimensional code in the image.

[0062] It can be understood that the first detection positioning model and the second detection positioning model can select lightweight models such as NanoDet-Plus, YOLOv5-n, and YOLOX-Nano. The first classification model and the second classification model can select lightweight image classification models such as ShuffleNetV2, MobileNetV3, and Efficientnet-lite. The specific selection can be selected according to actual application, and is not limited in this embodiment.

[0063] In the two-dimensional code detection method provided in the embodiment of the application, the to-be-detected image can be a small circuit breaker image. After the to-be-detected image is subjected to positioning detection in the first detection positioning model, the square of the character of a Chinese character area of the two-dimensional code can be obtained from the two-dimensional code area by the second detection positioning model. Finally, the square of the character of a Chinese character area and the area other than the square of the character of a Chinese character of the two-dimensional code are classified by the first classification model and the second classification model respectively to determine whether the two-dimensional code of the to-be-detected image is defective, thereby avoiding the technical problem that slight defects on the two-dimensional code are not easy to distinguish, and greatly improving the precision and efficiency of two-dimensional code detection.

[0064] In the two-dimensional code detection method provided in the embodiment of the application, the first detection positioning model and the second detection positioning model can both adopt a NanoDet-Plus lightweight model, and the first classification model, the second classification model and the third classification model can all adopt a ShuffleNetV2 lightweight image classification model. The NanoDet-Plus includes three parts of a Backbone network, a PAFPN network and a Prediction network, wherein the Backbone network is mainly used for feature extraction, the PAFPN network is mainly used for feature fusion, and the Prediction network is mainly used for positioning prediction. In the Prediction network, a 5x5 depth separable convolution is adopted, so that the receptive field of the detector is improved and the performance is improved with less parameter amount.

[0065] Specifically, the depth separable convolution process is divided into two steps: depthwise convolution and pointwise convolution. In the depthwise convolution process, one convolution kernel is responsible for one channel, and only one convolution kernel can be used for convolution on one channel. The pointwise convolution can effectively utilize the feature information of different channels at the same spatial position, and the size of the convolution kernel is 1x1xM, where M is the number of channels of the previous layer. At this time, the convolution operation will combine the map in the depth direction in the previous step to generate a new feature map. At the same time, when using the ShuffleNetV2 lightweight image classification model for image classification, based on the lightweight design principle, it is necessary to ensure that the input and output channel numbers are the same, the group convolution is changed to ordinary convolution, and the "Add" operation of the residual module is changed to "Concat" to improve the speed of the model.

[0066] In this embodiment, after the to-be-detected image passes through the Backbone network and the PAFPN network, a feature map of HxWx96 can be generated, and after passing through the Prediction network twice, a feature map with unchanged feature dimension is obtained, and then a 1x1 convolution kernel is used for convolution to generate a feature map of HxWx(R+C), wherein R represents the coordinates of the diagonal points of a single positioning frame, and C represents the number of categories to be predicted. Finally, the feature map of HxWx(R+C) is split into two branches, one branch passes through a sigmoid activation function, dimension transposition, flattening and other operations to obtain (HxW)xC category scores, and the other branch passes through dimension transposition, flattening and other operations to obtain HxWxR coordinates.

[0067] In other embodiments of the application, as shown in Figure 3 Step S110 includes steps S111 and S112.

[0068] S111, generating a feature pyramid of the image to be detected;

[0069] S112, predicting a two-dimensional code region of the image to be detected according to the features of the feature pyramid.

[0070] Specifically, the feature pyramid is formed by multiple feature extractions of the image to be detected, and each layer of the feature pyramid includes features with different resolutions and different semantic strengths, thereby realizing accurate positioning and detection of the two-dimensional code and the characters adjacent to the two-dimensional code.

[0071] In other embodiments of the application, as shown in Figure 4 The step S111 includes sub-steps S1111, S1112 and S1113.

[0072] S1111, performing feature extraction on the image to be detected to obtain a plurality of first feature maps of the image to be detected;

[0073] S1112, performing feature fusion on the plurality of first feature maps according to a first feature fusion module to obtain a plurality of second feature maps;

[0074] S1113, performing feature fusion on the plurality of second feature maps according to a second feature fusion module to obtain the feature pyramid.

[0075] Specifically, after the image to be detected is subjected to feature extraction by the Backbone network, the feature maps can be processed in the PAFPN network using the Ghost-PAN structure, that is, the Ghost-PAN structure uses the GhostBlock (redundancy module) in the GhostNet as a module for feature fusion between multiple layers, and the basic structural unit thereof can be composed of a group of 1x1 convolution and 3x3 Depthwise convolution, thereby greatly reducing the parameter quantity and the calculation amount.

[0076] In the present embodiment, as shown in Figure 5 The image to be detected is a small circuit breaker image, and after the image to be detected is subjected to feature extraction to obtain a plurality of second feature maps, the high-level feature maps with more abstraction and stronger semantics are up-sampled from top to bottom, and at the same time, the features are horizontally connected to the features of the previous layer, and a GhostBlock is used for feature fusion to obtain the plurality of second feature maps. From bottom to top, the features of the bottom layer are down-sampled, and at the same time, the features are horizontally connected to the features of the previous layer, and a GhostBlock is used for feature fusion to obtain the feature pyramid, thereby improving the positioning and detection accuracy of the Prediction network.

[0077] In other embodiments of the application, as shown in Figure 6As shown, before step S120, steps S120a and S120b are further included.

[0078] S120a, acquiring a character region adjacent to the two-dimensional code in the to-be-detected image according to the first detection positioning model;

[0079] S120b, inputting the character region into a preset third classification model to obtain a third classification result of the character adjacent to the two-dimensional code in the to-be-detected image.

[0080] In the embodiment, the first detection positioning model is further used for positioning detection of the character adjacent to the two-dimensional code in the to-be-detected image to acquire the two-dimensional code region of the to-be-detected image, the third classification result is data information of whether the character adjacent to the two-dimensional code has a defect, and the third classification model is mainly used for classification of the character adjacent to the two-dimensional code in the to-be-detected image to determine whether the character has a defect.

[0081] It can be understood that the third classification model can select a light image classification model such as ShuffleNetV2, MobileNetV3, and Efficientnet-lite, and the specific selection can be selected according to actual application, and the embodiment is not limited specifically.

[0082] The embodiment of the application further provides a model training method, which can be executed by the terminal devices 101, 102, and 103 in the Figure 1 , deployed to other terminal devices by the server 105, so as to enable other terminal devices to execute the two-dimensional code detection method, and the method can also be executed by the server 105 and deployed to other terminal devices by the server 105, so as to enable other terminal devices to execute the two-dimensional code detection method.

[0083] As shown in Figure 7 , the method includes the following steps S210-S240.

[0084] S210, acquiring a sample image set collected;

[0085] S220, labeling each sample image in the sample image set to obtain a labeled sample image set;

[0086] S230, pre-processing the labeled sample image set to obtain a first training set and a verification set;

[0087] S240, training the detection network of the two-dimensional code image according to the first training set and the validation set, wherein the detection network comprises a first detection positioning model, a second detection positioning model, a first classification model and a second classification model; the detection positioning category of the first detection positioning model comprises a two-dimensional code of an image, the detection category of the second detection positioning model comprises a square block of the two-dimensional code, the classification category of the first classification model comprises the square block, and the classification category of the second classification model comprises a non-square block in the two-dimensional code.

[0088] In the embodiment, the detection network comprises the first detection positioning model, the second detection positioning model, the first classification model and the second classification model, the first detection positioning model and the second detection positioning model adopt NanoDet-Plus lightweight detection positioning models, and the first classification model and the second classification model adopt ShuffleNetV2 lightweight image classification models. The first detection positioning model and the second detection positioning model increase an auxiliary module (AGM) in the training process, and cooperate with a dynamic soft label assignment strategy (DSLA) to solve the optimal label matching problem in the lightweight model. The auxiliary module can be composed of four 3x3 convolution kernels, uses group normalization (GN) as the Normalize layer, and shares parameters between feature maps of different scales. The auxiliary module consumes less training resources in the training process of the first classification model and the second classification model, and the auxiliary module can be directly removed after the first classification model and the second classification model are completed.

[0089] Specifically, when the first detection positioning model and the second detection positioning model adopt NanoDet-Plus lightweight detection positioning models, the loss function (GFL) can be as follows:

[0090]

[0091] wherein y l , y r represent binary labels (y l < y r ) that the model has, y represents a real label, P yl and P yr represent prediction probability values of the network for the two labels, P yl ≥ 0, P yr ≥ 0, P yl + P yr = 1.

[0092] When the first classification model and the second classification model adopt the ShuffleNetV2 lightweight image classification model, the loss function (Focal Loss, FL) can be as follows:

[0093] FL(p t )=-α t (1-p t ) γ log(p t )

[0094] wherein, α t represents a control class balance parameter, γ represents a control difficult and easy sample balance parameter, and p t represents a predicted probability value.

[0095] In addition, when there is a character adjacent to the two-dimensional code in the image to be detected, the detection network further includes a third classification model, and the detection and positioning categories of the first detection and positioning model further include the character adjacent to the two-dimensional code. The third classification model is used for classifying the character adjacent to the two-dimensional code. The third classification model can adopt the ShuffleNetV2 lightweight image classification model.

[0096] In the embodiment, the sample image set is a small circuit breaker image set, each small circuit breaker image has a two-dimensional code and a character adjacent to the two-dimensional code, and the image set includes an image without defects (OK) and an image with defects (NG). Since each small circuit breaker image has a two-dimensional code and a character adjacent to the two-dimensional code, the two-dimensional code, the character adjacent to the two-dimensional code, and the square block on the two-dimensional code need to be detected and positioned in the embodiment, and therefore the sample image and the two-dimensional code need to be labeled.

[0097] Specifically, the embodiment of the present application can use the Labelimg labeling tool to represent the two-dimensional code and the adjacent character position in the form of a rectangular frame, and name the labeled rectangular frame. After labeling, an XML format labeling file can be automatically generated. The XML file includes the image name, the label name of the labeled object, and the left and right endpoint information of the labeled rectangular frame. At the same time, after labeling the two-dimensional code and the adjacent character in the sample image, the two-dimensional code image needs to be cut out based on the two-dimensional code name and the rectangular frame coordinate information in the sample image and the XML file, and the Labelimg labeling tool L can be used again to represent the square block on the corner of the two-dimensional code in the form of a rectangular frame. The labeled two-dimensional code, the character adjacent to the two-dimensional code, and the square block on the two-dimensional code can be referred to in Figure 8 and Figure 9 .

[0098] In addition, before training each model, a sample image set can be directly collected from the production line and the two-dimensional code and characters in the sample image can be labeled in the Labelimg labeling tool, and the labeling names can be Qrcode and Character respectively. Meanwhile, the two-dimensional code in the sample image is segmented and the square character is labeled on the two-dimensional code, and the labeling name can be Square. After labeling, an XML format labeling file can be automatically generated, and thus the labeling of the sample image set is completed.

[0099] It can be understood that the labeling tool mentioned in the embodiment is not limited to the Labelimg labeling tool, and the Labelme, VOTT, etc. can be used as a substitute.

[0100] Specifically, after the sample image set is labeled, the labeled sample image set can be used to segment the characters adjacent to the two-dimensional code on the labeled sample image, and the square character and the non-square character on the two-dimensional code can be segmented. The labeled sample image set can be divided into a first training set and a validation set in a hierarchical manner at a ratio of 8:2. Finally, the data preprocessing script can be integrated into two preset format files, which can be json files, including train.json and validate.json files.

[0101] In addition, the first training set and the validation set need to be uniformly scaled to a certain size before training the first detection and positioning model, the second detection and positioning model, the first classification model, and the second classification model, and then propagated forward.

[0102] In other embodiments of the application, as shown in Figure 10 Step S240 includes steps S241, S242 and S243.

[0103] S241, according to the first training set, iteratively training the first detection and positioning model, the second detection and positioning model, the first classification model, and the second classification model, and optimizing the hyperparameters of each model based on a genetic algorithm until each model converges;

[0104] S242, according to the validation set, testing each model, and generating a second training set according to the test results and the first training set;

[0105] S243, according to the second training set, iteratively training each model until each model converges.

[0106] In this embodiment, after the first training set is input into the first detection and localization model, the second detection and localization model, the first classification model, the second classification model, and the third classification model, it propagates forward to predict the location information of the QR code and the square within the QR code. Then, the loss function is used, combined with the predicted location information and the annotation information, to calculate the loss value and the accuracy. The weight parameters are then updated slightly along the gradient direction based on the optimizer (such as Adam). The forward and backward propagation processes are repeated and iterated until the optimal weight with the minimum loss value or the highest accuracy is obtained.

[0107] Since a large number of variable hyperparameters are designed during the training of each model, and the different values ​​of these hyperparameters have a significant impact on the loss, accuracy, and recall, it is necessary to select the key hyperparameters that affect each model during the iterative training of the first training set. For example, the model can be trained with accuracy as the objective to obtain a set of optimal hyperparameters. Hyperparameter selection and optimization can be performed using manual adjustment strategies, grid search strategies, random search strategies, Bayesian optimization algorithms, and genetic algorithms. In this embodiment, the genetic algorithm is preferred for optimizing the hyperparameters of each model.

[0108] Specifically, after iteratively training each model on the first training set, the models can be tested based on the obtained optimal weights and validation set. Abnormal samples (samples with localization errors or classification errors, etc.) obtained during the testing process are added to the first training set to form a second training set. Then, the models are iteratively trained again using the second training set. During the iterative training of the models on the second training set, it is not necessary to optimize the hyperparameters of each model again.

[0109] In other embodiments of the invention, such as Figure 11 As shown, after step S240, steps S250 and S260 are also included.

[0110] S250. Based on the preset deployment framework, the trained models are converted to different formats to obtain the converted models.

[0111] S260. Combine the various models after format conversion to write forward inference scripts for detection and classification, and encapsulate the forward inference scripts.

[0112] Specifically, after each model is trained, each model needs to be deployed to a terminal to facilitate the terminal to perform detection and classification of the miniature circuit breaker image. When each model is deployed to the terminal, each model after training is first converted in format, so that the format of each model conforms to the format specified by the pre-set deployment framework, and then a forward inference script for detection and classification can be written according to each model after format conversion, so as to package and encapsulate each model.

[0113] The deployment framework can be NCNN, Caffe, Libtorch, OpenVINO, etc. The deployment framework used in the embodiment is preferably NCNN, which can realize third-party independent operation, cross-platform operation, and leading level in open source framework in terms of CPU operation speed on the mobile phone side. When the deployment framework is NCNN, each model needs to be converted into an ONNX format model, and then the ONNX format model is converted into a model file required by the NCNN framework. The model file can be a Bin file and a Param file.

[0114] In the embodiment, when each model is deployed, it can be deployed in the front end of the Windows system depending on the CPU, for example, a C++ project can be newly created in the Visual Studio development tool (for example, Visual Studio 2019), the NCNN library and the OPENCV vision library are configured, the converted each model is called to write a forward inference script, and then the forward inference script can be packaged into a DLL file for the front end to call.

[0115] In addition, when each model is deployed, it can also be deployed in a mobile terminal device without depending on the CPU, for example, the NCNN and OPENCV library files can be configured in the Android studio development tool, a forward inference script is written, an interface file is written, and the code required for forward inference is called, and then the interface provided by the interface file is called by Java for the mobile terminal to call.

[0116] The embodiment of the application also provides a two-dimensional code detection device, which is used to execute any one of the foregoing two-dimensional code detection methods.

[0117] Specifically, please refer to Figure 12 , Figure 12 is a schematic block diagram of the two-dimensional code detection device provided by the embodiment of the application.

[0118] As Figure 12 shown, the two-dimensional code detection device comprises a first acquisition unit 110, a second acquisition unit 120, a first input unit 130, and a second input unit 140.

[0119] The first obtaining unit 110 is configured to obtain a two-dimensional code region of a to-be-detected image according to a preset first detection positioning model.

[0120] The second obtaining unit 120 is configured to obtain a Hengzi type square region of a two-dimensional code in the to-be-detected image according to a preset second detection positioning model.

[0121] The first input unit 130 is configured to input the Hengzi type square region into a preset first classification model to obtain a first classification result of the Hengzi type square in the two-dimensional code.

[0122] The second input unit 140 is configured to obtain a non-Hengzi type square region of the two-dimensional code and input the non-Hengzi type square region into a preset second classification model to obtain a second classification result of the non-Hengzi type square in the two-dimensional code.

[0123] In other embodiments of the present application, the first obtaining unit 110 comprises a generating unit and a predicting unit.

[0124] The generating unit is configured to generate a feature pyramid of the to-be-detected image, and the predicting unit is configured to predict a two-dimensional code region of the to-be-detected image according to features of the feature pyramid.

[0125] In other embodiments of the present application, the generating unit comprises a feature extraction unit, a first feature fusion unit and a second feature fusion unit.

[0126] The feature extraction unit is configured to perform feature extraction on the to-be-detected image to obtain a plurality of layers of first feature maps of the to-be-detected image, the first feature fusion unit is configured to perform feature fusion on the plurality of layers of first feature maps according to a first feature fusion module to obtain a plurality of layers of second feature maps, and the second feature fusion unit is configured to perform feature fusion on the plurality of layers of second feature maps according to a second feature fusion module to obtain the feature pyramid.

[0127] In other embodiments of the present application, the two-dimensional code detection device further comprises a fourth obtaining unit and a third input unit.

[0128] The fourth obtaining unit is configured to obtain a character region adjacent to the two-dimensional code in the to-be-detected image according to the first detection positioning model.

[0129] The third input unit is configured to input the character region into a preset third classification model to obtain a third classification result of a character adjacent to the two-dimensional code in the to-be-detected image.

[0130] The two-dimensional code detection device provided by the embodiment of the present application is used to perform the above-mentioned operations of obtaining a two-dimensional code region of a to-be-detected image according to a preset first detection positioning model; obtaining a Hengzi type square region of a two-dimensional code in the to-be-detected image according to a preset second detection positioning model; inputting the Hengzi type square region into a preset first classification model to obtain a first classification result of the Hengzi type square in the two-dimensional code; obtaining a non-Hengzi type square region of the two-dimensional code, and inputting the non-Hengzi type square region into a preset second classification model to obtain a second classification result of the non-Hengzi type square in the two-dimensional code.

[0131] It should be noted that the specific implementation process of the two-dimensional code detection device and each unit can be clearly understood by those skilled in the art, and can refer to the corresponding description in the foregoing method embodiment. For the convenience and brevity of description, it will not be repeated here.

[0132] The embodiment of the present application also provides a model training device, which is used to perform any one of the foregoing embodiments of the two-dimensional code detection method.

[0133] Specifically, please refer to Figure 13 , Figure 13 is a schematic block diagram of the model training device provided by the embodiment of the present application.

[0134] As Figure 13 shown, the model training device comprises a third acquisition unit 210, a labeling unit 220, a preprocessing unit 230, and a training unit 240.

[0135] The third acquisition unit 210 is used to acquire a set of collected sample images.

[0136] The labeling unit 220 is used to label each sample image in the set of sample images to obtain a set of labeled sample images.

[0137] The preprocessing unit 230 is used to pre-process the set of labeled sample images to obtain a first training set and a verification set.

[0138] The training unit 240 is used to train a detection network of a two-dimensional code image according to the first training set and the verification set, wherein the detection network comprises a first detection positioning model, a second detection positioning model, a first classification model, and a second classification model; the detection positioning categories of the first detection positioning model comprise a two-dimensional code region, the detection categories of the second detection positioning model comprise a Hengzi type square region of the two-dimensional code, the classification categories of the first classification model comprise the Hengzi type square region, and the classification categories of the second classification model comprise a non-Hengzi type square region in the two-dimensional code.

[0139] In other embodiments of the application, the training unit 240 comprises a first iterative training unit, a testing unit and a second iterative training unit.

[0140] The first iterative training unit is configured to iteratively train the first detection and positioning model, the second detection and positioning model, the first classification model and the second classification model according to the first training set, and optimize the hyperparameters of each model based on a genetic algorithm until each model converges.

[0141] In other embodiments of the application, the model training apparatus further comprises a format conversion unit and a packaging unit.

[0142] The format conversion unit is configured to convert the format of each trained model based on a preset deployment framework to obtain each model after format conversion.

[0143] The packaging unit is configured to write a forward inference script for detection and classification in combination with each model after format conversion, and package the forward inference script.

[0144] The model training apparatus provided in the embodiments of the application is used to perform the above-mentioned steps of acquiring the collected sample image set, labeling each sample image in the sample image set to obtain a labeled sample image set, pre-processing the labeled sample image set to obtain a first training set and a validation set, training a detection network of a two-dimensional code image according to the first training set and the validation set, wherein the detection network comprises a first detection and positioning model, a second detection and positioning model, a first classification model and a second classification model, the detection and positioning category of the first detection and positioning model comprises a two-dimensional code of an image, the detection category of the second detection and positioning model comprises a square block of the two-dimensional code, the classification category of the first classification model comprises the square block, and the classification category of the second classification model comprises a non-square block in the two-dimensional code.

[0145] It should be noted that those skilled in the art can clearly understand the specific implementation process of the above-mentioned model training apparatus and each unit, which can be referred to the corresponding description in the foregoing method embodiments. For the convenience and brevity of description, it will not be repeated here.

[0146] The above-mentioned two-dimensional code detection apparatus or model training apparatus can be realized in the form of a computer program, which can run on an electronic device as shown in Figure 14 .

[0147] Please refer to Figure 14, Figure 14 is a schematic block diagram of an electronic device provided by an embodiment of the present application.

[0148] Referring to Figure 14 The device 500 includes a processor 502, a memory, and a network interface 505 connected through a system bus 501, wherein the memory can include a storage medium 503 and an internal memory 504.

[0149] The storage medium 503 can store an operating system 5031 and a computer program 5032. The computer program 5032, when executed, can cause the processor 502 to perform a two-dimensional code detection method or a model training method.

[0150] The processor 502 is configured to provide computing and control capabilities to support the operation of the entire device 500.

[0151] The internal memory 504 provides an environment for the execution of the computer program 5032 in the non-volatile storage medium 503, and the computer program 5032, when executed by the processor 502, can cause the processor 502 to perform a two-dimensional code detection method or a model training method.

[0152] The network interface 505 is configured to perform network communication, such as providing transmission of data information, etc. Those skilled in the art can understand that Figure 14 The structure shown in the figure is only a block diagram of part of the structure related to the present application scheme, and does not constitute a limitation on the device 500 to which the present application scheme is applied. The specific device 500 can include more or fewer components than those shown in the figure, or combine certain components, or have a different component arrangement.

[0153] The processor 502 is configured to run the computer program 5032 stored in the memory to implement the following functions: obtaining a two-dimensional code region of a to-be-detected image according to a preset first detection positioning model; obtaining a Hengzi type square region of a two-dimensional code in the to-be-detected image according to a preset second detection positioning model; inputting the Hengzi type square region into a preset first classification model to obtain a first classification result of the Hengzi type square in the two-dimensional code; obtaining a non-Hengzi type square region of the two-dimensional code, and inputting the non-Hengzi type square region into a preset second classification model to obtain a second classification result of the non-Hengzi type square in the two-dimensional code.

[0154] The processor 502 is configured to run the computer program 5032 stored in the memory to perform the following functions: obtaining a set of collected sample images; labeling each sample image in the set of sample images to obtain a set of labeled sample images; pre-processing the set of labeled sample images to obtain a first training set and a validation set; training a detection network of a two-dimensional code image according to the first training set and the validation set, wherein the detection network comprises a first detection and positioning model, a second detection and positioning model, a first classification model and a second classification model; the detection and positioning categories of the first detection and positioning model include a two-dimensional code of an image, the detection categories of the second detection and positioning model include a square block of the two-dimensional code, the classification categories of the first classification model include the square block, and the classification categories of the second classification model include non-square blocks in the two-dimensional code.

[0155] Those skilled in the art can understand that the embodiments of the device 500 shown in the specification do not constitute a limitation on the specific structure of the device 500, and in other embodiments, the device 500 can include more or fewer components than shown, or combine certain components, or different component arrangements. For example, in some embodiments, the device 500 can only include the memory and the processor 502, and in such embodiments, the structure and functions of the memory and the processor 502 are consistent with the embodiments shown in the specification, and will not be repeated here. Figure 14 Those skilled in the art can understand that the embodiments of the device 500 shown in the specification do not constitute a limitation on the specific structure of the device 500, and in other embodiments, the device 500 can include more or fewer components than shown, or combine certain components, or different component arrangements. For example, in some embodiments, the device 500 can only include the memory and the processor 502, and in such embodiments, the structure and functions of the memory and the processor 502 are consistent with the embodiments shown in the specification, and will not be repeated here. Figure 14 Those skilled in the art can understand that the embodiments of the device 500 shown in the specification do not constitute a limitation on the specific structure of the device 500, and in other embodiments, the device 500 can include more or fewer components than shown, or combine certain components, or different component arrangements. For example, in some embodiments, the device 500 can only include the memory and the processor 502, and in such embodiments, the structure and functions of the memory and the processor 502 are consistent with the embodiments shown in the specification, and will not be repeated here.

[0156] It should be understood that in the embodiments of the present application, the processor 502 can be a central processing unit (CPU), and the processor 502 can also be other general-purpose processors 502, digital signal processors 502 (DSP), application specific integrated circuits (ASIC), ready programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor 502 can be a microprocessor 502 or any conventional processor 502, etc.

[0157] In another embodiment of the present application, a computer storage medium is provided. The storage medium can be a non-volatile computer readable storage medium or a volatile storage medium. The storage medium stores a computer program 5032, wherein the computer program 5032, when executed by the processor 502, implements the following steps: obtaining a two-dimensional code region of a to-be-detected image according to a preset first detection positioning model; obtaining a Hengzi type square region of a two-dimensional code in the to-be-detected image according to a preset second detection positioning model; inputting the Hengzi type square region into a preset first classification model to obtain a first classification result of the Hengzi type square in the two-dimensional code; obtaining a non-Hengzi type square region of the two-dimensional code, and inputting the non-Hengzi type square region into a preset second classification model to obtain a second classification result of the non-Hengzi type square in the two-dimensional code.

[0158] Meanwhile, the storage medium stores a computer program 5032, wherein the computer program 5032, when executed by the processor 502, can also implement the following steps: obtaining a set of collected sample images; labeling each sample image in the set of sample images to obtain a set of labeled sample images; pre-processing the set of labeled sample images to obtain a first training set and a validation set; training a two-dimensional code image detection network according to the first training set and the validation set, wherein the detection network includes a first detection positioning model, a second detection positioning model, a first classification model, and a second classification model; the detection positioning categories of the first detection positioning model include two-dimensional codes of images, the detection categories of the second detection positioning model include Hengzi type squares of the two-dimensional codes, the classification categories of the first classification model include the Hengzi type squares, and the classification categories of the second classification model include non-Hengzi type squares in the two-dimensional codes.

[0159] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices, apparatuses and units can refer to the corresponding processes in the foregoing method embodiments, which will not be described here. Those of ordinary skill in the art can realize that the units and algorithm steps of the examples described in combination with the embodiments disclosed herein can be realized by electronic hardware, computer software or a combination of both. In order to clearly illustrate the interchangeability of hardware and software, the components and steps of the examples have been described in general terms in the foregoing description. Whether the functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0160] In several embodiments of the present application, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the above-described apparatus embodiments are merely illustrative, and the division of the units is merely logical function division. Actual implementation can have another division, or units with the same function can be combined into one unit, such as multiple units or components can be combined or integrated into another system, or some features can be omitted or not implemented. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interfaces, devices, or units, and can also be electrical, mechanical, or other forms of connection.

[0161] The units described as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, i.e., can be located in one place or distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the embodiments of the present application.

[0162] In addition, the functional units in each embodiment of the present application can be integrated into one processing unit, or each unit can exist physically, or two or more units can be integrated into one unit. The integrated unit can be implemented in the form of hardware or in the form of a software functional unit.

[0163] The integrated unit, if implemented in the form of a software functional unit and sold or used as an independent product, can be stored in a storage medium. Based on this understanding, the technical solutions of the present application essentially or the part that contributes to the prior art, or all or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes a plurality of instructions for causing an apparatus 500 (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in the embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a magnetic disk or an optical disk, and various storage media that can store program codes.

[0164] The above is merely a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of various equivalent modifications or replacements within the technical range disclosed by the present application, and these modifications or replacements should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A QR code detection method, characterized in that, including: obtaining the QR code area of the image to be detected according to a preset first detection and positioning model; obtaining the square area of the QR code in the image to be detected according to a preset second detection and positioning model; inputting the square area into a preset first classification model to obtain a first classification result of the square in the QR code; obtaining the non-square area of the QR code and inputting the non-square area into a preset second classification model to obtain a second classification result of the non-square in the QR code.

2. The QR code detection method according to claim 1, characterized in that, The obtaining the QR code area of the image to be detected according to a preset first detection and positioning model includes: generating a feature pyramid of the image to be detected; predicting the QR code area of the image to be detected according to the features of the feature pyramid.

3. The QR code detection method according to claim 2, characterized in that, The generating a feature pyramid of the image to be detected includes: performing feature extraction on the image to be detected to obtain multiple layers of first feature maps of the image to be detected; performing feature fusion on the multiple layers of first feature maps according to a first feature fusion module to obtain multiple layers of second feature maps; performing feature fusion on the multiple layers of second feature maps according to a second feature fusion module to obtain the feature pyramid.

4. The QR code detection method according to claim 1, characterized in that, Before obtaining the square area of the QR code in the image to be detected according to a preset second detection and positioning model, it further includes: obtaining the character area adjacent to the QR code in the image to be detected according to the first detection and positioning model; inputting the character area into a preset third classification model to obtain a third classification result of the character adjacent to the QR code in the image to be detected.

5. A model training method, characterized in that, including: obtaining a collected sample image set; annotating each sample image in the sample image set to obtain an annotated sample image set; performing preprocessing on the annotated sample image set to obtain a first training set and a validation set; training a detection network for QR code images according to the first training set and the validation set, where the detection network includes a first detection and positioning model, a second detection and positioning model, a first classification model, and a second classification model; the detection and positioning category of the first detection and positioning model includes the QR code of the image, the detection category of the second detection and positioning model includes the square of the QR code, the first classification model is used to determine whether there is a defect in the square, and the second classification model is used to determine whether there is a defect in the non-square of the QR code.

6. The model training method according to claim 5, characterized in that, The training the detection network for QR code images according to the first training set and the validation set includes: performing iterative training on the first detection and positioning model, the second detection and positioning model, the first classification model, and the second classification model according to the first training set, and optimizing the hyperparameters of each model based on a genetic algorithm until each model converges; testing each model according to the validation set, and generating a second training set according to the test results and the first training set; performing iterative training on each model according to the second training set until each model converges.

7. The model training method according to claim 5, characterized in that, After training the first detection and localization model, the second detection and localization model, the first classification model, and the second classification model according to the first training set and the validation set, the following steps are further included: Based on a preset deployment framework, perform format conversion on each trained model to obtain the models after format conversion; Write a forward inference script for detection and classification in combination with the models after format conversion, and encapsulate the forward inference script.

8. The model training method according to claim 5, characterized in that, The detection network further includes a third classification model. The detection and localization categories of the first detection and localization model further include a character region adjacent to the two-dimensional code. The third classification model is used to determine whether there are defects in the character region.

9. A QR code detection device, characterized in that, It includes: A first acquisition unit for acquiring the two-dimensional code region of the image to be detected according to a preset first detection and localization model; A second acquisition unit for acquiring the square frame region of the two-dimensional code in the image to be detected according to a preset second detection and localization model; A first input unit for inputting the square frame region into a preset first classification model to obtain a first classification result of the square frame in the two-dimensional code; A second input unit for acquiring the non-square frame region of the two-dimensional code and inputting the non-square frame region into a preset second classification model to obtain a second classification result of the non-square frame in the two-dimensional code.

10. A model training device, characterized in that, It includes: A third acquisition unit for acquiring a collected sample image set; A labeling unit for labeling each sample image in the sample image set to obtain a labeled sample image set; A preprocessing unit for preprocessing the labeled sample image set to obtain a first training set and a validation set; A training unit for training a detection network of two-dimensional code images according to the first training set and the validation set. The detection network includes a first detection and localization model, a second detection and localization model, a first classification model, and a second classification model. The detection and localization categories of the first detection and localization model include the two-dimensional code region. The detection categories of the second detection and localization model include the square frame region of the two-dimensional code. The first classification model is used to determine whether there are defects in the square frame region. The second classification model is used to determine whether there are defects in the non-square frame region of the two-dimensional code.

11. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the two-dimensional code detection method according to any one of claims 1 to 4 or the model training method according to any one of claims 5 to 8.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program. When the computer program is executed by the processor, the processor is caused to execute the two-dimensional code detection method according to any one of claims 1 to 4 or the model training method according to any one of claims 5 to 8.

Images