Multi-code decoding method, information matching method and product information management system

Abstract

The invention provides a multi-code decoding method, an information matching method and a product information management system, and the method comprises the steps: carrying out the first processing of a product image, and obtaining a plurality of image blocks with preset sizes; an improved YOLOv8 multi-target detection model and a TensorRT reasoning acceleration technology are adopted to carry out rapid detection on each image block; eliminating repeated codes in all the image blocks, and cutting each residual code from the image block to which the residual code belongs to obtain a code block of the residual code; and for each residual code, when the confidence coefficient of the residual code exceeds a preset threshold value, calling a plurality of decoding engines according to a priority sequence to analyze the code block of the residual code, judging whether any decoding engine outputs decoding information or not, and if so, determining that decoding succeeds. The invention provides a multi-code decoding method, an information matching method and a product information management system, which are used for solving the problems of management mode lag, high management cost, low management efficiency, asymmetric product information, management errors and the like in the existing product information management.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image recognition, and in particular to a multi-code decoding method, an information matching method and a product information management system. BACKGROUND

[0002] Enterprises are accelerating the pace of digital transformation, striving to build digital factories and intelligent production lines. Especially at the workshop level, efficient management of products and their information is the core of production and operation, directly related to the operational efficiency and economic benefits of enterprises.

[0003] However, most current enterprises still have many problems in product information management. On the one hand, the management method is lagging behind, relying on manual recording, random sampling and other traditional means, lacking an efficient and systematic information collection mechanism, making it difficult to track product information in real time, with high management costs and low efficiency; on the other hand, the automation level of equipment is uneven, further exacerbating the problem of product information asymmetry and management failure. SUMMARY

[0004] The present application provides a multi-code decoding method, an information matching method and a product information management system to solve the problems of lagging management method, high management cost, low management efficiency, product information asymmetry and management failure in existing product information management.

[0005] In a first aspect, the present application provides a multi-code decoding method, comprising:

[0006] performing first processing on a product image to obtain a plurality of image blocks of a predetermined size, the product image including a plurality of codes, the first processing including deblurring, HDR light recovery, compression, overlapping block and edge padding;

[0007] using an improved YOLOv8 multi-target detection model and TensorRT inference acceleration technology to respectively perform rapid detection on each image block, identify the codes in each image block, and output the detection coordinates, type number and confidence of the codes, the improved YOLOv8 multi-target detection model including Mosaic data enhancement, a backbone network, a neck network and a head network, the backbone network using a CSPDarknet architecture, a Transformer and a CBAM module, the neck network using a BiFPN structure and a self-attention layer, the head network using a CIoU loss function, the improved YOLOv8 multi-target detection model being trained using a PyTorch framework;

[0008] eliminating duplicate codes in all image blocks, cropping each remaining code from its corresponding image block to obtain a code block, and performing second processing on each code block, the second processing including grayscale and binarization;

[0009] For each remaining code, when its confidence exceeds a preset threshold, a plurality of decoding engines are called in priority order to parse its code block, and it is determined whether there is any decoding engine output decoding information, if yes, decoding is successful, if not, the resolution of the code block is adjusted to a preset size, and the plurality of decoding engines are called again in the priority order to parse the adjusted code block, and it is continuously determined whether there is any decoding engine output information, if yes, decoding is successful, if not, decoding fails;

[0010] For each code successfully decoded, a range box and decoding information are labeled on the product image to obtain a labeled image;

[0011] The labeled image and the detection coordinates and decoding information of all codes successfully decoded are output.

[0012] Optionally, the code includes a two-dimensional code and a bar code.

[0013] Optionally, the deblurring is implemented by a DeblurGAN-v2 improved network, the DeblurGAN-v2 improved network adopts an improved loss function, the improved loss function adds a structure perception loss, the structure perception loss includes a perception loss, an edge gradient loss and a module proportion loss, and a formula of the improved loss function is as follows:

[0014] L total =L adv +λ1L percep +λ2L edge +λ3L ratio

[0015] Wherein, L adv =αL local +βL global is an adversarial loss, and α and β are used to balance the control of the dual-scale discriminator on the local details and the global structure to generate the authenticity of the image, L local is a local area authenticity loss, L global is a global structure authenticity loss, λ1, λ2 and λ3 are used to quantify the constraint strength of three physical structures of texture, edge and proportion, L percep is the perception loss, L edge is the edge gradient loss, and L ratio is the module proportion loss.

[0016] Optionally, the overlapping block includes cutting the product image into image blocks according to the preset size, and there is a preset width of overlapping area between adjacent image blocks.

[0017] Optionally, the removing of the repeated codes in all image blocks includes:

[0018] convert the detection coordinates of each code into its coordinates in the product image, and calculate the intersection-over-union between the each code and other codes after conversion;

[0019] determine whether there is a case that the intersection-over-union between two codes is greater than a first threshold value, and if so, eliminate the code with smaller area of the two codes.

[0020] Optionally, the cutting out of each remaining code from the image block to which it belongs to obtain its code block comprises:

[0021] based on the detection coordinates of each remaining code, extending outward by a preset pixel as its code block size, and cutting out its code block from the image block to which it belongs according to the detection coordinates of each remaining code and its code block size.

[0022] The above scheme can efficiently and accurately identify multiple small size codes in a super high resolution image in parallel, suppress complex working condition interference, and be compatible with multiple formats of codes, thereby providing accurate information acquisition and management support for industrial production, and solving the problems of low efficiency, easy error, and repetitive labor in the traditional manual code scanning mode.

[0023] In a second aspect, the present application provides an information matching method, according to any of the above methods, comprising:

[0024] determining whether the intersection-over-union between the detection coordinates of each code successfully decoded and its standard coordinates exceeds a second threshold value, and if so, determining its bit number according to its standard coordinates, the relationship between the standard coordinates and the bit number being pre-configured;

[0025] determining whether the number of all codes successfully decoded is equal to a preset number, and if not, determining an unrecognized code according to the labeled image, manually decoding the unrecognized code to obtain its decoding information, and manually determining the bit number of the unrecognized code;

[0026] outputting the labeled image, the detection coordinates of all codes successfully decoded, and the decoding information and bit number of all codes.

[0027] The above scheme matches the code and the bit number through the detection coordinates and the standard coordinates, realizes the automatic binding of the bit number and the product information, facilitates tracking management, and improves production efficiency; decodes the unrecognized code, ensures the completeness and accuracy of the product information, and realizes the complete closed-loop collection of the product information.

[0028] In a third aspect, the present application provides a product information management system, comprising a client and a server, the client comprising a data acquisition module, a bit number matching module, an information supplementary scanning module, and an information output module, and the server comprising a first processing module, a multi-code detection module, a second processing module, a multi-code decoding module, an image labeling module, and a decoding output module, wherein:

[0029] The data collection module is configured to collect product images and send the product images to the server, wherein the product images include multiple codes;

[0030] The first processing module is configured to perform first processing on the product images to obtain multiple image blocks of a preset size, wherein the first processing includes deblurring, HDR light recovery, compression, overlapping blocking, and edge padding;

[0031] The multi-code detection module is configured to use an improved YOLOv8 multi-object detection model and a TensorRT inference acceleration technology to perform rapid detection on each image block, identify the codes in each image block, and output the detection coordinates, type number, and confidence of the codes, wherein the improved YOLOv8 multi-object detection model includes Mosaic data enhancement, a backbone network, a neck network, and a head network, the backbone network uses a CSPDarknet architecture, a Transformer, and a CBAM module, the neck network uses a BiFPN structure and a self-attention layer, the head network uses a CIoU loss function, and the improved YOLOv8 multi-object detection model is trained using a PyTorch framework;

[0032] The second processing module is configured to remove duplicate codes in all image blocks, crop each remaining code from the image block to which it belongs to obtain a code block, and perform second processing on each code block, wherein the second processing includes grayscale and binarization;

[0033] The multi-code decoding module is configured to, for each remaining code, when the confidence thereof exceeds a preset threshold, call multiple decoding engines in a priority order to parse the code block thereof, determine whether any decoding engine outputs decoding information, and if so, decode successfully, and if not, adjust the resolution of the code block to a preset size, call the multiple decoding engines again in the priority order to parse the adjusted code block, continue to determine whether any decoding engine outputs information, and if so, decode successfully, and if not, decode unsuccessfully;

[0034] The image labeling module is configured to, for each code that decodes successfully, label a range frame and decoding information on the product image to obtain a labeled image;

[0035] The decoding output module is configured to output the labeled image and the detection coordinates and decoding information of all codes that decode successfully to the client;

[0036] The bit number matching module is configured to determine whether the intersection over union between the detection coordinates of each code that decodes successfully and the standard coordinates thereof exceeds a second threshold, and if so, determine the bit number of the code according to the standard coordinates, wherein the relationship between the standard coordinates and the bit number is preconfigured.

[0037] The information supplementing module is configured to determine whether the number of all codes successfully decoded is equal to a preset number, and if not, determine an unrecognized code according to the marked image, manually decode the unrecognized code to obtain decoding information of the unrecognized code, and artificially determine a bit number of the unrecognized code.

[0038] The information output module is configured to output the marked image, the detection coordinates of all codes successfully decoded, and the decoding information and the bit number of all codes.

[0039] The above scheme constructs a product information acquisition and management system integrating detection, decoding and management, realizes batch, non-contact and non-intervention product information acquisition, forms a closed-loop system from image acquisition, information recognition, data verification to online management, solves problems existing in the prior product information management, such as lagging management mode, high management cost, low management efficiency, product information asymmetry and management errors, and provides a high-reliability information solution for manufacturing enterprises.

[0040] In a fourth aspect, the present application provides a computing device, comprising:

[0041] a memory for storing program instructions;

[0042] a processor for invoking the program instructions stored in the memory to perform any of the above methods according to the obtained program execution.

[0043] In a fifth aspect, the present application provides a computer-readable non-volatile storage medium comprising computer-readable instructions, which, when read and executed by a computer, cause the computer to perform any of the above methods. BRIEF DESCRIPTION OF DRAWINGS

[0044] Figure 1 A flowchart of a multi-code decoding method provided by an embodiment of the present application;

[0045] Figure 2 An architecture diagram of a DeblurGAN-v2 network provided by an embodiment of the present application;

[0046] Figure 3 Architecture diagrams of an improved YOLOv8 multi-target detection model and an original YOLOv8 model provided by an embodiment of the present application;

[0047] Figure 4 A structure diagram of a product information management system provided by an embodiment of the present application. DETAILED DESCRIPTION

[0048] In order to make the objects, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work belong to the protection scope of the present application.

[0049] It should be noted that the terms "first", "second", etc. involved in the documents of the present application are used to distinguish similar objects, and do not necessarily describe a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0050] Figure 1 The flow of a multi-code decoding method provided by an embodiment of the present application is shown, which includes:

[0051] S101, a first processing is performed on the product image to obtain a plurality of image blocks of a preset size.

[0052] Specifically, the product image includes a plurality of codes.

[0053] In an example, the code includes a two-dimensional code and a bar code.

[0054] Since the size of the YOLOv8 detection picture is fixed at 640x640, in an example, the preset size is 640x640 pixels.

[0055] Specifically, the first processing includes deblurring, HDR light recovery, compression, overlapping block and edge padding to improve the image quality before recognition and the code recognition success rate.

[0056] In the direction of image deblurring, the DeblurGAN series method is based on a generative adversarial network, which effectively restores the motion blurred image through residual blocks, feature pyramids and other structures, and has strong adaptability and stable performance.

[0057] In an example, the deblurring is implemented by a DeblurGAN-v2 improved network, the DeblurGAN-v2 improved network adopts an improved loss function, and a structure-aware perceptual loss is added in the improved loss function, the structure-aware perceptual loss includes a perceptual loss, an edge gradient loss and a module proportion loss, and a formula of the improved loss function is as follows:

[0058] L total =L adv +λ1L percep +λ2L edge +λ3L ratio

[0059] Wherein, L adv =αL local +βL global is an adversarial loss, and alpha and beta are respectively used to balance the local details and global structure control generated by the dual-scale discriminator to balance the reality of the image, increasing alpha will make the generator pay more attention to the pixel-level reality of the local area (such as a single two-dimensional code module), avoid blur or noise in the module, but may cause distortion of the global structure (such as the position relationship between modules), increasing beta will force the generator to maintain the overall geometric structure of the two-dimensional code, such as the relative position of the positioning pattern and the module arrangement rule, but may cause excessive smoothing of local details (such as module edges), L local is a reality loss of a local area, L global is a reality loss of a global structure, λ1, λ2 and λ3 are respectively used to quantify the constraint strength of texture, edge and proportion of three physical structures to preserve structured features, L percep is a perceptual loss, L edge is an edge gradient loss, and L ratio is a module proportion loss.

[0060] Figure 2 The architecture of the DeblurGAN-v2 network is shown, which is based on a generative adversarial network (GAN) framework, and combines a feature pyramid network (FPN) and a dual-scale discriminator structure, which significantly improves the recovery ability of detail edge information, the architecture of the DeblurGAN-v2 improved network is basically the same as that of the DeblurGAN-v2 network, the difference is that the structure-aware perceptual loss is added in the loss function, so that the generated image is more suitable for subsequent identification.

[0061] DeblurGAN-v2 improves the network to adapt to the image reconstruction strategy of the industrial environment, especially for the code blur problem caused by factors such as jitter, motion blur, and imaging degradation in the industrial on-site code scanning process, to realize fast and low-loss deblurring processing of images. The introduction of DeblurGAN-v2 improved network for blurred image recovery not only improves the code clarity and edge distinguishability, but also further improves the resolution of blurred images. The recovery accuracy of DeblurGAN-v2 improved network when processing blurred industrial image dataset is about 18% higher than that of traditional deblurring methods, which makes the code recognition rate of blurred images increase to more than 85%, with real-time and practicality.

[0062] In an example, HDR light recovery, overlapping partitioning, and edge padding are implemented through OpenCV.

[0063] HDR light recovery improves the recognizability of the code in scenes such as reflection, mirror material, and uneven lighting.

[0064] In an example, overlapping partitioning includes cutting product images into image blocks with a preset size and having a preset width of overlapping area between adjacent image blocks.

[0065] Overlapping partitioning is used to ensure the integrity of the code, avoid splitting the same code into different image blocks, and each image block does not have the complete state of the code, thereby causing recognition omission and failure.

[0066] The preset width is determined according to the size of the two-dimensional code and the bar code. In an example, since the size of the two-dimensional code is less than 200x200 pixels, the preset width is 200 pixels. When the size of the bar code is larger than 200x200 pixels, the preset width can be adjusted according to the size of the bar code.

[0067] For example, when the preset size is 640x640 pixels and the preset width is 200 pixels, the overlapping area between the left and right adjacent image blocks is 200x640 pixels, and the overlapping area between the upper and lower adjacent image blocks is 640x200 pixels.

[0068] When overlapping partitioning, if the size of the edge image block is less than 640x640 pixels, edge padding is needed, which can be filled with white.

[0069] In an example, compression is implemented through the Deflate compression algorithm.

[0070] Compressing and overlapping partitioning the image reduces memory consumption and speeds up image processing, which can ensure fast response and efficient processing of high-resolution (such as 10240x5120 pixels) industrial images.

[0071] S102, using the improved YOLOv8 multi-object detection model and TensorRT inference acceleration technology to detect each image block quickly, identify the code in each image block, and output the detection coordinates, type number and confidence of the code.

[0072] Since the YOLO series algorithm was proposed in YOLOv1, it has been widely used in industrial image recognition scenarios due to its end-to-end structure and high-speed detection capability. YOLOv4 introduces Mosaic enhancement and CSPDarknet backbone network, and achieves excellent results on public datasets such as COCO. YOLOv8 further integrates Transformer structure and attention mechanism, achieving a good balance between precision and speed, and has industrial deployment value.

[0073] Specifically, the improved YOLOv8 multi-object detection model includes Mosaic data enhancement, backbone network, neck network and head network. The backbone network uses CSPDarknet architecture, Transformer and CBAM module, the neck network uses BiFPN structure and self-attention layer, and the head network uses CIoU loss function. The improved YOLOv8 multi-object detection model is trained using the PyTorch framework.

[0074] Figure 3 The architecture comparison of the improved YOLOv8 multi-object detection model and the original YOLOv8 model is shown, and the specific improvements include:

[0075] The backbone network is based on the CSPDarknet architecture, which is composed of multiple CSP modules and convolutional layers. The Transformer can be embedded in the end stage of the backbone network. After the CSPDarknet architecture extracts the basic features, the global geometric structure of the code is modeled through the Transformer, and then the feature saliency is enhanced through the CBAM module;

[0076] The neck network uses the BiFPN (Bi-directional Feature Pyramid Network) structure for multi-scale feature fusion to improve the small target detection accuracy. The self-attention layer in the neck network is a lightweight design of the Transformer, which is different from the complete Transformer in the backbone network. The purpose is to optimize the efficiency of multi-scale feature fusion;

[0077] Fusion of Mosaic data enhancement and CIoU loss function in the head network improves the training robustness.

[0078] The improved YOLOv8 multi-object detection model is suitable for multi-code detection in industrial scenarios, and improves the accuracy of multi-code positioning and classification (divided into two-dimensional code and barcode). It faces the problem of multi-code positioning and classification in high-density and complex background, and has obvious frontier and practicality.

[0079] In view of the characteristics of small targets in high-resolution industrial images, such as large number of small targets and blurred boundaries, an improved YOLOv8 target detection model is designed, and a CBAM module and a BiFPN structure are introduced to improve the detection accuracy and concurrent recognition ability of small size codes. The model can accurately identify about 40 codes in product images, with an identification accuracy of more than 95%, ensuring complete detection of high-density targets.

[0080] The improved YOLOv8 multi-target detection model is trained using the PyTorch framework and exported as an ONNX format model file for TensorRT inference acceleration technology for compilation and optimization, thereby improving the inference speed.

[0081] TensorRT inference acceleration technology is a high-performance deep learning inference optimization engine developed by NVIDIA, designed for GPU platforms to speed up the operation process. TensorRT inference acceleration technology realizes high-performance inference in a GPU environment, and can achieve a total time of ≤5 seconds per image from image input to code recognition, ensuring fast response and adapting to real-time industrial production environment.

[0082] The improved YOLOv8 multi-target detection model uses the PyTorch framework and utilizes GPU for parallel processing, improving computational efficiency and processing efficiency, optimizing memory utilization, and greatly shortening target detection time. Combined with the improved YOLOv8 multi-target detection model and TensorRT inference acceleration technology, efficient and accurate multi-code positioning and classification are realized.

[0083] Note that there can be multiple codes identified in each image block.

[0084] In one example, the detection coordinates include the coordinates of the upper left corner and the lower right corner in each image block.

[0085] In one example, the type number is used to distinguish between two-dimensional codes and barcodes.

[0086] S103, remove all duplicate codes in the image blocks, crop each remaining code from its image block to obtain its code block, and perform a second processing on each code block.

[0087] In one example, removing all duplicate codes in the image blocks includes:

[0088] Convert the detection coordinates of each code into its coordinates in the product image, and calculate the intersection over union between each code and other codes after conversion;

[0089] Determine whether there is a case where the intersection over union between two codes is greater than a first threshold value, and if so, remove the smaller code of the two codes.

[0090] It is noted that the intersection-over-union is an index for measuring the degree of coincidence of two rectangular frames, and the range is between 0 and 1, and the greater the value, the more the overlapping part of the two codes.

[0091] In an example, the first threshold value is 0.7.

[0092] In an example, each remaining code is cropped from the image block to which it belongs to obtain its code block, including:

[0093] Based on the detection coordinates of each remaining code, a preset pixel is expanded outward as the code block size of the remaining code, and the code block of the remaining code is cropped from the image block to which it belongs according to the detection coordinates of the remaining code and the code block size of the remaining code.

[0094] In an example, the preset pixel is 10 pixels, and the outward expansion of 10 pixels means that the detection coordinates of each remaining code are expanded outward by 10 pixels in the horizontal and vertical directions.

[0095] Specifically, the second processing includes graying and binarization.

[0096] In an example, the graying and binarization are implemented by OpenCV.

[0097] S104, for each remaining code, when the confidence thereof exceeds a preset threshold value, a plurality of decoding engines are called in priority order to parse the code block thereof, to determine whether there is any decoding engine outputting decoding information, if yes, the decoding is successful, if not, the resolution of the code block is adjusted to a preset size, and the plurality of decoding engines are called again in priority order to parse the adjusted code block, to continue to determine whether there is any decoding engine outputting information, if yes, the decoding is successful, if not, the decoding fails.

[0098] In an example, the plurality of decoding engines include WeChatCV, ZBar and Libdmtx.

[0099] The plurality of decoding engines ensure the compatibility of different two-dimensional codes (such as QR Code, DataMatrix, PDF417, etc.) and bar codes, improve the fault tolerance and recognition success rate, and meet the diversified needs of industrial production.

[0100] The decoding information is a string of characters in a specific form obtained by decoding, which represents product information, the first 8 bits of which represent the material code of the zero device, and the remaining part represents the serial number (SN number) of the zero device.

[0101] During decoding, Python multi-threading is used and the CPU is utilized for parallel processing, realizing parallel computing, so that multiple codes are decoded at the same time, reducing the overall processing delay and shortening the running time.

[0102] S105, label the range frame and decoding information of each code on the product image for each decoding success, and obtain a labeled image.

[0103] In an example, the labeling is implemented by the function functions cv2.rectangle and cv2.putText of OpenCV.

[0104] According to the labeling information in the labeled image, the recognition situation of the code in the product image can be more intuitively viewed, the position of the missed scanning code is found, and the missed scanning is supplemented in time.

[0105] S106, output the labeled image and the detection coordinates and decoding information of all codes decoding successfully.

[0106] Since the JSON format has been standardized, and is good in compatibility, easy to parse and more flexible, in an example, the labeled image and the detection coordinates and decoding information of all codes decoding successfully are output in the JSON format.

[0107] Through statistical analysis of actual test experiments, it is obtained that the multi-code recognition accuracy reaches ≥95%, the missed scanning rate is not more than 3%, and the multi-code misrecognition rate is controlled to be ≤3%, which ensures efficient and accurate recognition of multi-codes in complex industrial environments, and correct results can still be stably output even when different illuminations, background interference and blur influence are considered.

[0108] The above scheme can efficiently and accurately recognize multiple small size codes in a super high resolution image in parallel, suppress complex working condition interference and be compatible with multiple formats of codes, and provide accurate information acquisition and management support for industrial production, and solve the problems of low efficiency, easy error, repeated labor and the like existing in the traditional manual code scanning mode.

[0109] Based on the same inventive concept, an information matching method provided by an embodiment of the application is provided according to the above method, and the method comprises:

[0110] S201, determine whether the intersection-over-union between the detection coordinates of each code decoding successfully and the standard coordinates thereof exceeds a second threshold value, and if yes, determine the position number thereof according to the standard coordinates thereof.

[0111] Specifically, the relationship between the standard coordinates and the position number is preconfigured.

[0112] The standard coordinates are the positions of the two-dimensional codes on the zero parts, and the position number is the position number of the zero parts. In the same batch of products, the positions of the zero parts in the products are fixed, and the two-dimensional codes (or bar codes) are printed on the fixed positions of the zero parts.

[0113] The matching of the code and the position number through the detection coordinates and the standard coordinates realizes the automatic binding of the position number and the product information, facilitates the tracking management, and improves the production efficiency.

[0114] S202, judging whether the number of all codes successfully decoded is equal to the preset number, if not, determining the unrecognized code according to the labeled image, manually decoding the unrecognized code to obtain the decoding information of the unrecognized code, and manually determining the position number of the unrecognized code.

[0115] The number of codes on the single side of the fixed model product is fixed, and the number is used to judge whether all codes on the product image are recognized.

[0116] Decoding the unrecognized code ensures the integrity and accuracy of the product information, and realizes the complete closed-loop collection of the product information.

[0117] In an example, manually decoding the unrecognized code to obtain the decoding information of the unrecognized code is realized by a code scanning gun.

[0118] S203, outputting the labeled image, the detection coordinates of all codes successfully decoded, and the decoding information and position number of all codes.

[0119] In an example, the labeled image, the detection coordinates of all codes successfully decoded, and the decoding information and position number of all codes are output in a JSON format, so as to facilitate subsequent storage, analysis and display.

[0120] Based on the same inventive concept, Figure 4 The structure of a product information management system provided by the embodiment of the application is shown, which comprises a client and a server, the client comprises a data acquisition module, a position number matching module, an information supplementary scanning module and an information output module, the server comprises a first processing module, a multi-code detection module, a second processing module, a multi-code decoding module, an image labeling module and a decoding output module, and wherein:

[0121] The data acquisition module is used to acquire a product image and send the product image to the server, and the product image comprises a plurality of codes;

[0122] The first processing module is used to perform first processing on the product image to obtain a plurality of image blocks of a preset size, and the first processing comprises deblurring, HDR light recovery, compression, overlapping blocking and edge padding.

[0123] The multi-code detection module is configured to detect each image block quickly by using an improved YOLOv8 multi-object detection model and a TensorRT inference acceleration technology, identify the codes in each image block, and output the detection coordinates, type number, and confidence of the codes, the improved YOLOv8 multi-object detection model comprises a Mosaic data enhancement, a backbone network, a neck network, and a head network, the backbone network adopts a CSPDarknet architecture, a Transformer, and a CBAM module, the neck network adopts a BiFPN structure and a self-attention layer, and the head network adopts a CIoU loss function, and the improved YOLOv8 multi-object detection model is trained by using a PyTorch framework;

[0124] The second processing module is configured to remove the repeated codes in all the image blocks, crop each remaining code from the image block to which the code belongs to obtain a code block of the code, and perform a second processing on each code block, the second processing comprising graying and binarization;

[0125] The multi-code decoding module is configured to, for each remaining code, call multiple decoding engines in a priority order to parse the code block of the code when the confidence of the code exceeds a preset threshold, determine whether there is any decoding engine outputting decoding information, and if yes, the decoding is successful, if not, adjust the resolution of the code block to a preset size, call the multiple decoding engines in the priority order to parse the adjusted code block again, and continue to determine whether there is any decoding engine outputting information, and if yes, the decoding is successful, if not, the decoding fails;

[0126] The image labeling module is configured to, for each code whose decoding is successful, label a range frame and decoding information of the code on a product image to obtain a labeled image;

[0127] The decoding output module is configured to output the labeled image and the detection coordinates and decoding information of all the codes whose decoding is successful to a client;

[0128] The bit number matching module is configured to determine whether an intersection over union between the detection coordinates of each code whose decoding is successful and a standard coordinate of the code exceeds a second threshold, and if yes, determine a bit number of the code according to the standard coordinate, and the relationship between the standard coordinate and the bit number is preconfigured;

[0129] The information supplementing module is configured to determine whether the number of all the codes whose decoding is successful is equal to a preset number, and if not, determine an unrecognized code according to the labeled image, manually decode the unrecognized code to obtain decoding information of the code, and artificially determine a bit number of the unrecognized code;

[0130] The information output module is configured to output the labeled image, the detection coordinates of all the codes whose decoding is successful, and the decoding information and bit numbers of all the codes.

[0131] The service end combines the use of GPU and CPU, wherein the GPU is responsible for performing core calculation of model inference, and utilizes parallel calculation to accelerate deep learning tasks; and the CPU undertakes data preprocessing, post-processing, system scheduling and other tasks.

[0132] In an example, the product image acquisition is achieved by taking a picture through a camera and uploading a local picture.

[0133] When taking a picture through the camera, if a scene with missing image content or serious interference is encountered, the camera-captured image is repaired in combination with a historical image caching technology to improve the decoding success rate and fault tolerance. The historical image caching technology stores recently processed high-quality image frames in the memory to form a time sequence cache pool for subsequent repair of missing or damaged frames.

[0134] Since the Base64 format has better compatibility during transmission, errors caused by encoding problems can be avoided, and in an example, the product image and the labeled image are in the Base64 format.

[0135] The system adopts a modular architecture, the client develops an image acquisition and information entry interface based on C#, and the service end completes image decoding and data storage by using Python and a MySQL database. In combination with the mixed development architecture of C# and Python, the cross-platform compatibility and deployment flexibility of the system are improved, long-term stable operation of the system in actual use is ensured, and the industrial-grade real-time requirement is met.

[0136] Through the interface connection with the process data of the process platform, the system can also automatically associate the recognition result with product flow, process, equipment number and other information, build a complete data closed loop, and support traceability analysis according to time, process, product and other dimensions. The system improves the product management efficiency by more than 3 times, and provides a digital transformation technical path that can be implemented for traditional manufacturing enterprises.

[0137] The above scheme builds a product information acquisition and management system integrating detection, decoding and management, realizes batch, non-contact and non-intervention product information acquisition, forms a closed loop system from image acquisition, information recognition, data verification to online management, solves the problems of lagging management mode, high management cost, low management efficiency, asymmetric product information and management errors in existing product information management, and provides a high-reliability information solution for manufacturing enterprises.

[0138] Based on the same inventive concept, the embodiment of the present application provides a computing device, comprising:

[0139] a memory for storing program instructions;

[0140] a processor for calling the program instructions stored in the memory and executing the above method according to the obtained program instructions.

[0141] Based on the same inventive concept, the embodiment of the present application provides a computer readable nonvolatile storage medium, characterized in that, comprising computer readable instructions, when the computer reads and executes the computer readable instructions, the computer executes the above method.

[0142] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage devices, etc.) containing computer-usable program code.

[0143] The present application is described with reference to flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of flows and / or blocks in the flowcharts and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing apparatus to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing apparatus produce a device implemented in the flowcharts and / or block diagrams. Figure 1 one or more flows and / or blocks Figure 1 an apparatus that performs the functions specified in the flow(s) or block(s).

[0144] These computer program instructions can also be stored in a computer readable memory that can direct the computer or other programmable data processing apparatus to work in a specific manner, so that the instructions stored in the computer readable memory produce a manufactured product including instruction apparatus, which implements the flowcharts and / or block diagrams. Figure 1 one or more flows and / or blocks Figure 1 an apparatus that performs the functions specified in the flow(s) or block(s).

[0145] These computer program instructions can also be loaded into a computer or other programmable data processing apparatus, so that a series of operation steps are performed on the computer or other programmable data processing apparatus to produce a computer implemented process, so that the instructions executed on the computer or other programmable data processing apparatus provide a process for implementing the flowcharts and / or block diagrams. Figure 1 one or more flows and / or blocks Figure 1 an apparatus that performs the functions specified in the flow(s) or block(s).

[0146] While the preferred embodiments of the application have been described, additional variations and modifications can be made to these embodiments by those skilled in the art once they have the benefit of the present disclosure without departing from the spirit and scope of the application. Accordingly, it is intended that such additions and modifications be included within the scope of the application. It is the following claims, including any amendments thereto, which define the scope of the application.

[0147] Obviously, numerous modifications and variations of the present application are possible in light of the above teachings. It is therefore to be understood that within the scope of the appended claims and their equivalents, the application can be practiced otherwise than as specifically described.

Claims

1. A multi-code decoding method, characterized in that, include: The product image is subjected to a first processing to obtain multiple image blocks of preset size. The product image includes multiple codes. The first processing includes deblurring, HDR illumination restoration, compression, overlapping block segmentation, and edge filling. An improved YOLOv8 multi-object detection model and TensorRT inference acceleration technology are used to quickly detect each image block, identify the code in each image block, and output the detection coordinates, type number, and confidence score of the code. The improved YOLOv8 multi-object detection model includes Mosaic data augmentation, a backbone network, a neck network, and a head network. The backbone network adopts the CSPDarknet architecture, Transformer, and CBAM modules. The neck network adopts the BiFPN structure and self-attention layer. The head network adopts the CIoU loss function. The improved YOLOv8 multi-object detection model is trained using the PyTorch framework. Remove duplicate codes from all image blocks, crop each remaining code from its respective image block to obtain its code block, and perform a second processing on each code block, the second processing including grayscale conversion and binarization. For each remaining code, when its confidence exceeds a preset threshold, multiple decoding engines are called in order of priority to parse its code block. It is determined whether any decoding engine outputs decoding information. If yes, the decoding is successful. If no, the resolution of its code block is adjusted to a preset size. The multiple decoding engines are called again in order of priority to parse the adjusted code block. It is then determined whether any decoding engine outputs information. If yes, the decoding is successful. If no, the decoding fails. For each code that is successfully decoded, its bounding box and decoding information are marked on the product image to obtain a marked image; Output the labeled image and the detection coordinates and decoding information of all successfully decoded codes.

2. The method according to claim 1, characterized in that, The codes include QR codes and barcodes.

3. The method according to claim 1, characterized in that, The deblurring is achieved through an improved DeblurGAN-v2 network, which employs an improved loss function. This improved loss function incorporates a structure-aware loss, which includes perceptual loss, edge gradient loss, and module scaling loss. The formula for the improved loss function is as follows: L total =L adv +λ1L percep +λ2L edge +λ3L ratio Among them, L adv =αL local +βL global To counteract the loss, α and β are used to balance the realism dimension of the generated image by the dual-scale discriminator in terms of local detail and global structure control, respectively. local For the loss of realism in local areas, L global For the realism loss of the global structure, λ1, λ2, and λ3 are used to quantize the constraint strength of the three physical structures: texture, edge, and scale, respectively. percep For the perceived loss, L edge For the edge gradient loss, L ratio This refers to the proportional loss of the module.

4. The method according to claim 1, characterized in that, The overlapping segmentation includes cutting the product image into image blocks according to the preset size, with an overlapping area of ​​preset width between adjacent image blocks.

5. The method according to claim 1, characterized in that, The process of removing duplicate codes from all image blocks includes: The detection coordinates of each code are converted into its coordinates in the product image, and the intersection-over-union ratio (IoU) between each code and other codes is calculated after the conversion. Determine whether there exists a case where the intersection-union ratio between two codes is greater than a first threshold. If so, discard the code with the smaller area among the two codes.

6. The method according to claim 1, characterized in that, The step of cropping each remaining code from its corresponding image block to obtain its code block includes: Based on the detection coordinates of each remaining code, a preset number of pixels are extended outward as its code block size, and the code block is cropped from its respective image block according to the detection coordinates and code block size of each remaining code.

7. An information matching method, according to any one of claims 1-6, characterized in that, include: Determine whether the intersection-union ratio (IUU) between the detection coordinates and the standard coordinates of each successfully decoded code exceeds a second threshold. If so, determine its bit number based on its standard coordinates. The relationship between the standard coordinates and the bit number is pre-configured. Determine whether the number of all successfully decoded codes is equal to the preset number. If not, determine the unrecognized codes based on the labeled image, manually decode the unrecognized codes to obtain their decoding information, and manually determine the bit number of the unrecognized codes. Output the labeled image, the detection coordinates of all successfully decoded codes, and the decoding information and bit number of all codes.

8. A product information collection and management system, characterized in that, The system includes a client and a server. The client includes a data acquisition module, a tag matching module, an information supplementation module, and an information output module. The server includes a first processing module, a multi-code detection module, a second processing module, a multi-code decoding module, an image annotation module, and a decoding output module. The data acquisition module is used to acquire product images and send the product images to the server. The product images include multiple codes. The first processing module is used to perform a first processing on the product image to obtain multiple image blocks of preset size. The first processing includes deblurring, HDR illumination restoration, compression, overlapping block segmentation, and edge filling. The multi-code detection module is used to quickly detect each image block using an improved YOLOv8 multi-object detection model and TensorRT inference acceleration technology, identify the codes in each image block, and output the detection coordinates, type number, and confidence score of the codes. The improved YOLOv8 multi-object detection model includes Mosaic data augmentation, a backbone network, a neck network, and a head network. The backbone network adopts the CSPDarknet architecture, Transformer, and CBAM modules. The neck network adopts the BiFPN structure and self-attention layer. The head network adopts the CIoU loss function. The improved YOLOv8 multi-object detection model is trained using the PyTorch framework. The second processing module is used to remove duplicate codes in all image blocks, crop each remaining code from its respective image block to obtain its code block, and perform a second processing on each code block, the second processing including grayscale conversion and binarization. The multi-code decoding module is used to, for each remaining code, when its confidence exceeds a preset threshold, call multiple decoding engines in priority order to parse its code block, determine whether any decoding engine outputs decoding information, if yes, decoding is successful, if no, adjust the resolution of its code block to a preset size, and call the multiple decoding engines again in priority order to parse the adjusted code block, continue to determine whether any decoding engine outputs information, if yes, decoding is successful, if no, decoding fails; The image annotation module is used to annotate the bounding box and decoding information on the product image for each code that is successfully decoded, thereby obtaining an annotated image. The decoding output module is used to output the labeled image and the detection coordinates and decoding information of all successfully decoded codes to the client; The bit number matching module is used to determine whether the intersection-union ratio between the detection coordinates and the standard coordinates of each successfully decoded code exceeds a second threshold. If so, the bit number is determined according to its standard coordinates. The relationship between the standard coordinates and the bit number is pre-configured. The information supplementation module is used to determine whether the number of all successfully decoded codes is equal to a preset number. If not, it determines the unrecognized codes based on the labeled image, manually decodes the unrecognized codes to obtain their decoding information, and manually determines the bit number of the unrecognized codes. The information output module is used to output the labeled image, the detection coordinates of all successfully decoded codes, and the decoding information and bit number of all codes.

9. A computing device, characterized in that, include: Memory, used to store program instructions; A processor is configured to invoke program instructions stored in the memory and execute the method as described in any one of claims 1-6 according to the obtained program.

10. A computer-readable non-volatile storage medium, characterized in that, It includes computer-readable instructions that, when read and executed by a computer, cause the computer to perform the method as described in any one of claims 1-6.

Images