A Method for Visual Feature Location and Information Decoding of Multi-Faceted Distributed Dense Tags

Abstract

The present invention discloses a method for visual feature positioning and information decoding of multi-sided distributed dense tags, which relates to the technical field of image feature recognition. The present invention includes respectively extracting feature points of an input image and a template tag image, creating and starting a tag positioning thread, extracting feature points of the input image and the template tag image, and positioning a suspected target tag area through feature point matching, obtaining the suspected tag position and extracting the suspected tag image. By performing image acquisition, tag positioning, and information decoding in independent processes, the present invention realizes a data acquisition and processing pipeline structure, thereby improving production efficiency. By using image feature detection technology, the images collected from all directions of the product package are processed respectively to obtain the suspected tag positions in the images of each side of the package, so as to achieve the purpose of multi-sided dense tag positioning of the package.

Description

Technical Field

[0001] The present invention belongs to the technical field of image feature recognition, and particularly relates to a method for visual feature positioning and information decoding of multi-sided distributed dense labels for packages. Background Art

[0002] There is a wide variety of product packages, and the types, quantities, and shapes of commodities in each product package are different. Therefore, methods for automatically identifying product packages based on various different factors need to be improved urgently. For example, Chinese Patent CN113318989A discloses an in-warehouse logistics sorting method and system. In this in-warehouse logistics sorting method, each lower package slot in the sorting device corresponds to a box installation position. A first label reader / writer is installed at each box installation position. Each box installation position is used to place a collection box. An electronic label is set on each collection box. By setting an electronic label on the collection box and a label reader / writer at each box installation position, the label printing module can automatically detect the arrival of a full collection box and print the corresponding waybill, thus providing reliable guidance for the staff at the delivery station to perform the packing and outbound operation, and reducing the difficulty of coordinated work in the package sorting department.

[0003] However, according to the actual requirements of the application scenario of the multi-sided label scanning software system for product packages, to simultaneously take into account the accuracy and real-time requirements of label positioning, currently, the main difficulties involved are as follows:

[0004] (1) For the same batch of products, it is necessary to position multiple different types of template labels, and the processing tasks are heavy;

[0005] (2) To ensure the overall operation efficiency of the system, there are real-time requirements for the time between two adjacent product packages;

[0006] (3) The lighting conditions on different sides of the product package are different, and the collected information is affected by the lighting conditions;

[0007] (4) Due to reasons such as the camera perspective and the placement position of the product package, the deformation of the suspected label area in the captured image is caused, resulting in inaccurate recognition;

[0008] (5) There is a problem that some details of different labels are similar, and it is difficult to distinguish the similar details of many labels.

[0009] Therefore, the detection and recognition algorithms based on traditional image processing cannot well solve problems such as incomplete label barcode information, image distortion, unclear images, etc. At the same time, the current methods for identifying label barcode information based on deep learning mostly output standard rectangular frames. The label barcode information images often contain some non-label areas, image distortion areas, etc. Moreover, there is a lack of processing for the orientation of the label barcode. Based on this, the present invention provides a method for visual feature localization and information decoding of multi-sided distributed dense labels for packages. Summary of the Invention

[0010] The purpose of the present invention is to provide a method for visual feature localization and information decoding of multi-sided distributed dense labels for packages. By using image feature detection technology, the images collected from all directions of the product package are processed respectively to obtain the suspected label positions in the images of each side of the package, so as to achieve the purpose of multi-sided dense label localization of the package. According to the position information of each label and the template information, each different position of each label is divided, and different methods are used to analyze and fuse information such as text (OCR), barcodes, and two-dimensional codes, and then they are immediately packaged and sent. The existing technical problems are solved.

[0011] To solve the above technical problems, the present invention is realized through the following technical solutions:

[0012] As the first aspect provided by the present invention, the present invention is a method for visual feature localization of multi-sided distributed dense labels for packages. By using image feature detection technology, the images collected from all directions of the product package are processed respectively to obtain the suspected label positions in the images of each side of the package, so as to achieve the purpose of multi-sided dense label localization of the package. According to the label position information obtained by the following processing method, and then according to the position information of each label and the template information, each different position of each label is divided.

[0013] Specifically, it includes the following steps:

[0014] Step S1: Create an input image cache for storing the images of each side of the product package collected and a label image cache for storing the label images obtained by extraction and processing.

[0015] Step S2: Calculate the feature information of each type of label image.

[0016] Step S3: Create and start a label localization thread, extract the feature points of the input image and the template label image, and locate the suspected target label area through feature point matching, obtain the suspected label position and extract the suspected label image.

[0017] Furthermore, the input image cache and the label image cache adopt a "first in, first out" structure. The depths and data structures of the input image cache and the label image cache can be configured. Data access and communication are carried out through memory sharing among different threads. The processes from image acquisition, label positioning to information decoding are all carried out in independent processes, realizing a data acquisition and processing pipeline structure, thereby improving production efficiency. The images of each surface of the product package collected by the camera acquisition thread are respectively cached into the corresponding input image cache. The label positioning thread sequentially takes out the images from the input image cache for processing, and the obtained label images are cached into the label image cache corresponding to the template.

[0018] Furthermore, in step S3:

[0019] The label positioning thread sequentially takes out the input images from the input image cache for image feature point extraction and matching, obtains all regions similar to the template label image from the input image, extracts the similar image regions and sequentially performs affine transformation and size transformation to obtain a suspected label image with the same specification size as the template label image. Furthermore, the information position in the suspected label image can be determined according to the information position marked in the template, and further, the information can be parsed item by item.

[0020] (1) Further, Feature Detection: The goal of this stage is to find key and stable feature points in the image, which are crucial for understanding the image content. Key point detection algorithms include SIFT (Scale-Invariant Feature Transform), SURF (Speeded-Up Robust Features), and ORB (Oriented FAST and Rotated BRIEF), etc.; Feature Description: After detecting the feature points, we need to generate a descriptor for each point, which is the feature encoding of the local area around the point. Descriptors are usually high-dimensional, numerical vectors used to uniquely identify each feature point. For example, the SIFT descriptor captures the gradient direction and scale information around the point; ORB uses BRIEF (Binary Robust Independent Elementary Features) encoding, which is simple and efficient; Feature Matching: Compare the feature points in one image with those in another image to find the most similar pairs. This step is usually achieved by calculating the distance or similarity between descriptors (such as Brute Force, FLANN, etc.); Data Structure and Filtering (Feature Descriptor Matching): Use efficient data structures (such as BF trees or KD trees) to accelerate the pairing process, and at the same time, use some strategies (such as RANSAC or Lowe's Ratio Test) to exclude inaccurate matches. Image Stitching and Correction: Correct the geometric relationship between images by utilizing feature matching.

[0021] Further, in step S3: The SURF algorithm is used to extract image feature points from the input image.

[0022] Further, in step S3: The approximate nearest neighbor matching algorithm FLANN (Fast Library for Approximate Nearest Neighbors) is used to obtain similar feature point pairs, and the optimal matching feature point pairs between the template label and the input image are obtained through a distance criterion.

[0023] Further, the method of affine transformation is as follows:

[0024] Obtain the optimal matching feature point pairs;

[0025] Calculate the feature point set of the suspected label area of the input image;

[0026] Use the RANSAC algorithm to calculate the homography transformation matrix;

[0027] The irregular quadrilateral region in the input image is transformed into a regular image with the same scale and shape as the template label image by using a transformation matrix.

[0028] Further, the method for judging the similarity between the suspected label image and the template label image is as follows:

[0029] It is determined by the hash fingerprint of the image and the SSIM (Structural Similarity Index) metric.

[0030] Further, the similarity judgment by the hash fingerprint of the image includes the following steps:

[0031] The hash fingerprints of the suspected label image and the template label image are designed and verified as 64-bit binary coded values;

[0032] By calculating the correlation of the hash fingerprints of the two images, it is judged whether the located suspected label image is a valid label image.

[0033] Further, the SSIM metric obtains the final similarity index by calculating the weighted product of three factors: the luminance factor, the contrast factor, and the structure factor of the image; among them, the luminance factor and the contrast factor are calculated by the mean and variance, and the structure factor is calculated by the covariance; the present invention selects SSIM as another criterion for the similarity between the suspected label and the template, which can effectively solve the problem of the difference between the input image and the template image caused by factors such as illumination between different surfaces of the product package.

[0034] As the second aspect provided by the present invention, the present invention is a method for decoding multi-sided distributed dense label information of a package. The information decoding method is used to decode the suspected label image obtained by the visual feature localization method described in the first aspect, and different methods are used to analyze and fuse information such as text (OCR), barcodes, and two-dimensional codes, and then instantaneously packetize and send.

[0035] The present invention has the following beneficial effects:

[0036] The present invention locates the suspected target label area by separately extracting the feature points of the input image and the template label image, and then performing feature point matching; by comparing the feature descriptors of the input image and the template image, similar feature point pairs are found, and the feature point set of the suspected label area of the input image is calculated. The image acquisition, label localization, and information decoding are all carried out in independent processes, realizing a data acquisition and processing pipeline structure, thereby improving production efficiency. By using image feature detection technology, the images collected in all directions of the product package are processed separately to obtain the positions of suspected labels in the images of each surface of the package, so as to achieve the purpose of multi-sided dense label localization of the package.

[0037] Of course, it is not necessary for any product implementing the present invention to achieve all of the above-mentioned advantages simultaneously. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following will briefly introduce the drawings required for describing the embodiments. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0039] Figure 1 It is a flowchart for positioning the visual features of the labels of the present invention;

[0040] Figure 2 It is a schematic plan view of a complex label;

[0041] Figure 3 It is a schematic diagram of the working principle of the system software;

[0042] Figure 4 It is a block diagram of the composition of the system software;

[0043] Figure 5 It is a processing flowchart of the multi-sided label scanning software system for product packages;

[0044] Figure 6 It is a processing flowchart of the multi-sided label scanning software system for product packages. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0045] In the following description, for the purpose of illustration rather than limitation, specific details such as specific system architectures, technologies, etc. are set forth in order to provide a thorough understanding of the embodiments of the present application. However, those skilled in the art should understand that the present application can also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid unnecessary details from interfering with the description of the present application.

[0046] It should be understood that when used in the specification of the present application and the appended claims, the term "comprising" indicates the presence of the described features, wholes, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components, and / or their combinations.

[0047] It should also be understood that the term "and / or" used in the specification of the present application and the appended claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0048] As used in the specification of this application and the appended claims, the term "if" may be construed contextually as "when", or "once", or "in response to determining", or "in response to detecting". Similarly, the phrases "if determined" or "if [the described condition or event] is detected" may be construed contextually to mean "once determined", or "in response to determining", or "once [the described condition or event] is detected", or "in response to detecting [the described condition or event]".

[0049] In addition, in the description of the specification of this application and the appended claims, the terms "first", "second", "third", etc. are only used for differential description and should not be construed as indicating or implying relative importance.

[0050] Reference to "one embodiment" or "some embodiments" or the like described in the specification of this application means that a specific feature, structure, or characteristic described in connection with that embodiment is included in one or more embodiments of this application. Thus, statements such as "in one embodiment", "in some embodiments", "in other some embodiments", "in still other embodiments", etc. that appear in different places in this specification do not necessarily all refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in another way. The terms "comprising", "including", "having" and their variants all mean "including but not limited to", unless otherwise specifically emphasized in another way.

[0051] Embodiment 1:

[0052] As Figure 1 shown, as the first embodiment provided by the present invention, the present invention is a method for visual feature positioning of multi-sided distributed dense labels on a package. A method for positioning multi-sided distributed dense labels proposed by the present invention processes the images collected from all directions of the product package by using image feature detection technology to obtain the suspected label positions in the images of each side of the package, so as to achieve the purpose of positioning multi-sided dense labels on the package, including the following steps:

[0053] Create an input image cache for storing the images of each side of the product package collected and a label image cache for storing the label images obtained by extraction and processing; calculate the feature information of each type of label image; create and start a label positioning thread, extract the feature points of the input image and the template label image, and locate the suspected target label area through feature point matching, obtain the suspected label position and extract the suspected label image;

[0054] Specifically, it includes the following steps:

[0055] (1) Create input image caches respectively according to the number of sides of the product package for caching the images collected from each side of the product package;

[0056] (2) Creating a label image cache for each type of label on each face based on the number of templates established for the required label types, and caching the valid label images obtained by positioning for the convenience of subsequent character extraction or barcode recognition processing;

[0057] (3) Calculate the feature information of each category label image;

[0058] (4) Create and start the tag positioning thread and enter the tag positioning thread loop;

[0059] (5) In the tag positioning thread, it is prioritized to determine whether the input image cache is empty. If it is empty, the thread is in a polling state until the input image cache is not empty and enters the next processing step;

[0060] (6) Extract the input image from the input image cache and perform image preprocessing such as filtering, noise reduction, and enhancement on the input image; Image preprocessing: Denoising, enhancement, and normalization are performed on the input image to facilitate subsequent processing;

[0061] (7) Calculating image features for the entire preprocessed input image;

[0062] (8) Matching the calculated input image features with the template features using the set conditions to determine whether there is a best matching feature that meets the conditions;

[0063] (9) If the best matching feature exists, the suspected tag position is calculated based on the best matching feature. If not, the processing of the current frame input image is completed, and the process returns to step (5) to continue polling the input image cache status until the next frame image is cached.

[0064] (10) extracting the irregular region of the label image from the input image according to the suspected label position calculated in step (9), and obtaining a suspected label image with a regular shape that is equal in size to the template image and orthogonal in length and width by performing an affine transformation on the image;

[0065] (11) Use image information entropy, signal-to-noise ratio, hash value and other information to conduct secondary screening and judgment on the similarity between the suspected label image and the template image to exclude abnormally located objects;

[0066] (12) For suspected label images that meet the similarity determination conditions, transfer them to the label image cache; if they do not meet the determination conditions, remove the suspected label image;

[0067] (13) When the processing of a suspected label image is completed, first determine whether the maximum number of label localizations is exceeded. If it is exceeded, return to step (5) and continue to poll the status of the input image buffer until the next frame of image is buffered. If the maximum number of label localizations is not exceeded, proceed to the next step to process the next suspected label image;

[0068] (14) Among the best-matched features calculated in step (8), remove the features of the suspected label image region positions obtained in steps (9) to (12) to obtain the remaining best-matched features. Jump to step (9) to calculate the new suspected label position, and iterate in this way until the maximum number of label localizations is exceeded or there are no qualified best-matched features, indicating that the processing of the current frame of input image is completed;

[0069] (15) If a label localization thread abort signal is received during the processing, abort the label localization thread and clear the image buffer, and the label localization process ends.

[0070] The label localization algorithm mainly involves the process of obtaining all suspected label images of class templates from the input image. From the perspective of image processing, it is a process of relatively accurately identifying and locating the regions similar to the given image in the input image by adopting certain processing methods. This process mainly involves image matching technology. As Figure 2 shown, Figure 2 is a schematic plan view of a complex label. The wrapped green frame (frame 1) represents the localization of the specified label, and the red frame (frame 2) represents the definition of different positions of the label after the specified label is localized. At the same time, different methods are used to classify and analyze barcodes, QR codes or text information.

[0071] As an embodiment provided by the present invention, preferably, the input image buffer and the label image buffer adopt a "first in first out" structure, and the depths and data structures of the input image buffer and the label image buffer can be configured. Data access and communication are carried out through memory sharing among different threads, and data acquisition, label localization and information decoding are all carried out in independent processes to realize a data acquisition and processing pipeline structure, thereby improving production efficiency; The images of each surface of the product package collected by the camera acquisition thread are respectively buffered to the corresponding input image buffer, and the label localization thread sequentially takes out the images from the input image buffer for processing, and the processed label images are buffered to the label image buffer corresponding to the template. The subsequent decoding link can take them out sequentially for processing.

[0072] As an embodiment provided by the present invention, preferably, the specific data information included in the input image buffer and the label image buffer is shown in Table 1:

[0073] Table 1

[0074]

[0075]

[0076] Note: ① The sub - pictures in the table represent the suspected label images obtained by positioning. ② Sub - picture category: For the convenience of the information decoding thread to synchronously encapsulate the decoding data of all sub - pictures obtained by the same template positioning of the same input picture, identification data with empty sub - picture data will be inserted into the label image cache at the start and end of the label positioning thread respectively. When the information decoding thread detects that the sub - picture category is 0x01, it starts to reset the decoding environment and prepares for information decoding of the sub - picture data from the new input picture. When it detects that the sub - picture category is 0x03, it represents the current package.

[0077] As an embodiment provided by the present invention, preferably, the information decoding content involved in the product package multi - surface label scanning software system mainly includes characters, barcodes, and QR codes. Therefore, the algorithm design for information decoding needs to select different information decoding algorithms for different contents. Among them, optical character recognition (OCR) algorithm is required for character information decoding, and barcode and QR code positioning and decoding algorithms are required for barcode and QR code information decoding, where:

[0078] Optical Character Recognition (OCR) is a technology that converts text in an image into editable, searchable, and processable electronic text. The mainstream OCR algorithms mainly include traditional OCR methods and modern deep - learning methods. The specific process mainly includes natural scene text region detection and character recognition. Currently, well - known Internet companies have successively launched powerful OCR algorithm libraries, and after continuous iteration and upgrade, they have been widely used in the industrial field, such as Baidu PaddleOCR, Alibaba Cloud OSSOCR, Tencent YouTu TencentOCR, Megvii Face++ OpenCV OCR, iFlytek OpenCV OCR, etc.

[0079] As an embodiment provided by the present invention, preferably, the barcode and QR code positioning and decoding algorithm mainly includes two parts: code region positioning and information parsing. Similar to OCR, the current mainstream method is to integrate and call third - party algorithm libraries in the system to implement the function. Common libraries mainly include Zbar, libdmtx, QRCode, ZXing, WeChatQRCode, etc. It is worth mentioning that the libdmtx library focuses on one - dimensional barcodes, especially the generation and recognition functions of the DataMatrix format.

[0080] As an embodiment provided by the present invention, preferably, after verification tests, the PaddleOCR algorithm is selected for decoding system software character information, Zbar is selected for barcodes, WeChat QRCode is selected for QR codes, and libdmtx is selected for DataMatrix.

[0081] As an embodiment provided by the present invention, preferably, the core function of label positioning is to find all regions in the input image that are similar to the template label image, extract the image regions, and successively perform affine transformation and size transformation to obtain suspected label images with the same specifications as the template image. Then, the information positions in the suspected label images can be determined according to the information positions marked in the template, and further, the information can be parsed item by item.

[0082] Specifically, the suspected target label region is located by respectively extracting the feature points of the input image and the template label image, and then performing feature point matching. The label positioning thread successively takes out the input images from the input image buffer for image feature point extraction and matching, obtains all regions in the input image that are similar to the template label image, extracts the similar image regions, and successively performs affine transformation and size transformation to obtain suspected label images with the same specifications as the template label image. Then, the information positions in the suspected label images can be determined according to the information positions marked in the template, and further, the information can be parsed item by item.

[0083] As an embodiment provided by the present invention, preferably, the SURF algorithm is used to extract image feature points from the input image, and the SURF (Speeded Up Robust Features) key points and descriptors are used as feature points.

[0084] As an embodiment provided by the present invention, preferably, SURF is an algorithm for image feature extraction and matching. It is a local feature-based method that can detect robust and unique feature points in an image and has good invariance to scale, rotation, and brightness changes. The main steps of using the SURF algorithm to extract image feature points from the input image include:

[0085] SS1: Scale space construction: Detect feature points at different scales in the input image by using the Difference of Gaussian pyramid;

[0086] SS2: Key point detection: Determine key points by detecting local extreme points in each scale space;

[0087] SS3: Direction assignment: Assign a main direction to each key point to improve the rotational invariance of the feature descriptor;

[0088] SS4: Feature description: Calculate the feature descriptor based on the image region around the key points for subsequent feature matching.

[0089] As an embodiment provided by the present invention, preferably, by comparing the feature descriptors of the input image and the template image, similar feature point pairs are found, and the matching results are screened according to certain criteria (such as distance, similarity, etc.) to remove incorrect matches and find the optimal matching features. According to the matching results, the pose transformation parameters of the suspected label in the input image, such as rotation, translation, etc., are calculated. Specifically, the approximate nearest neighbor matching algorithm FLANN (Fast Library for Approximate Nearest Neighbors) is used to obtain similar feature point pairs, and the optimal matching feature point pairs between the template label and the input image are obtained through the distance criterion.

[0090] As an embodiment provided by the present invention, preferably, the method of affine transformation is as follows:

[0091] Obtain the optimal matching feature point pairs obtained by feature matching;

[0092] Calculate the set of feature points in the suspected label area of the input image;

[0093] Use the RANSAC algorithm to calculate the homography transformation matrix;

[0094] Use the transformation matrix to transform the irregular quadrilateral area in the input image into a regular image with the same scale and shape as the template label image.

[0095] As an embodiment provided by the present invention, preferably, the suspected label image obtained through image affine transformation needs to be judged for similarity with the template image. By setting effective threshold parameters, similar but different types of label targets can be effectively excluded. The similarity of the present invention is mainly judged by the hash fingerprint of the image and the SSIM (Structural Similarity Index) index. The method for judging the similarity between the suspected label image and the template label image is as follows:

[0096] Judgment is carried out through the hash fingerprint of the image and the SSIM (Structural Similarity Index) index. The image hash fingerprint is a technology for image recognition and similarity comparison. It represents the features of the image by converting the image into a fixed-length binary code. This binary code is called the hash fingerprint, which can be used for quick comparison and matching of images. In the present invention, the hash fingerprints of the suspected label image and the template image are designed and verified as 64-bit binary coded values, and the correlation of the hash fingerprints of the two images is calculated to judge whether the located suspected label image is a valid label image.

[0097] As an embodiment provided by the present invention, preferably, the similarity judgment by the hash fingerprint of the image includes the following steps:

[0098] Design and verify the hash fingerprints of the suspected label image and the template label image as 64-bit binary coded values;

[0099] Judge whether the located suspected label image is a valid label image by calculating the correlation of the hash fingerprints of the two images.

[0100] As an embodiment provided by the present invention, preferably, SSIM (Structural Similarity Index) is an index used to measure the similarity between two images. It not only considers the brightness, contrast and structural information of the images, but also considers the characteristics of human eye perception. The value range of SSIM is between 0 and 1, and the closer the value is to 1, the more similar the two images are. The SSIM index obtains the final similarity index by calculating the weighted product of three factors: the brightness factor, the contrast factor and the structural factor of the image; among them, the brightness factor and the contrast factor are calculated by the mean value and variance, and the structural factor is calculated by the covariance; The present invention selects SSIM as another criterion for the similarity between the suspected label and the template, which can effectively solve the problem of the difference between the input image and the template image caused by factors such as illumination between different surfaces of the product package.

[0101] Embodiment 2:

[0102] As the second embodiment provided by the present invention, the present invention is a method for decoding distributed dense label information on multiple sides of a package. The information decoding method is used to decode the suspected label image obtained by the visual feature positioning method described in the first embodiment, and obtain the label position information according to the processing method provided in Embodiment 1. Then, according to the position information of each label and the template information, each different position of each label is divided, and different methods are used to analyze information such as text (OCR), barcodes, and two-dimensional codes, and information fusion is performed using existing technologies, and then it is immediately packetized and sent.

[0103] As an embodiment provided by the present invention, preferably, for each type and each label, after precise positioning according to the template features, according to the template information, the area to be decoded for each template is matched (i.e., the relative coordinate area in the label coordinate system), and the information represented by its content is defined. The method uses the PaddleOCR, Z-xing and libdmtx modules for decoding or text recognition. At the same time, an enhancement method for low signal-to-noise ratio barcode images is used to increase the decoding rate, and the recognized information is given the actual physical meaning in combination with the coordinate position of the template.

[0104] Embodiment 3:

[0105] As Figure 3 shown, as yet another embodiment provided by the present invention, the present invention provides a multi-sided distributed dense label visual feature positioning system for packages. This system is used to implement the multi-sided distributed dense label visual feature positioning method provided in Embodiments 1 and 2. After the package multi-sided distributed dense label visual feature positioning system selects and loads the template and starts decoding, it will first calculate the template image features. When the product package passes through the scanning area, it will trigger the WMS volume measurement module to perform volume measurement, and at the same time, it will trigger each side to perform image acquisition through the camera device hardware trigger function. The acquired images are used for display in the video display window on the one hand, and on the other hand, they will be temporarily stored in the input image buffer. Since there are time differences before and after the triggering moments of each side, and the system software does not access external synchronization signals, the software will generate synchronization signals according to the triggering moments of each side's images. The generation of the synchronization signal indicates that all sides have received image data; at the same time, the software sends a volume data acquisition request to the WMS through the HTTP / POST protocol to obtain the volume data measured by the WMS. The software temporarily stores the volume data fed back by the WMS for subsequent information summary and upload; when the positioning and decoding threads of each side receive the synchronization signal, they will take out a frame of image from the image buffer, calculate the features of this image, and then perform feature matching on the template image features and the input image features; according to the optimal feature point pairs found by the feature matching, calculate all suspected label positions; use the suspected label positions to sequentially extract all suspected label area images in the input image, and transform the suspected label area images into suspected label images of the same size as the template image through affine transformation; according to the information area marked during the template creation process, the positions of all information to be decoded can be determined in the suspected label image. Based on the decoded information positions, information position images can be obtained, and then according to the information categories (characters, barcodes, QR codes, etc.) of different information fields in the template, different information will be sent to different information decoding processes for decoding; all the decoded result information will be numbered according to different input images. After all the input images are decoded, the data will be summarized and encoded according to the rules required by the user, and finally the summarized information will be uploaded to the WMS through the HTTP / POST protocol.

[0106] As Figure 4 shown, as an embodiment provided by the present invention, preferably, the multi-sided distributed dense label visual feature positioning system for packages includes modules such as system task scheduling management, video image acquisition, digital image processing, template management and maintenance, WMS network communication, file and data management, etc. Among them:

[0107] The system task scheduling and management module is the bridge and transfer station for information interaction between the system software user interface and each module of the system. The system task scheduling and management module mainly includes the system module scheduling module, the system parameter setting module, and the system data visualization module. A: The system module scheduling module is mainly used to implement the initialization, data synchronization, scheduling management of each functional module of the system, and data interaction with the software user interface. Its core function lies in the control of the software decoding process, such as the preparation of the decoding environment, the start and stop of the decoding process, the post-processing and distribution of decoding information, etc.; B: The system parameter setting module is implemented through the system option interface and mainly completes the global parameter setting of the software system, including device data, network communication parameters, global parameters for label positioning and decoding, etc.; C: The system data visualization module intuitively presents and expresses the data during the software operation on the software user interface. The visualized data mainly includes the video images triggered for acquisition, the result information of information decoding and encapsulation, the system operation log information, etc.

[0108] The video image acquisition module mainly includes the camera device management module, the camera device parameter setting module, and the camera image acquisition module. A: The camera device management module is used to implement the enumeration, binding, and control of camera devices. Among them, camera enumeration traverses all available devices through the API interface functions provided by the camera device supplier. Device binding is to bind each camera to its corresponding product wrapping surface. Device control is to perform operations such as connecting and disconnecting the device, starting or stopping acquisition, etc.; B: The camera device parameter setting module queries and configures some parameters of the camera through the API interface functions provided by the camera device supplier, such as trigger source, trigger mode, exposure, contrast, region of interest, etc.; C: The camera image acquisition module creates an independent image acquisition thread for each camera and calls the API interface functions related to camera device image acquisition through the callback method to implement the process of obtaining the video frame images triggered for acquisition from the camera device. The obtained video frame images will be sent to the main interface video split-screen display window for display through the system task scheduling and management module on the one hand, and will be put into the image cache on the other hand and used as the input image for label positioning.

[0109] The digital image processing module is the core functional module for realizing the multi-sided label scanning function of product packages. It is an important part that determines the performance of the system software and production efficiency, and mainly includes a label positioning module and an information decoding module. A: The label positioning module extracts features from the sub-template images pre-created and generated by the template management and maintenance module and the video frame images triggered and collected by the video image acquisition module respectively, performs feature matching according to certain rules to find the optimal matching point pairs, then calculates all suspected label positions in sequence based on the optimal matching point pairs, and obtains all suspected label images with the same size and specifications as the sub-template images through coordinate and affine transformations, etc.; B: The information decoding module extracts the information area from the previously extracted suspected label images according to the valid information field area specified when creating the sub-template, and then performs different information decoding processes according to the types of valid information (characters, barcodes, QR codes, etc.). For example, it performs optical character recognition on character information to obtain character information, performs barcode recognition and decoding on barcodes or QR codes to obtain barcode information, and finally encapsulates all the obtained information in the organizational form required by the user and uploads it to the WMS through the WMS network communication module.

[0110] The template management and maintenance module mainly includes a template grouping management module, a sub-template maintenance module, and a template global parameter setting module. The design of the template management and maintenance module is mainly for users to complete the creation of sub-templates step by step using the sub-template creation wizard, and then uniformly group the sub-templates according to the decoding task requirements in the template grouping management module. Before the decoding task starts, the user then loads the template by manually selecting the template group. A: The template grouping management module realizes grouping the created sub-templates according to template categories, corresponding faces of product packages, etc., and can also realize operations such as addition, deletion, difference, and modification of template groups; B: The sub-template maintenance module is mainly designed for operations such as addition, deletion, query, and modification of sub-templates, which are specifically completed through the sub-template creation wizard; C: The template global parameter setting module sets the parameters applicable to all templates, such as the template storage directory, template remote switching parameters, the maximum number of template field groups, etc.

[0111] The WMS network communication module is designed to realize data interaction between the software system and WMS. The data interaction is carried out through the HTTP protocol. The transmitted data mainly includes obtaining volume data from WMS, receiving remote switching instructions for WMS templates, uploading decoding information and template grouping information to WMS. Among them, obtaining volume data from WMS and uploading decoding information and template grouping information to WMS are carried out by sending HTTP / POST requests to the WMS server for data transmission. The uploading of template grouping information is carried out after the template grouping operation in the template grouping management of the template management and maintenance module is completed; receiving the WMS template remote instruction is to create an HTTP server for template remote switching in the software system, and then WMS sends an HTTP / POST request to the software system to initiate a remote template switching task.

[0112] The file and data management module uniformly manages various files involved in the software system by adopting a singleton design. The path, format, storage period, etc. of file storage can be set respectively through the software user interface. The files involved mainly include system configuration files, decoding information files, decoding exception picture files, sub-template description files, etc. Among them, the system configuration files include software configuration files and decoding configuration files. The software configuration file is a file used to save the software user configuration parameters during the software startup and loading process. The decoding information file and the decoding configuration file are files used to describe and record the decoding tasks to avoid users repeatedly configuring the same decoding tasks.

[0113] Embodiment 4:

[0114] Based on Embodiments 1-3, as Figure 5 shown, the processing flow of the product package multi-sided label scanning software system is mainly divided into image acquisition and processing modules S7, S8, S9, S10, S11 and information comprehensive processing module S19.

[0115] Among them, the image acquisition and processing module includes single-sided image processing modules for five sides of the product packaging box, specifically including the front processing module S7, the back processing module S8, the left processing module S9, the right processing module S10, and the top processing module S11. Each single-sided image processing module consists of an image acquisition module, a label positioning module, and an information decoding module. Each single-sided image processing module synchronously receives and responds to an external trigger signal S1, obtains the image data of the current side from the industrial camera through the image acquisition module, and sends it to the label positioning module for label positioning of the current side. The located label is sent to the information decoding module for decoding and extraction of specific information, and the obtained information is sequentially transmitted to the information comprehensive processing module S19 to generate an information sequence of the current side of the packaging box.

[0116] The information comprehensive processing module S19 receives the product information processed by the image acquisition and processing module, generates the comprehensive information of multiple sides of the entire packaging box through information synthesis S17, and sends or uploads it through the information sending module S18.

[0117] As an embodiment provided by the present invention, preferably, as Figure 6 shown, each single-sided image processing module in the information processing module in the processing flow of the product package multi-sided label scanning software system includes an image acquisition module 21, a label positioning module 31, and an information decoding module 41, where:

[0118] The image acquisition module 21 is composed of an image acquisition 211, a frame synchronization counter 212, and an image buffer 213. 211 is used to control an industrial camera to acquire image data in response to an external trigger signal, generate a frame synchronization count value and send it to the frame synchronization counter 212, and the acquired image data is sent to the image buffer 213. As shown in 214, the image buffer 213 adopts a "first in first out" structure to ensure that the subsequent processing link obtains the image acquired at the earliest moment from the image buffer. The frame synchronization count value generated by the frame synchronization counter 212 is used for information synchronization and encapsulation of all information sequences of the single-sided image processing module.

[0119] The label positioning module 31 is used to process the cached image data, and dynamically create a label positioning process, an information extraction process, a label sequence cache, and an information sequence cache according to the label template information preset by the user. This process can respectively locate the position and quantity information of different category labels in the image, and the obtained label image information is sequentially stored in the label sequence cache. The number of creations of the label positioning process, the information extraction process, the label sequence cache, and the information sequence cache depends on the number of label categories preset by the user label template. The number of label categories set in this embodiment is m. As Figure 2 shown, 311 represents the positioning process of one category of label. The label sequence cache adopts a "first in first out" technology and is used to cache the label images obtained by the positioning process and other relevant information for characterizing the labels. The cache depth represents the number of labels. 313 represents the label sequence cache of the mth label positioning process. The number of labels obtained by the positioning process of this label positioning process in this embodiment is n.

[0120] The information decoding module 41 is used to sequentially obtain the located label images from the label sequence cache, and create an information extraction process according to the field information in the user label template. This process can identify or decode the information or barcodes at specific positions in the label images, and the processed information fields are sequentially stored in the information sequence cache. As Figure 6As shown, 411 represents the information extraction process for one of the category labels. The information sequence cache adopts the "first in, first out" technology and is used to cache the field information recognized or decoded by each information extraction process respectively. The cache depth represents the number of information fields. 413 represents the information sequence cache of the m-th information extraction process. In this embodiment, the number of information fields processed by this information extraction process is k.

[0121] All the information sequences obtained by the information decoding module 41 are finally processed by the information integration processing module S19 to obtain the product information of the entire product packaging box, which is used for sending or transmitting to other warehouse management systems.

[0122] A method for visual feature positioning and information decoding of multi-sided distributed dense labels of a package. The label positioning algorithm process is to use image processing algorithms to process the video images collected from each side of the product package in a process flow, and extract all suspected label images that have a high similarity with the template image; the information decoding algorithm process is to, according to the template field information defined by the user, use various different information decoding algorithms to sequentially decode all suspected label images obtained by the label positioning algorithm to obtain specific information, and finally perform information combination and encapsulation through the rules set by the user to obtain the comprehensive information of all sides of the product package and the process of structured data; other algorithms involved in the system software are mainly reflected in the algorithm design optimization of some intermediate processing links, aiming to improve the overall operation efficiency and performance of the system; the suspected target label area is located by extracting the feature points of the input image and the template label image respectively and then performing feature point matching; by comparing the feature descriptors of the input image and the template image, similar feature point pairs are found, and the feature point set of the suspected label area of the input image is calculated. The image acquisition, label positioning and information decoding are all carried out in independent processes to realize the data acquisition and processing pipeline structure, thereby improving production efficiency. By using image feature detection technology, the images collected from all directions of the product package are processed respectively to obtain the positions of suspected labels in the images of each side of the package, so as to achieve the purpose of multi-sided dense label positioning of the package.

[0123] In the description of this specification, the descriptions referring to terms such as "one embodiment", "example", "specific example", etc. mean that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in a suitable manner in any one or more embodiments or examples.

[0124] The preferred embodiments of the present invention disclosed above are only used to help illustrate the present invention. The preferred embodiments do not describe all the details in detail, nor do they limit the invention to the specific embodiments described. Obviously, many modifications and variations can be made according to the content of this specification. These embodiments are selected and specifically described in this specification to better explain the principles and practical applications of the present invention, so that those skilled in the art can well understand and utilize the present invention. The present invention is only limited by the claims and their full scope and equivalents.

Claims

1. A visual feature localization method for multi-sided distributed dense tags, characterized in that, Including the following steps: Step S1: Create an input image cache for storing images of each side of the collected product package and a label image cache for storing the located label images; each single-sided image processing module synchronously receives and responds to an external trigger signal, obtains the image data of the current side from an industrial camera through an image acquisition module, and sends it to a label positioning module for label positioning of the current side. The located label is sent to an information decoding module for decoding and extracting specific information, and the obtained information is sequentially transmitted to an information comprehensive processing module to generate an information sequence of the current side of the packing box. Step S2: Calculate the feature information of each type of label image. Step S3: Create and start a label positioning thread, extract the feature points of the input image and the template label image, and locate the suspected target label area through feature point matching, obtain the suspected label position and extract the suspected label image; in the label positioning thread, first judge whether the input image cache is empty. If it is empty, the thread is in a polling state until the input image cache is not empty and then enters the next processing link.

2. The visual feature positioning method for multi-sided distributed dense tags according to claim 1, characterized in that The input image cache and the label image cache adopt a "first-in, first-out" structure, and the depths and data structures of the input image cache and the label image cache can be configured. Data access and communication are carried out through memory sharing among different threads; the images of each side of the product package collected by the camera acquisition thread are respectively cached to the corresponding input image cache, and the label positioning thread sequentially takes out the images from the input image cache for processing, and the processed label images are cached to the label image cache corresponding to the template.

3. A visual feature localization method for a multi-sided distributed dense label package according to claim 1, characterized in that In the said Step S3: The label positioning thread sequentially takes out the input images from the input image cache for image feature point extraction and matching, obtains all the regions similar to the template label image from the input image, cuts out the similar image regions and sequentially undergoes affine transformation and size transformation to obtain a suspected label image with the same specification size as the template label image.

4. A method for visually locating the features of a multi-sided distributed dense label according to claim 3, characterized in that, In the said Step S3: The SURF algorithm is used to extract the image feature points of the input image.

5. A method for visual feature localization of a multi-sided distributed dense label package according to claim 4, characterized in that, In the said Step S3: The approximate nearest neighbor matching algorithm FLANN is used to obtain similar feature point pairs, and the optimal matching feature point pairs between the template label and the input image are obtained through a distance criterion.

6. A method for visually locating multi-sided distributed dense tags of a package according to claim 3, characterized in that The method of the said affine transformation is as follows: Obtain the optimal matching feature point pairs; Calculate the feature point set of the suspected label area of the input image; Use the RANSAC algorithm to calculate the homography transformation matrix; Use the transformation matrix to transform the irregular quadrilateral area in the input image into a regular image with the same scale and shape as the template label image.

7. A method for visually locating multi-sided distributed dense labels of a package according to claim 3, characterized in that, The method for judging the similarity between the suspected label image and the template label image is as follows: Judge through the hash fingerprint and SSIM index of the image.

8. A method for visually locating multi-sided distributed dense tags of a package, according to claim 7, wherein The similarity judgment through the hash fingerprint of the image includes the following steps: Design and verify the hash fingerprints of the suspected label image and the template label image as 64-bit binary coded values; Judge whether the located suspected label image is a valid label image by calculating the correlation of the hash fingerprints of the two images.

9. A method for visually locating multi-sided distributed dense labels of a package, according to claim 7, characterized in that The SSIM index obtains the final similarity index by calculating the weighted product of three factors: the luminance factor, the contrast factor, and the structure factor of the image; among them, the luminance factor and the contrast factor are calculated through the mean and variance, and the structure factor is calculated through the covariance.

10. A method for decoding multi-sided distributed dense label information, characterized in that, The information decoding method is used to decode the suspected label image obtained by the visual feature localization method according to any one of claims 1-9.

Images