Parcel multi-surface distributed dense label visual feature positioning and information decoding method

Abstract

The invention discloses a parcel multi-surface distributed dense label visual feature positioning and information decoding method, and relates to the technical field of image feature recognition. The method comprises the steps of respectively extracting feature points of an input image and a template label image, creating and starting a label positioning thread, extracting feature points of the input image and the template label image, positioning a suspected target label area through feature point matching, obtaining a suspected label position and extracting a suspected label image. According to the method, image acquisition, label positioning and information decoding are all carried out in independent processes, a data acquisition and processing pipeline structure is realized, so that the production efficiency is improved, the image feature detection technology is utilized to respectively process images acquired by product packages in all directions, suspected label positions in the images of all surfaces of the packages are obtained, and the detection accuracy is improved. Therefore, the purpose of wrapping multi-surface dense label positioning is achieved.

Description

Technical Field

[0001] The present invention belongs to the technical field of image feature recognition, and in particular relates to a method for visual feature positioning and information decoding of multi-sided distributed dense labels of a package. Background Art

[0002] There are many kinds of product packages, and the types, quantities, and shapes of goods in each product package are different. Therefore, the method of automatically identifying product packages based on a variety of different factors needs to be improved. For example, Chinese patent CN113318989A discloses a warehouse logistics sorting method and system. In the warehouse logistics sorting method, each lower bag opening in the sorting device corresponds to a box body installation position. A first label reader is installed on each box body installation position. Each box body installation position is used to place a collection box. An electronic label is provided on the collection box. By setting an electronic label on the collection box and setting a label reader at each box body installation position, the label printing module can automatically detect the arrival of a full collection box and print out the corresponding delivery note, thereby providing reliable guidance for the delivery station staff to carry out packaging and outbound operations, and reducing the difficulty of coordination work of the package sorting department.

[0003] However, according to the actual needs of the application scenarios of the product package multi-sided label scanning software system, it is necessary to take into account the accuracy and real-time requirements of label positioning. At present, the main difficulties involved are:

[0004] (1) The same batch of products needs to locate multiple different types of template labels, which is a heavy task;

[0005] (2) To ensure the overall operating efficiency of the system, the time between two adjacent product packages must be real-time;

[0006] (3) The lighting conditions on different sides of the product package are different, and the collected information is affected by the lighting conditions;

[0007] (4) The suspected label area in the triggered image is deformed due to factors such as the camera angle of view and the placement of the product package, resulting in inaccurate recognition;

[0008] (5) There is a problem that some details of different labels are similar, and it is difficult to distinguish similar details of many labels.

[0009] Therefore, the detection and recognition algorithm based on traditional image processing cannot solve the problems of incomplete label barcode information, image deformation, unclear image, etc. At the same time, the current label barcode information recognition method based on deep learning outputs mostly standard rectangular frames, and the label barcode information image often contains some non-label areas, image deformation areas, etc. Moreover, the orientation processing of the label barcode is lacking. Based on this, the present invention provides a method for visual feature positioning and information decoding of multi-sided distributed dense labels of packages. Summary of the invention

[0010] The purpose of the present invention is to provide a method for visual feature positioning and information decoding of multi-sided distributed dense labels on a package. By using image feature detection technology, the images collected from all directions of the product package are processed separately to obtain the suspected label positions in the images of each side of the package, thereby achieving the purpose of locating the multi-sided dense labels on the package. According to the position information of each label and the template information, each different position of each label is divided, and different methods are used to parse and fuse the information such as text (OCR), barcode, QR code, etc., and then the information is immediately packaged and sent. The existing technical problems are solved.

[0011] In order to solve the above technical problems, the present invention is achieved through the following technical solutions:

[0012] As a first aspect provided by the present invention, the present invention is a method for visual feature positioning of multi-sided distributed dense labels on a package, which uses image feature detection technology to process images collected from all directions of the product package respectively, and obtain the suspected label positions in the images of each side of the package, thereby achieving the purpose of locating multi-sided dense labels on the package. The label position information is obtained according to the following processing method, and then each different position of each label is divided according to the position information of each label and the template information.

[0013] The specific steps include:

[0014] Step S1: creating an input image cache for storing the collected images of each surface of the product package and a label image cache for storing the label image obtained by extraction processing;

[0015] Step S2: Calculate the feature information of each type of label image;

[0016] Step S3: Create and start a label positioning thread, extract feature points of the input image and the template label image, and locate the suspected target label area through feature point matching, obtain the suspected label position and extract the suspected label image.

[0017] Furthermore, the input image cache and label image cache adopt a "first-in-first-out" structure, and the depth and data structure of the input image cache and label image cache can be configured. Data access and communication are performed through memory sharing between different threads. From image acquisition, label positioning to information decoding, they are all carried out in independent processes, realizing a data acquisition and processing pipeline structure, thereby improving production efficiency; the images of each side of the product package collected by the camera acquisition thread are cached in the corresponding input image cache respectively, and the label positioning thread takes out the images from the input image cache in turn for processing, and the processed label images are cached in the label image cache of the corresponding template.

[0018] Furthermore, in step S3:

[0019] The label positioning thread takes out the input images from the input image cache in turn to extract and match image feature points, obtains all areas similar to the template label image from the input image, extracts similar image areas and successively undergoes affine transformation and size transformation to obtain a suspected label image of the same size as the template label image, and then determines the information position in the suspected label image based on the information position marked in the template, and further parses the information one by one.

[0020] (1) Further, Feature Detection: The goal of this stage is to find key, stable feature points in the image, which are crucial for understanding the image content. Key point detection algorithms include SIFT (Scale Invariant Feature Transform), SURF (Speeded Up Robust Features) and ORB (Oriented FAST and Rotated BRIEF), etc.; Feature Description: After detecting the feature points, we need to generate a descriptor for each point, which is the feature encoding of the local area around the point. The descriptor is usually a high-dimensional, numerical vector that uniquely identifies each feature point. For example, the SIFT descriptor captures the gradient direction and scale information around the point; ORB uses BRIEF (Binary Robust Independent Elementary Features) encoding, which is simple and efficient; Matching: Compare the feature points in one image with the feature points in another image to find the most similar pairing. This step is usually achieved by calculating the distance or similarity between descriptors (such as Brute Force, FLANN, etc.); Data structure and screening (Feature Descriptor Matching): Use efficient data structures (such as BF tree or KD tree) to speed up the pairing process, and use some strategies (such as RANSAC or Lowe's Ratio Test) to exclude inaccurate matches. Image stitching and correction (Image Stitching): Correct the geometric relationship between images by using feature matching.

[0021] Furthermore, in step S3: a SURF algorithm is used to extract image feature points from the input image.

[0022] Furthermore, in step S3: an approximate nearest neighbor matching algorithm FLANN (Fast Library for Approximate Nearest Neighbors) is used to obtain similar feature point pairs, and the best matching feature point pair between the template label and the input image is obtained through a distance criterion.

[0023] Furthermore, the affine transformation method is:

[0024] Get the best matching feature point pair;

[0025] Calculate and obtain the feature point set of the suspected label area of ​​the input image;

[0026] Use the RANSAC algorithm to calculate the homography transformation matrix;

[0027] The irregular quadrilateral region in the input image is transformed into a regular image with the same scale and shape as the template label image using a transformation matrix.

[0028] Furthermore, the similarity determination method between the suspected label image and the template label image is as follows:

[0029] The judgment is made through the image’s hash fingerprint and SSIM (Structural Similarity Index) indicator.

[0030] Furthermore, similarity determination by hash fingerprint of the image includes the following steps:

[0031] The hash fingerprints of the suspected label image and the template label image are designed and verified as 64-bit binary coded values;

[0032] By calculating the correlation between the hash fingerprints of the two images, it is determined whether the located suspected label image is a valid label image.

[0033] Furthermore, the SSIM index obtains the final similarity index by calculating the weighted product of the brightness factor, contrast factor and structure factor of the image; wherein the brightness factor and contrast factor are obtained by calculating the mean and variance, and the structure factor is obtained by calculating the covariance; the present invention selects SSIM as another criterion for the similarity between the suspected label and the template, which can effectively solve the problem of the difference between the input image and the template image caused by factors such as lighting between different surfaces of the product package.

[0034] As a second aspect provided by the present invention, the present invention is a method for decoding multi-sided distributed dense label information of a package, wherein the information decoding method is used to decode the suspected label image obtained by the visual feature positioning method described in the first aspect, and adopts different methods to parse and fuse information such as text (OCR), barcode, QR code, and then instantly seal, package and send it.

[0035] The present invention has the following beneficial effects:

[0036] The present invention locates the suspected target label area by respectively extracting feature points of an input image and a template label image and then matching the feature points; by comparing the feature descriptors of the input image and the template image, similar feature point pairs are found, and a feature point set of the suspected label area of ​​the input image is calculated. Image acquisition, label positioning and information decoding are all carried out in an independent process to realize a data acquisition and processing pipeline structure, thereby improving production efficiency. By utilizing image feature detection technology, images collected from all directions of product packages are processed separately to obtain the suspected label positions in the images of each side of the package, thereby achieving the purpose of locating dense labels on multiple sides of the package.

[0037] Of course, any product implementing the present invention does not necessarily need to achieve all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for describing the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0039] Figure 1 This is a flowchart of label visual feature positioning of the present invention;

[0040] Figure 2 It is a schematic plan view of a complex label;

[0041] Figure 3 This is a schematic diagram of the working principle of the system software;

[0042] Figure 4 Provides an architecture diagram for the system software;

[0043] Figure 5 Processing flow chart of software system for scanning labels on multiple sides of product packages;

[0044] Figure 6 Processing flow chart of the software system for scanning multi-sided labels on product packages. DETAILED DESCRIPTION

[0045] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0046] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.

[0047] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0048] As used in the specification and appended claims of this application, the term "if" can be interpreted as "when" or "uponce" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "uponce it is determined" or "in response to determining" or "uponce [described condition or event] is detected" or "in response to detecting [described condition or event]", depending on the context.

[0049] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0050] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0051] Embodiment 1:

[0052] like Figure 1 As shown in the figure, as the first embodiment provided by the present invention, the present invention is a method for visual feature positioning of multi-sided distributed dense labels on a package. The method for positioning multi-sided distributed dense labels on a package proposed by the present invention uses image feature detection technology to process images collected from all directions of the product package respectively, and obtain the suspected label positions in the images of each side of the package, thereby achieving the purpose of positioning multi-sided dense labels on the package, and includes the following steps:

[0053] Create an input image cache for storing the collected images of each surface of the product package and a label image cache for storing the label images obtained through extraction and processing; calculate the feature information of each type of label image; create and start a label positioning thread, extract the feature points of the input image and the template label image, and locate the suspected target label area through feature point matching, obtain the suspected label position and extract the suspected label image;

[0054] Specifically, the following steps are included:

[0055] (1) creating input image caches according to the number of sides of the product package, for caching images collected from each side of the product package;

[0056] (2) Creating a label image cache for each type of label on each face according to the number of templates established for the required types of labels to be located, and caching the valid label images obtained by location for the convenience of subsequent character extraction or barcode recognition processing;

[0057] (3) Calculate the feature information of each category label image;

[0058] (4) Create and start the tag positioning thread and enter the tag positioning thread loop;

[0059] (5) In the tag positioning thread, it is first determined whether the input image cache is empty. If it is empty, the thread is in a polling state until the input image cache is not empty and enters the next processing step;

[0060] (6) Extract the input image from the input image buffer and perform image preprocessing such as filtering, denoising, and enhancement on the input image; Image preprocessing: perform operations such as denoising, enhancement, and normalization on the input image to facilitate subsequent processing;

[0061] (7) Calculating image features for the entire preprocessed input image;

[0062] (8) Matching the calculated input image features with the template features using set conditions to determine whether there is a best matching feature that meets the conditions;

[0063] (9) If the best matching feature exists, the suspected tag position is calculated according to the best matching feature. If not, the processing of the current frame input image is completed, and the process returns to step (5) to continue polling the input image cache status until the next frame image is cached.

[0064] (10) extracting the irregular region of the label image from the input image according to the suspected label position calculated in step (9), and obtaining a suspected label image of a regular shape having the same size as the template image and orthogonal length and width by affine transformation of the image;

[0065] (11) Use image information entropy, signal-to-noise ratio, hash value and other information to perform secondary screening and determination of the similarity between the suspected label image and the template image to exclude abnormally located objects;

[0066] (12) For suspected label images that meet the similarity determination conditions, the suspected label images are transferred to the label image cache. If the suspected label images do not meet the determination conditions, the suspected label images are discarded;

[0067] (13) When the processing of a suspected tag image is completed, it is first determined whether the maximum number of tag positioning is exceeded. If it is exceeded, it returns to step (5) and continues to poll the input image cache status until the next frame image is cached. If it does not exceed the maximum number of tag positioning, it proceeds to the next step to process the next suspected tag image;

[0068] (14) From the best matching features calculated in step (8), remove the features of the suspected label image area obtained in steps (9) to (12) to obtain the remaining best matching features, jump to step (9), calculate a new suspected label position, and iterate in this way until the maximum number of label locations is exceeded or there is no best matching feature that meets the conditions, indicating that the processing of the current frame input image is completed;

[0069] (15) If a tag positioning thread abort signal is received during the processing, the tag positioning thread is aborted and the image cache is cleared, and the tag positioning process ends.

[0070] The label localization algorithm is mainly the process of obtaining all suspected label images of the class template from the input image. From the perspective of image processing, it is the process of relatively accurately identifying and locating the area in the input image that is similar to the given image by using a certain processing method. This process mainly involves image matching technology. Figure 2 As shown, Figure 2 This is a schematic plan view of a complex label. The green box (box 1) on the package represents the location of the specified label, and the red box (box 2) represents the definition of different positions of the label after the specified label is located. At the same time, different methods are used to classify and parse the barcode, QR code or text information.

[0071] As an embodiment provided by the present invention, preferably, the input image cache and the label image cache adopt a "first-in-first-out" structure, and the depth and data structure of the input image cache and the label image cache can be configured. Data access and communication are performed through memory sharing between different threads, and all processes from image acquisition, label positioning to information decoding are performed in independent processes, realizing a data acquisition and processing pipeline structure, thereby improving production efficiency; the images of each surface of the product package acquired by the camera acquisition thread are cached in the corresponding input image cache respectively, and the label positioning thread takes out the images from the input image cache in turn for processing, and the processed label images are cached in the label image cache of the corresponding template, and the subsequent decoding links can take them out in turn for processing.

[0072] As an embodiment provided by the present invention, preferably, the data information specifically included in the input image cache and the label image cache is shown in Table 1:

[0073] Table 1

[0074]

[0075]

[0076] Note: ① The sub-image in the table indicates the suspected label image obtained by positioning; ② Sub-image category: In order to facilitate the information decoding thread to synchronously encapsulate all sub-image decoding data obtained by positioning the same input image and the same template, the identification data indicating that the sub-image data is empty will be inserted into the label image cache at the beginning and end of the label positioning thread. When the information decoding thread detects that the sub-image category is 0x01, it starts to reset the decoding environment and prepare for information decoding of the sub-image data from the new input image. When the sub-image category is detected to be 0x03, it indicates the current package.

[0077] As an embodiment provided by the present invention, preferably, the information decoding content involved in the product package multi-sided label scanning software system mainly includes characters, bar codes and two-dimensional codes. Therefore, the information decoding algorithm design needs to select different information decoding algorithms for different contents, wherein the character information decoding needs to adopt the optical character recognition (OCR) algorithm, and the bar code and two-dimensional code information decoding needs to be decoded by adopting the bar code or two-dimensional code positioning and decoding algorithm, wherein:

[0078] Optical Character Recognition (OCR) is a technology that converts text in an image into editable, searchable, and processable electronic text. Mainstream OCR algorithms mainly include traditional OCR methods and modern deep learning methods. The specific process mainly includes natural scene text area detection and character recognition. At present, well-known Internet companies have successively launched powerful OCR algorithm libraries, and after continuous iteration and upgrading, they have been widely used in the industrial field, such as Baidu PaddleOCR, Alibaba Cloud OSSOCR, Tencent Youtu TencentOCR, Megvii Face++OpenCV OCR, iFlytek OpenCV OCR, etc.

[0079] As an embodiment provided by the present invention, preferably, the barcode and two-dimensional code positioning and decoding algorithm mainly includes two parts: positioning of the code area and information analysis. Similar to OCR, the current mainstream method is to integrate and call the method of the third-party algorithm library in the system to realize the function. Common libraries mainly include Zbar, libdmtx, QRCode, ZXing, WeChatQRCode, etc. It is worth mentioning that the libdmtx library focuses on one-dimensional barcodes, especially the generation and recognition functions of DataMatrix format.

[0080] As an embodiment provided by the present invention, preferably, after verification and testing, the system software character information decoding selects PaddleOCR algorithm, barcode selects Zbar, QR code selects WeChat QRCode, and DataMatrix selects libdmtx.

[0081] As an embodiment provided by the present invention, preferably, the core function of label positioning is to find all areas similar to the template label image in the input image, extract the image area and sequentially perform affine transformation and size transformation to obtain a suspected label image of the same specification and size as the template image, and then determine the information position in the suspected label image according to the information position marked in the template, and further parse the information one by one.

[0082] Specifically, the suspected target label area is located by respectively extracting feature points of the input image and the template label image and then performing feature point matching. The label positioning thread sequentially takes out the input image from the input image cache to extract and match image feature points, obtains all areas similar to the template label image from the input image, extracts similar image areas and sequentially undergoes affine transformation and size transformation to obtain a suspected label image of the same size as the template label image. Then, the information position in the suspected label image can be determined based on the information position marked in the template, and the information can be further parsed item by item.

[0083] As an embodiment provided by the present invention, preferably, a SURF algorithm is used to extract image feature points from an input image, with SURF (Speeded Up Robust Features) key points and descriptors as feature points.

[0084] As an embodiment provided by the present invention, preferably, SURF is an algorithm for image feature extraction and matching. It is a method based on local features, which can detect robust and unique feature points in an image and has good invariance to changes in scale, rotation and brightness. The main steps of extracting image feature points from an input image using the SURF algorithm include:

[0085] SS1: Scale space construction: Detect feature points at different scales in the input image by using Gaussian difference pyramid;

[0086] SS2: Key point detection: In each scale space, key points are determined by detecting local extreme points;

[0087] SS3: Direction assignment: Assign a main direction to each key point to improve the rotation invariance of the feature descriptor;

[0088] SS4: Feature description: Calculate feature descriptors based on the image area around the key points for subsequent feature matching.

[0089] As an embodiment provided by the present invention, preferably, by comparing the feature descriptors of the input image and the template image, similar feature point pairs are found, and the matching results are screened according to certain criteria (such as distance, similarity, etc.), wrong matches are removed, and the optimal matching features are found. According to the matching results, the posture transformation parameters of the suspected label in the input image are calculated, such as rotation, translation, etc. Specifically, the approximate nearest neighbor matching algorithm FLANN (Fast Library for Approximate Nearest Neighbors) is used to obtain similar feature point pairs, and the optimal matching feature point pair between the template label and the input image is obtained by the distance criterion.

[0090] As an embodiment provided by the present invention, preferably, the method of affine transformation is:

[0091] Obtain the best matching feature point pair obtained by feature matching;

[0092] Calculate and obtain the feature point set of the suspected label area of ​​the input image;

[0093] Use the RANSAC algorithm to calculate the homography transformation matrix;

[0094] The irregular quadrilateral region in the input image is transformed into a regular image with the same scale and shape as the template label image using a transformation matrix.

[0095] As an embodiment provided by the present invention, preferably, the suspected label image obtained by image affine transformation needs to be similar to the template image. By setting an effective threshold parameter, similar but different types of label targets can be effectively excluded. The similarity of the present invention is mainly determined by the hash fingerprint and SSIM (Structural Similarity Index) index of the image. The similarity determination method between the suspected label image and the template label image is:

[0096] The image hash fingerprint and SSIM (Structural Similarity Index) are used for judgment. Image hash fingerprint is a technology used for image recognition and similarity comparison. It represents the characteristics of the image by converting the image into a fixed-length binary code. This binary code is called a hash fingerprint, which can be used to quickly compare and match images. The present invention verifies the hash fingerprint of the suspected label image and the template image as a 64-bit binary code value, and determines whether the located suspected label image is a valid label image by calculating the correlation between the hash fingerprints of the two images.

[0097] As an embodiment provided by the present invention, preferably, similarity judgment by hash fingerprint of the image includes the following steps:

[0098] The hash fingerprints of the suspected label image and the template label image are designed and verified as 64-bit binary coded values;

[0099] By calculating the correlation between the hash fingerprints of the two images, it is determined whether the located suspected label image is a valid label image.

[0100] As an embodiment provided by the present invention, preferably, SSIM (Structural Similarity Index) is an indicator for measuring the similarity between two images. It not only takes into account the brightness, contrast and structural information of the image, but also takes into account the characteristics of human eye perception. The value range of SSIM is between 0 and 1, and the closer the value is to 1, the more similar the two images are. The SSIM indicator obtains the final similarity index by calculating the weighted product of the three factors of the image brightness factor, contrast factor and structure factor; wherein the brightness factor and contrast factor are calculated by the mean and variance, and the structure factor is calculated by the covariance; the present invention selects SSIM as another criterion for the similarity between the suspected label and the template, which can effectively solve the problem of the difference between the input image and the template image caused by factors such as lighting between different surfaces of the product package.

[0101] Embodiment 2:

[0102] As a second embodiment provided by the present invention, the present invention is a method for decoding multi-sided distributed dense label information of a package, the information decoding method is used to decode the suspected label image obtained by the visual feature positioning method described in the first embodiment, and obtain the label position information according to the processing method provided in the first embodiment, and then divide each different position of each label according to the position information of each label and the template information, use different methods to parse text (OCR), barcode, QR code and other information, use existing technology to fuse the information, and then immediately package and send them.

[0103] As an embodiment provided by the present invention, preferably, each type and each label is accurately positioned according to the template features, and then the area that needs to be decoded for each template (i.e., the relative coordinate area in the label coordinate system) is matched according to the template information, and the information represented by its content is defined. The method uses PaddleOCR, Z-xing and libdmtx modules for decoding or text recognition. At the same time, an enhancement method for low signal-to-noise ratio barcode images is used to increase the decoding rate, and the actual physical meaning is given to the recognized information in combination with the coordinate position of the template.

[0104] Embodiment three:

[0105] like Figure 3 As shown, as another embodiment provided by the present invention, the present invention provides a system for visual feature positioning of multi-sided distributed dense labels of packages. This system is used to implement the method for visual feature positioning of multi-sided distributed dense labels of packages provided in embodiments 1 and 2. After selecting the template to be loaded and starting decoding, the system for visual feature positioning of multi-sided distributed dense labels of packages will give priority to calculating the template image features. When the product package passes through the code scanning area, the WMS volume measurement module will be triggered to perform volume measurement. At the same time, the hardware trigger function of the camera device will be used to trigger each surface for image acquisition. The acquired image is used for display in the video display window on the one hand, and is temporarily stored in the input image cache on the other hand. Since there are differences in the triggering moments of each surface, and the system software is not connected to the external synchronization signal, the software will generate a synchronization signal according to the moment when the image of each surface is triggered. The generation of the synchronization signal indicates that all surfaces have received the image data. At the same time, the software sends a volume data acquisition request to the WMS through the HTTP / POST protocol to obtain the volume data measured by the WMS. The software sends the volume data fed back by the WMS to the WMS. The data is temporarily stored for subsequent information aggregation and uploading; when the positioning and decoding threads of each surface receive the synchronization signal, a frame of image will be taken out from the image cache, and the image features will be calculated, and then the template image features and the input image features will be matched; based on the optimal feature point pair found by feature matching, all suspected label positions are calculated; all suspected label area images are sequentially extracted from the input image using the suspected label positions, and the suspected label area images are transformed into suspected label images of the same size as the template image through affine transformation; based on the information area marked during the template creation process, the position of all information to be decoded can be determined in the suspected label image, and the information position image can be obtained based on the decoded information position, and then different information is sent to different information decoding processes for decoding according to the information categories (characters, barcodes, QR codes, etc.) of different information fields in the template; all decoded result information will be numbered according to different input images, and after all input images are decoded, the data will be aggregated and encoded according to the rules required by the user, and finally the aggregated information will be uploaded to WMS through the HTTP / POST protocol.

[0106] like Figure 4 As shown, as an embodiment provided by the present invention, preferably, the package multi-surface distributed dense label visual feature positioning system includes system task scheduling management, video image acquisition, digital image processing, template management and maintenance, WMS network communication, file and data management and other modules. Among them:

[0107] The system task scheduling management module is a bridge and transfer station for the information exchange between the system software user interface and the various modules of the system. The system task scheduling management module mainly includes the system module scheduling module, the system parameter setting module, and the system data visualization module. A: The system module scheduling module is mainly used to realize the initialization, data synchronization, scheduling management, and data interaction between the various functional modules of the system and the software user interface. Its core function is to control the software decoding process, such as the preparation of the decoding environment, the start and stop of the decoding process, the post-processing and distribution of the decoding information, etc.; B: The system parameter setting module is implemented through the system option interface, and mainly completes the global parameter setting of the software system, including device data, network communication parameters, label positioning decoding global parameters, etc.; C: The system data visualization module is to intuitively present and express the data in the software user interface during the software operation process. The visualized data mainly includes the video images of the triggered acquisition, the result information of the information decoding and packaging, and the system operation log information.

[0108] The video image acquisition module mainly includes the camera device management module, the camera device parameter setting module, and the camera image acquisition module. A: The camera device management module is used to realize the enumeration, binding, and control of camera devices, among which the camera enumeration is to traverse all available devices through the API interface function provided by the camera device supplier, the device binding is to bind and set each camera with its corresponding product package surface, and the device control is to connect and disconnect the device, open or close the acquisition, etc.; B: The camera device parameter setting module is to query and configure some parameters of the camera through the API interface function provided by the camera device supplier, such as the trigger source, trigger mode, exposure, contrast, area of ​​interest and other parameters; C: The camera image acquisition module is to create an independent image acquisition thread for each camera, and call the API interface function related to the camera device image acquisition through callback to realize the process of obtaining the video frame image obtained by triggering the acquisition from the camera device. The obtained video frame image will be sent to the main interface video split-screen display window through the system task scheduling management module for display, and on the other hand, it will be placed in the image cache and used as the input image for label positioning.

[0109] The digital image processing module is the core functional module for realizing the multi-sided label scanning function of product packages. It is an important component that determines the performance and production efficiency of the system software. It mainly includes the label positioning module and the information decoding module. A: The label positioning module extracts features from the sub-template image pre-created by the template management and maintenance module and the video frame image triggered by the video image acquisition module, and performs feature matching according to certain rules to find the optimal matching point pair, and then calculates all suspected label positions in turn according to the optimal matching point pair, and obtains all suspected label images of the same size and specification as the sub-template image through coordinate and affine transformations; B: The information decoding module extracts the information area from the suspected label image extracted above according to the valid information field area specified when creating the sub-template, and then performs different information decoding processes according to the valid information category (character, barcode, QR code, etc.), such as performing optical character recognition on character information to obtain character information, and performing barcode recognition and decoding on barcode or QR code to obtain barcode information. Finally, all the information obtained is packaged in the organizational form required by the user and uploaded to WMS through the WMS network communication module.

[0110] The template management and maintenance module mainly includes the template grouping management module, the sub-template maintenance module, and the template global parameter setting module. The template management and maintenance module is designed mainly for users to complete the creation of sub-templates step by step using the sub-template creation wizard, and then uniformly group the sub-templates according to the decoding task requirements in the template grouping management module. Before the decoding task is started, the user manually selects the template group to load the template. A: The template grouping management module realizes the grouping of the created sub-templates according to the template category, the corresponding surface of the product package, etc., and can also realize the addition, deletion, difference, and modification of the template group; B: The sub-template maintenance module is mainly designed for the addition, deletion, query, and modification of sub-templates, which are specifically completed through the sub-template creation wizard; C: The template global parameter setting module is to set the parameters applicable to all templates, such as the template storage directory, template remote switching parameters, and the maximum number of template field groups.

[0111] The WMS network communication module is designed to realize data interaction between the software system and WMS. The data interaction is carried out through the HTTP protocol. The transmission data content mainly includes obtaining volume data from WMS, receiving WMS template remote switching instructions, and uploading decoding information and template grouping information to WMS. Among them, obtaining volume data from WMS and uploading decoding information and template grouping information to WMS are transmitted by sending HTTP / POST requests to the WMS server. Template grouping information is uploaded after the template grouping operation is completed in the template management and maintenance module template grouping management; receiving WMS template remote instructions is to create an HTTP server for template remote switching in the software system, and then WMS sends HTTP / POST requests to the software system to initiate the remote template switching task.

[0112] The file and data management module uses a single-instance design to unify the management of various files of different purposes involved in the software system. The path, format, and retention period of the file can be set separately through the software user interface. The files involved mainly include system configuration files, decoding information files, decoding abnormality image files, sub-template description files, etc. Among them, the system configuration file includes software configuration files and decoding configuration files. The software configuration file is a file used to save the software user configuration parameters during the software startup loading process. The decoding information file and the decoding configuration file are files that describe and record the decoding tasks to avoid users repeatedly configuring the same decoding tasks.

[0113] Embodiment 4:

[0114] Based on embodiments 1 to 3, Figure 5 As shown, the processing flow of the product package multi-sided label scanning software system is mainly divided into image acquisition processing modules S7, S8, S9, S10, S11 and information comprehensive processing module S19.

[0115] Among them, the image acquisition and processing module includes single-sided image processing modules for the five sides of the product packaging box, specifically including the front processing module S7, the back processing module S8, the left processing module S9, the right processing module S10, and the top processing module S11. Each single-sided image processing module is composed of an image acquisition module, a label positioning module, and an information decoding module. Each single-sided image processing module synchronously receives and responds to the external trigger signal S1, obtains the image data of the current side from the industrial camera through the image acquisition module, and sends it to the label positioning module for current side label positioning. The located label is sent to the information decoding module for decoding and extraction of specific information, and the obtained information is sequentially transmitted to the information comprehensive processing module S19 to generate the information sequence of the current side of the packaging box.

[0116] The information integration processing module S19 receives the product information processed by the image acquisition processing module, generates the integrated information of multiple surfaces of the entire packaging box through the information integration S17, and sends or uploads it through the information sending module S18.

[0117] As an embodiment provided by the present invention, preferably, Figure 6 As shown, each single-side image processing module in the information processing module in the product package multi-side label scanning software system processing flow includes an image acquisition module 21, a label positioning module 31, and an information decoding module 41, wherein:

[0118] The image acquisition module 21 is composed of an image acquisition module 211, a frame synchronization counter 212, and an image cache 213. 211 is used to respond to an external trigger signal to control the industrial camera to collect image data, and generate a frame synchronization count value to be sent to the frame synchronization counter 212, and the collected image data is sent to the image cache 213. As shown in 214, the image cache 213 adopts a "first-in-first-out" structure to ensure that the subsequent processing links obtain the earliest collected image from the image cache. The frame synchronization count value generated by the frame synchronization counter 212 is used for information synchronization and packaging of all information sequences of the single-sided image processing module.

[0119] The label positioning module 31 is used to process the cached image data and dynamically create a label positioning process, an information extraction process, a label sequence cache and an information sequence cache according to the label template information preset by the user. The process can respectively locate the position and quantity information of different categories of labels in the image, and the obtained label image information is sequentially stored in the label sequence cache. The number of label positioning processes, information extraction processes, label sequence caches and information sequence caches created depends on the number of label categories preset by the user's label template. The number of label categories set in this embodiment is m. Figure 2 As shown, 311 represents the positioning process of one category label. The label sequence cache adopts the "first in first out" technology to cache the label image obtained by the positioning process and other related information used to characterize the label. The cache depth represents the number of labels. 313 represents the label sequence cache of the mth label positioning process. In this embodiment, the number of labels obtained by the positioning process of the label positioning process is n.

[0120] The information decoding module 41 is used to sequentially obtain the located label images from the label sequence cache, and create an information extraction process based on the field information in the user label template. The process can identify or decode the information or barcode at a specific position in the label image, and the processed information fields are sequentially stored in the information sequence cache. Figure 6As shown, 411 represents the information extraction process of one category label. The information sequence cache adopts the "first in first out" technology to cache the field information obtained by each information extraction process identification or decoding, and the cache depth represents the number of information fields. 413 represents the information sequence cache of the mth information extraction process. In this embodiment, the number of information fields obtained by the information extraction process is k.

[0121] All information sequences obtained by the information decoding module 41 are finally processed by the information comprehensive processing module S19 to obtain the product information of the entire product packaging box, which is used to send or transmit to other warehouse management systems.

[0122] A method for visual feature positioning and information decoding of distributed dense labels on multiple sides of a package. The label positioning algorithm processing is a process of extracting all suspected label images that are highly similar to the template image by using an image processing algorithm to process the video images triggered and collected from each side of the product package in a streamlined manner; the information decoding algorithm processing is a process of decoding all suspected label images obtained by the label positioning algorithm in sequence according to the template field information defined by the user by using various information decoding algorithms to obtain specific information, and finally combining and packaging the information according to the rules set by the user to obtain structured data containing comprehensive information of all sides of the product package; other algorithms involved in the system software are mainly manifested in some intermediate processing links The algorithm design optimization aims to improve the overall operating efficiency and performance of the system; by extracting the feature points of the input image and the template label image respectively, and then performing feature point matching to locate the suspected target label area; by comparing the feature descriptors of the input image and the template image, similar feature point pairs are found, and the feature point set of the suspected label area of ​​the input image is calculated. From image acquisition, label positioning to information decoding, they are all carried out in an independent process to realize the data acquisition and processing pipeline structure, thereby improving production efficiency. By utilizing image feature detection technology, the images collected from all directions of the product package are processed separately, and the suspected label positions in the images of each side of the package are obtained, thereby achieving the purpose of dense label positioning on multiple sides of the package.

[0123] In the description of this specification, the description with reference to the terms "one embodiment", "example", "specific example", etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0124] The preferred embodiments of the present invention disclosed above are only used to help illustrate the present invention. The preferred embodiments do not describe all the details in detail, nor do they limit the invention to the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the present invention, so that those skilled in the art can understand and use the present invention well. The present invention is limited only by the claims and their full scope and equivalents.

Claims

1. A method for locating visual features of multi-sided distributed dense labels of a package, characterized in that: The following steps are involved: Step S1: creating an input image cache for storing the collected images of each surface of the product package and a label image cache for storing the label image obtained by positioning; Step S2: Calculate the feature information of each type of label image; Step S3: Create and start a label positioning thread, extract feature points of the input image and the template label image, and locate the suspected target label area through feature point matching, obtain the suspected label position and extract the suspected label image.

2. According to claim 1, a method for locating multi-surface distributed dense label visual features of a package is characterized in that: The input image cache and label image cache adopt a "first-in-first-out" structure, and the depth and data structure of the input image cache and label image cache can be configured. Data access and communication are performed through memory sharing between different threads; the images of each surface of the product package collected by the camera acquisition thread are cached in the corresponding input image cache respectively, and the label positioning thread takes out the images from the input image cache in turn for processing, and the processed label images are cached in the label image cache of the corresponding template.

3. The method for locating multi-surface distributed dense label visual features of a package according to claim 1, characterized in that: In step S3: the label positioning thread sequentially takes out the input image from the input image cache to extract and match the image feature points, obtains all the regions similar to the template label image from the input image, extracts the similar image regions and sequentially performs affine transformation and size transformation to obtain the suspected label image of the same specification and size as the template label image.

4. The method for locating multi-surface distributed dense label visual features of a package according to claim 3, characterized in that: In the step S3: the SURF algorithm is used to extract image feature points from the input image.

5. The method for locating multi-surface distributed dense label visual features of a package according to claim 4, characterized in that: In the step S3: the approximate nearest neighbor matching algorithm FLANN is used to obtain similar feature point pairs, and the best matching feature point pairs between the template label and the input image are obtained through the distance criterion.

6. The method for locating multi-surface distributed dense label visual features of a package according to claim 3, characterized in that: The method of the affine transformation is: Get the best matching feature point pair; Calculate and obtain the feature point set of the suspected label area of ​​the input image; Use the RANSAC algorithm to calculate the homography transformation matrix; The irregular quadrilateral region in the input image is transformed into a regular image with the same scale and shape as the template label image using a transformation matrix.

7. The method for locating multi-surface distributed dense label visual features of a package according to claim 3, characterized in that: The similarity judgment method between the suspected label image and the template label image is: The judgment is made through the hash fingerprint and SSIM index of the image.

8. The method for locating multi-surface distributed dense label visual features of a package according to claim 7, characterized in that: Similarity judgment based on the hash fingerprint of an image includes the following steps: The hash fingerprints of the suspected label image and the template label image are designed and verified as 64-bit binary coded values; By calculating the correlation between the hash fingerprints of the two images, it is determined whether the located suspected label image is a valid label image.

9. The method for locating multi-surface distributed dense label visual features of a package according to claim 7, characterized in that: The SSIM index obtains the final similarity index by calculating the weighted product of the brightness factor, contrast factor and structure factor of the image; wherein the brightness factor and contrast factor are calculated by the mean and variance, and the structure factor is calculated by the covariance.

10. A method for decoding multi-surface distributed dense label information of a package, characterized in that: The information decoding method is used to decode the suspected label image obtained by the visual feature positioning method described in any one of claims 1-9.

Images