An image processing-based warehouse article intelligent recognition and classification method, system and related device

Abstract

The application discloses a warehouse article intelligent recognition and classification method and system based on image processing and related equipment, and relates to the technical field of computer vision, which comprises the following steps: acquiring an RGB image and a depth image of a scene to be recognized; performing region segmentation, determining the occlusion relationship between the regions of articles by using depth information, and analyzing the occlusion of each article region; performing spatial partitioning on each article region and extracting multi-modal features, and performing weighted fusion on the features of each partition according to the occlusion; matching and recognizing each article region with an article feature library, performing reliability evaluation on the recognition result, and outputting a candidate recognition result and a reliability mark; performing correlation correction by using the co-occurrence relationship between article categories to determine a final recognition result; and updating the article feature library according to the final recognition result. Through the occlusion perception mechanism based on depth information and the priority-driven recognition strategy, the application improves the recognition accuracy of warehouse articles in an occlusion scene.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, and in particular to a warehouse item intelligent recognition and classification method and system based on image processing and related equipment. BACKGROUND

[0002] With the rapid development of e-commerce logistics and intelligent warehousing, the demand for automatic recognition and classification of warehouse items is increasingly urgent. Traditional manual identification methods are inefficient and prone to errors, and automatic identification schemes based on barcode scanning have insufficient recognition rates in cases of barcode damage, obstruction, etc. Computer vision technology provides a new solution for warehouse item recognition, enabling automatic classification through image acquisition, feature extraction, and pattern matching, and has become an important research direction in this field.

[0003] Existing image recognition methods face some challenges in practical applications. The lighting conditions in the warehouse environment are complex and changeable, affecting the quality of image acquisition and recognition stability. Items are stacked and placed in a way that blocks each other, increasing the difficulty of item segmentation and feature extraction. Existing methods rely on a single type of image information and features, and have limited adaptability to complex scenarios. At the same time, the recognition system usually lacks an evaluation and feedback mechanism for the accuracy of the results, making it difficult to ensure the reliability of the recognition. SUMMARY

[0004] The present application provides a warehouse item intelligent recognition and classification method based on image processing, which solves the technical problems of unstable image quality under complex lighting conditions, difficulty in recognizing items in occlusion scenarios, and insufficient reliability of recognition results in the prior art.

[0005] The first aspect of the present application provides a warehouse item intelligent recognition and classification method based on image processing, comprising:

[0006] Obtaining an RGB image and a depth image of a scene to be recognized, and performing illumination normalization on the RGB image based on the depth image;

[0007] Performing region segmentation based on the depth image and the RGB image to obtain a plurality of item regions, determining the occlusion relationship between the item regions using depth information, and analyzing the occlusion conditions of each item region;

[0008] Partitioning each item region in space and extracting multi-modal features, weighting and fusing the features of each partition according to the occlusion conditions to obtain a feature representation vector;

[0009] Matching and recognizing each item region with an item feature library based on barcode information and the feature representation vector, evaluating the reliability of the recognition results, and outputting the candidate recognition results and reliability labels;

[0010] According to the reliability mark, the candidate recognition results of each article region are associated and corrected by using the co-occurrence relationship between article categories to determine a final recognition result;

[0011] The article feature library is updated according to the final recognition result.

[0012] Optionally, the RGB image is subjected to illumination normalization based on the depth image, including:

[0013] The depth image is subjected to filtering processing according to the depth confidence information and the neighborhood depth value, and the depth missing region is repaired;

[0014] The RGB image is subjected to global correction, and the corrected RGB image is converted to the LAB color space to separate the luminance component and the color component;

[0015] The luminance component is divided into multiple depth layers based on the depth image, and the adaptive histogram equalization processing is performed on each depth layer;

[0016] The color component is kept unchanged, the processed luminance component and the color component are recombined and converted back to the RGB color space to obtain the illumination normalized RGB image;

[0017] The illumination normalized RGB image is subjected to local contrast enhancement, and the mirror reflection region is detected and repaired.

[0018] Optionally, the occlusion condition of each article region is analyzed, including:

[0019] The depth image is subjected to initial region segmentation based on the depth continuity to obtain a preliminary segmentation region;

[0020] The boundary of the preliminary segmentation region is subjected to fine processing based on the color edge information of the RGB image to obtain a finely segmented article region;

[0021] The contact boundary between each article region is recognized, the occlusion relationship between adjacent article regions is determined based on the depth information at the contact boundary, and an occlusion relationship graph is constructed;

[0022] According to the occlusion relationship graph and the contact boundary, the occlusion boundary position of each article region is located, and the occlusion boundary type is labeled;

[0023] According to the occlusion boundary position and the article region, a visible part mask of each article region is generated, and the occlusion metric information is determined based on the visible part mask.

[0024] Optionally, each partition feature is weighted and fused according to the occlusion condition, including:

[0025] Each article region is spatially partitioned according to the occlusion condition, and the visible region range of each partition is determined based on the visible part mask.

[0026] detecting the barcodes of each partition, and extracting barcode information if a barcode is detected;

[0027] extracting visual features, geometric features, depth features, and semantic features of each partition;

[0028] According to the position of the occlusion boundary, the partition adjacent to the occlusion boundary is marked as a boundary partition;

[0029] Weighted fusion of features of each partition based on boundary marking and occlusion severity level to obtain feature representation vector of each item region.

[0030] Optionally, each item region is matched and recognized with an item feature library, including:

[0031] According to the barcode detection result of each item region and the feature library matching situation, the item region is prioritized;

[0032] Different recognition strategies are used to recognize different priority item regions;

[0033] The confidence of the recognition result is evaluated, and the reliability is judged in combination with the occlusion measurement information, and the candidate category, confidence and reliability mark of each item region are output.

[0034] Optionally, the co-occurrence relationship between item categories is used to associate and correct the candidate recognition result of each item region, including:

[0035] An item category co-occurrence probability matrix is obtained;

[0036] For the item region to be associated and verified, the co-occurrence probability of the candidate category and other recognized categories in the same scene is calculated, and the confidence is adjusted according to the co-occurrence probability;

[0037] For the item region to be manually reviewed, if there are multiple candidate categories, the candidate categories are sorted according to the co-occurrence probability, and the category with the highest co-occurrence probability is taken as the recommended category;

[0038] The final recognition category, confidence and reliability mark of each item region are output.

[0039] Optionally, the occlusion measurement information is determined based on the visible part mask, including:

[0040] A binary mask is generated on each item region according to the position of the occlusion boundary;

[0041] The proportion of visible pixels of each item region is counted, and the occlusion degree is calculated;

[0042] The occlusion severity level is divided according to the occlusion degree;

[0043] The potential key region of each article region is identified based on image feature analysis, the overlap of the potential key region and the visible part mask is analyzed, and key region visibility analysis is generated;

[0044] The occlusion degree, occlusion severity level and key region visibility analysis are integrated to generate occlusion metric information of each article region.

[0045] The second aspect of the present application provides an image processing-based intelligent identification and classification system for warehouse articles, comprising:

[0046] The illumination normalization module is configured to obtain an RGB image and a depth image of a scene to be identified, and perform illumination normalization processing on the RGB image based on the depth image;

[0047] The occlusion-aware region segmentation module is configured to perform region segmentation based on the depth image and the RGB image to obtain a plurality of article regions, determine the occlusion relationship between the article regions using depth information, and analyze the occlusion of each article region;

[0048] The partition feature fusion module is configured to perform spatial partitioning on each article region and extract multi-modal features, and perform weighted fusion on each partition feature according to the occlusion to obtain a feature representation vector;

[0049] The hierarchical matching and identification module is configured to match and identify each article region with an article feature library based on barcode information and the feature representation vector, evaluate the reliability of the identification result, and output a candidate identification result and a reliability label;

[0050] The correlation correction decision module is configured to correct the candidate identification result of each article region according to the reliability label and the co-occurrence relationship between article categories to determine a final identification result;

[0051] The feature library dynamic updating module is configured to update the article feature library according to the final identification result.

[0052] The third aspect of the present application provides a computer device, comprising a memory and a processor, the memory stores a computer program, and the computer program is executed by the processor to execute the steps of the image processing-based intelligent identification and classification method for warehouse articles according to any one of the above aspects.

[0053] The fourth aspect of the present application provides a computer readable storage medium, which stores a computer program, and the computer program is executed to implement the steps of any one of the above aspects.

[0054] Beneficial effects: The present application combines RGB images and depth images for illumination normalization processing and region segmentation, uses depth information to determine occlusion relationship and weighted fusion of features, effectively solves the recognition problem under complex lighting and occlusion of goods scene. Through reliability evaluation of the recognition result, and using the co-occurrence relationship of the goods category for correlation correction, the recognition accuracy and system robustness are improved. By dynamically updating the goods feature library according to the recognition result, the system has self-adaptive ability and can continuously optimize the recognition performance. The overall scheme combines illumination processing, occlusion analysis, multi-modal feature fusion, reliability evaluation and correlation correction, which significantly improves the accuracy, reliability and practicability of intelligent recognition of warehouse goods. BRIEF DESCRIPTION OF DRAWINGS

[0055] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiment description. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0056] Figure 1 A flowchart of a warehouse goods intelligent recognition and classification method based on image processing provided by the embodiment of the present application.

[0057] Figure 2 A depth-guided illumination normalization flowchart of a warehouse goods intelligent recognition and classification method based on image processing provided by the embodiment of the present application.

[0058] Figure 3 A partition feature fusion flowchart of a warehouse goods intelligent recognition and classification method based on image processing provided by the embodiment of the present application.

[0059] Figure 4 A structure block diagram of a warehouse goods intelligent recognition and classification system based on image processing provided by the embodiment of the present application. DETAILED DESCRIPTION

[0060] The present application provides a warehouse goods intelligent recognition and classification method based on image processing, which is used to solve the technical problems of unstable image quality under complex lighting conditions, difficult recognition of goods occlusion scene and insufficient reliability of recognition results in the prior art.

[0061] In order to make the inventive purposes, features and advantages of the present application more obvious and easy to understand, the technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. Obviously, the embodiments described below are only some of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0062] As shown in Figure 1 , the present application provides a warehouse item intelligent recognition and classification method based on image processing, comprising:

[0063] S1: acquiring an RGB image and a depth image of a scene to be recognized, and performing illumination normalization processing on the RGB image based on the depth image.

[0064] Specifically, the depth-guided illumination normalization flowchart is as shown in Figure 2 , comprising:

[0065] S1.1: acquiring an RGB image, a depth image and a depth confidence map of a scene to be recognized.

[0066] Wherein, the pixel positions of the RGB image and the depth image correspond one by one.

[0067] S1.2: filtering processing the depth image according to the depth confidence information and the neighborhood depth value, and repairing the depth missing area.

[0068] In one embodiment, the low-confidence pixel area is marked, isolated abnormal depth points are detected and removed, spatial filtering of edge preservation is performed on the depth image to smooth noise, and interpolation repair is performed on small-area depth missing areas.

[0069] S1.3: performing global correction on the RGB image, converting the corrected RGB image to the LAB color space, and separating to obtain the luminance component and the color component.

[0070] Specifically, the global correction includes white balance processing and exposure correction.

[0071] S1.4: dividing the luminance component into multiple depth layers based on the depth image, and performing adaptive histogram equalization processing on each depth layer.

[0072] Further, the depth value distribution of the depth image is analyzed to determine a depth range and divide into N depth layers; according to the depth value of each pixel in the depth image, each pixel of the luminance component is assigned to a corresponding depth layer; the luminance pixels of each depth layer are respectively subjected to adaptive histogram equalization processing to obtain equalized luminance values of each depth layer; according to the depth layer attribution of each pixel, the original luminance values are replaced with the equalized luminance values to generate a processed luminance component.

[0073] In one embodiment, the value of N is determined according to the depth range of the scene, preferably 3-10.

[0074] In another embodiment, the adaptive histogram equalization adopts a limited-contrast adaptive histogram equalization algorithm, and a contrast limit threshold is set to avoid over-enhancement.

[0075] S1.5: The color components remain unchanged, the processed luminance component is recombined with the color components and converted back to the RGB color space to obtain a light normalized RGB image.

[0076] S1.6: The light normalized RGB image is subjected to local contrast enhancement to detect and repair the mirror reflection region.

[0077] In one preferred embodiment, the dark region in the light normalized RGB image is subjected to local contrast enhancement to improve the visibility of details, while the mirror reflection region is detected and repaired based on the neighborhood pixels. The mirror reflection region is identified by luminance threshold detection and appears as a locally overexposed highlight region.

[0078] Through the above processing, this step respectively adjusts the luminance of the objects at different depth positions based on the depth-layered light normalization method, solves the problem that the traditional global light correction cannot take into account the foreground and background objects, and makes the same type of objects at different spatial positions present consistent visual features.

[0079] S2: Based on the depth image and the RGB image, a plurality of object regions are obtained by region segmentation, the occlusion relationship between the object regions is determined by using the depth information, and the occlusion of each object region is analyzed.

[0080] Specifically, S2 includes:

[0081] S2.1: The depth image is subjected to initial region segmentation based on depth continuity to obtain preliminary segmentation regions.

[0082] S2.2: The boundaries of the preliminary segmentation regions are subjected to refinement processing based on the color edge information of the RGB image to obtain finely segmented object regions.

[0083] In one embodiment, color edges inside the preliminary segmented region are detected, and when the average depth value difference between the regions on both sides of the color edge is less than the average depth gradient at the boundary of the region, and the color edge penetrates the region, the region is further segmented along the color edge, avoiding the merging of multiple independent items with similar depths into a single region.

[0084] S2.3: Identify the contact boundary between each item region, determine the occlusion relationship between adjacent item regions based on the depth information at the contact boundary, and construct an occlusion relationship graph.

[0085] Specifically, shared boundary segments between finely segmented item regions are detected; the depth difference and depth gradient on both sides of the boundary segment are extracted, the front and rear relationship of adjacent regions is determined according to the depth gradient direction, and the region with smaller depth value is determined as the occlusion side; the occlusion determination of the boundary segment is evaluated for reliability, and unreliable boundaries with too small average depth gradient value or too low depth gradient direction consistency are filtered; based on the reliable occlusion relationship, a directed occlusion relationship graph between item regions is constructed.

[0086] S2.4: According to the occlusion relationship graph and the contact boundary, the occlusion boundary position of each item region is located, and the occlusion boundary type is labeled.

[0087] Further, according to the occlusion relationship graph, the occlusion relationship of each item region is determined, for each pair of occlusion relationships, the occlusion boundary position of the occluded item is labeled on the corresponding contact boundary segment; the spatial distribution of all occlusion boundaries of each item region is analyzed, and the direction of the occlusion boundary is determined; according to the direction distribution characteristics of the occlusion boundary, the occlusion type is divided into single-sided occlusion, multi-sided occlusion and surrounding occlusion; for each item region, all occlusion boundary positions and occlusion boundary types are integrated to generate a complete occlusion boundary description.

[0088] Wherein, single-sided occlusion refers to the occlusion boundary concentrated in one direction of the item region, multi-sided occlusion refers to the occlusion boundary distributed in two or more non-continuous directions, and surrounding occlusion refers to the occlusion boundary distributed around the periphery of the item region.

[0089] S2.5: According to the occlusion boundary position and the item region, a visible part mask of each item region is generated, and occlusion metric information is determined based on the visible part mask.

[0090] Preferably, a binary mask is generated on each item region according to the occlusion boundary position, and the occluded part and the visible part are marked respectively; the proportion of visible pixels in each item region is counted, and the occlusion degree is calculated; the occlusion severity level is divided according to the occlusion degree; the potential key region of each item region is identified based on image feature analysis, and the potential key region includes a texture complex region, an edge dense region and a high contrast region; the overlap of the potential key region and the visible part mask is analyzed to generate key region visibility analysis; the occlusion degree, the occlusion severity level and the key region visibility analysis are integrated to generate occlusion measurement information of each item region.

[0091] In one embodiment, the edge dense region is identified by gradient amplitude statistics, the texture complex region is identified by local variance analysis, and the high contrast region is identified by brightness contrast calculation. These regions usually contain identification key features such as barcodes, characters, brand logos, etc.

[0092] In another embodiment, the occlusion degree less than 30% is marked as light occlusion, the occlusion degree between 30% and 70% is marked as moderate occlusion, and the occlusion degree greater than 70% is marked as heavy occlusion. In other embodiments, the above threshold values can be adjusted according to the actual application scenario.

[0093] Based on the above processing, the region segmentation is performed by fusing depth continuity and color edge information, which avoids the false merging of items with similar depths and the false segmentation of items with similar colors in single modal segmentation; the directed occlusion relationship graph is constructed based on the depth information of the contact boundary, and the visible part mask and the occlusion measurement information of each item region are generated, which clearly defines the occlusion type, the occlusion degree and the key region visibility of each item region, and provides accurate occlusion analysis results for subsequent feature weighting and priority division.

[0094] S3: Spatially partitioning each item region and extracting multi-modal features, and weighting and fusing each partition feature according to the occlusion condition to obtain a feature representation vector.

[0095] Specifically, the partition feature fusion flowchart is as shown in Figure 3 , which includes:

[0096] S3.1: Spatially partitioning each item region according to the occlusion condition, and determining the visible region range of each partition based on the visible part mask.

[0097] In one embodiment, according to the occlusion severity level in the occlusion metric information, the partition strategy of each item region is determined: the item region with a light occlusion severity level adopts fine partition, the item region with a moderate or heavy occlusion severity level adopts coarse partition, and the fine partition and the coarse partition are implemented by different grid division densities; the item region is divided into grids according to the partition strategy; the number of visible pixels of each grid partition is calculated based on the visible part mask; invalid partitions with a number of visible pixels less than an effective pixel threshold are filtered, and the effective pixel threshold is set to 500 pixels, which is determined according to the minimum input size requirement of the feature extraction algorithm; the remaining effective partitions are numbered, and the effective partition list of each item region and the visible area range of each partition are output.

[0098] S3.2: Bar code detection is performed on each partition, and if a bar code is detected, bar code information is extracted.

[0099] Specifically, bar code detection and decoding are performed on the visible area of each partition; if decoding is successful, the bar code content and the detection confidence are recorded, and the detection confidence is output by the bar code recognition algorithm; if multiple partitions of the same item region detect the same bar code, the highest detection confidence is taken as the final confidence; if multiple partitions of the same item region detect different bar codes, it is marked as a bar code anomaly; the bar code information, the detection confidence or the no-bar code mark of each item region are output.

[0100] In one embodiment, the bar code detection adopts the ZBar or ZXing algorithm, and the detection confidence is determined by the decoding quality score output by the algorithm, with a value range of 0-1.

[0101] S3.3: Visual features, geometric features, depth features and semantic features are extracted from each partition.

[0102] Among them, the visual features include color features and texture features, the geometric features include shape features, the depth features include depth statistical features and depth geometric features, and the semantic features include text features and identification features.

[0103] S3.4: According to the occlusion boundary position, the partitions adjacent to the occlusion boundary are marked as boundary partitions.

[0104] S3.5: The features of each partition are weighted and fused based on the boundary mark and the occlusion severity level, and the feature representation vector of each item region is obtained.

[0105] Preferably, a fusion weight is assigned to the features of each partition: the feature weight of a boundary partition is lower than that of a non-boundary partition; the weight of each type of feature is adjusted according to the occlusion severity level of the item region: when the occlusion severity level is mild, the weights of the visual features and the geometric features are higher; when the occlusion severity level is moderate or severe, the weight of the depth feature is higher; the weights of each type of feature are normalized; and the weighted features of all valid partitions in each item region are fused to generate a feature representation vector of the item region.

[0106] By using the above processing, the partition strategy is dynamically adjusted according to the occlusion severity level, and the multi-modal feature extraction and differential weighted fusion are combined, so that the fine features of the mildly occluded item can be fully utilized, and the spatial features such as depth are mainly used for the heavily occluded item, thereby avoiding feature redundancy or feature loss caused by a unified processing strategy, and improving the feature representation quality of items with different occlusion levels.

[0107] S4: Based on the barcode information and the feature representation vector, each item region is matched and recognized with the item feature library, the reliability of the recognition result is evaluated, and a candidate recognition result and a reliability mark are output.

[0108] Specifically, S4 includes:

[0109] S4.1: Obtain the item feature library.

[0110] It should be noted that the item feature library stores the barcode information, the feature representation vector, the key feature region label, the physical size and the category label of each type of item.

[0111] In one embodiment, the establishment process of the item feature library includes: collecting the barcode information of each type of item, establishing a mapping relationship between the barcode information and the category label; collecting the standard image of each type of item, and extracting the feature representation vector; labeling the key feature region of each type of item, the key feature region including the trademark region, the pattern region or the text region; recording the physical size of each type of item; and establishing an index and storing it in the item feature library. The establishment of the feature library can be completed in an offline manner, and supports dynamic updating and expansion.

[0112] S4.2: Prioritize the item regions according to the barcode detection results of each item region and the matching of the feature library.

[0113] Specifically, the item regions are divided into three priorities according to the barcode detection confidence and the matching of the barcode in the feature library: the item regions with the barcode detection confidence higher than a first preset threshold and successfully matched to the corresponding category in the item feature library are divided into a first priority; the item regions with the barcode detection confidence higher than the first preset threshold but not matched to the corresponding category in the item feature library are divided into a second priority; the item regions with the barcode detection confidence between a second preset threshold and the first preset threshold are divided into a third priority; and the item regions without detecting the barcode or with the barcode detection confidence lower than the second preset threshold are divided into a fourth priority.

[0114] In an embodiment, the first preset threshold is set to 0.85, representing the standard of clear and reliable barcode; and the second preset threshold is set to 0.50, representing the minimum standard of usable barcode. The above-mentioned thresholds are set by comprehensively considering the reliability of the barcode detection technology and the actual distribution of the barcode quality in the warehouse scene. In other embodiments, the above-mentioned thresholds can be adjusted according to the accuracy of the barcode detection device and the quality of the item barcode.

[0115] S4.3: Different identification strategies are adopted for the item regions of different priorities for identification.

[0116] Specifically, the barcode matching is performed on the item regions of the first priority: the corresponding commodity category of the barcode is queried in the item feature library, and the commodity category and the barcode detection confidence are output.

[0117] Further, the barcode information output and visual matching are performed on the item regions of the second priority: the barcode information and the unmatching mark of the feature library are output, and the feature representation vector of the item region is calculated with the feature vectors of each commodity category in the item feature library for similarity, and the candidate category list and the similarity of each candidate category are output as auxiliary reference information for assisting the artificial judgment of the commodity category or the detection of barcode recognition error.

[0118] Further, the barcode matching and visual verification are performed on the item regions of the third priority: the corresponding commodity category of the barcode is queried in the item feature library; if the matching is successful, the standard feature vector of the commodity category is obtained, the visual similarity between the standard feature vector and the feature representation vector of the item region is calculated, and the commodity category, the barcode detection confidence and the visual similarity are output; if the matching fails, the barcode information and the matching failure mark are output, and the visual matching is performed, the feature representation vector of the item region is calculated with the feature vectors of each commodity category in the item feature library for similarity, and the candidate category list and the similarity of each candidate category are output.

[0119] Further, the visual matching is performed on the item regions of the fourth priority: the feature representation vector of the item region is calculated with the feature vectors of each commodity category in the item feature library for similarity, and the candidate category list and the similarity of each candidate category are output.

[0120] In one embodiment, the similarity calculation employs cosine similarity or Euclidean distance for measurement.

[0121] S4.4: Confidence evaluation is performed on the recognition result, and the reliability is judged in combination with the occlusion measurement information, and the candidate category, confidence and reliability mark of each item region are output.

[0122] Specifically, for the recognition result of the first priority item region, the bar code detection confidence is taken as the final confidence; and the reliability is judged according to the occlusion severity level: if the occlusion severity level is light, the reliability mark is directly confirmed; if the occlusion severity level is moderate or severe, the reliability mark is to be associated with verification.

[0123] Further, for the recognition result of the second priority item region, the bar code information is taken as the main recognition result, the bar code detection confidence is taken as the final confidence, and the reliability mark is to be manually reviewed (suggested to update the feature library); and the visual matching result is taken as auxiliary reference information and output together.

[0124] Further, for the recognition result of the third priority item region, the bar code matching situation is processed according to the bar code matching situation: if the bar code is successfully matched in the item feature library, the final confidence is calculated according to the bar code detection confidence and the visual similarity; the reliability is judged according to the visual similarity: if the visual similarity is higher than the verification threshold, the reliability mark is directly confirmed; if the visual similarity is lower than the verification threshold but not lower than the minimum verification threshold, the reliability mark is to be associated with verification; if the visual similarity is lower than the minimum verification threshold, the mark is feature abnormal, and the reliability mark is to be manually reviewed.

[0125] Further, if the bar code fails to match in the item feature library, the confidence of the candidate category of the visual matching is evaluated: the difference between the highest similarity and the second highest similarity is calculated; if the highest similarity is not lower than the similarity threshold and the difference is not lower than the minimum distinction threshold, the highest similarity is taken as the confidence, and the reliability mark is to be associated with verification; if the highest similarity is lower than the similarity threshold or the difference is less than the minimum distinction threshold, the reliability mark is to be manually reviewed.

[0126] Further, for the visual matching result of the fourth priority item region, firstly, the occlusion severity level is judged: if the occlusion severity level is severe, it is directly marked as to be manually reviewed; if the occlusion severity level is mild or moderate, the difference between the highest similarity and the second highest similarity is calculated for further evaluation: when the occlusion severity level is mild, if the highest similarity is not lower than the high-trust similarity threshold and the difference is greater than the distinguishability threshold, the highest similarity is taken as the confidence, and the reliability is marked as directly confirmable; if the highest similarity is lower than the high-trust similarity threshold but not lower than the similarity threshold, and the difference is not lower than the minimum distinguishability threshold, the reliability is marked as to be associated verified; otherwise, the reliability is marked as to be manually reviewed. When the occlusion severity level is moderate, if the highest similarity is not lower than the similarity threshold and the difference is greater than the distinguishability threshold, the highest similarity is taken as the confidence, and the reliability is marked as to be associated verified; otherwise, the reliability is marked as to be manually reviewed.

[0127] Further, the identification results of the item regions are summarized, and the candidate categories, confidence and reliability mark are output.

[0128] In one embodiment, the verification threshold is set to 0.75; the minimum verification threshold is set to 0.50; the high-trust similarity threshold is set to 0.75; the similarity threshold is set to 0.60; the distinguishability threshold is set to 0.15; and the minimum distinguishability threshold is set to 0.08. The above-mentioned threshold settings comprehensively consider the differences between barcode reliability and visual matching confidence, the influence of occlusion on identification, and the appearance similarity characteristics of warehouse items. In other embodiments, the above-mentioned thresholds can be adjusted according to the characteristics of item categories and business needs.

[0129] In the above manner, the item regions are prioritized according to the barcode detection confidence and the feature library matching situation, different identification strategies are adopted for different priorities, and the reliability of the identification results is evaluated in combination with the occlusion measurement information, realizing the synergy and complementarity of barcode identification and visual matching, improving the identification accuracy, and at the same time, distinguishing the item regions that can be directly confirmed, to be associated verified and to be manually reviewed through the reliability mark, providing a basis for subsequent associated correction and manual intervention.

[0130] S5: According to the reliability mark, the candidate identification results of each item region are associated corrected using the co-occurrence relationship between item categories to determine the final identification results.

[0131] Specifically, S5 includes:

[0132] S5.1: Obtain an item category co-occurrence probability matrix;

[0133] It should be noted that the item category co-occurrence probability matrix stores the co-occurrence probabilities between various categories of items, and the co-occurrence probability represents the probability of two categories of items appearing at the same time in the same scene.

[0134] S5.2: For the item region to be verified for association, calculate the co-occurrence probability of the candidate category with other identified categories in the same scene, and adjust the confidence level according to the co-occurrence probability.

[0135] Specifically, the identification category of the item region in the same scene that is marked as directly verifiable is obtained as the reference category; if the reference category is empty, the co-occurrence relationship adjustment is skipped and the original identification result is maintained; otherwise, the co-occurrence probability of the candidate category of the item region to be associated and verified and each reference category is calculated, and the highest co-occurrence probability is taken as the co-occurrence score; if the co-occurrence score is not lower than the co-occurrence threshold, the confidence is multiplied by the co-occurrence gain coefficient to improve it, and the reliability label is modified to be directly verifiable; if the co-occurrence score is lower than the co-occurrence threshold, the original confidence and reliability label are maintained.

[0136] S5.3: For areas of items to be manually reviewed, if there are multiple candidate categories, the candidate categories are sorted according to their co-occurrence probability, and the category with the highest co-occurrence probability is selected as the recommended category.

[0137] Specifically, obtain the reference category; if the reference category is empty, skip the sorting optimization and keep the original candidate category order; otherwise, calculate the co-occurrence score of each candidate category with the reference category; re-sort the candidate categories according to the co-occurrence score; if the category with the highest co-occurrence score is not lower than the co-occurrence threshold and the category originally ranked first in similarity, modify the reliability label to pending association verification; otherwise, keep the original reliability label and output the sorted candidate category list for manual reference.

[0138] S5.4: Summarize and output the final identification category, confidence level, and reliability label for each item region.

[0139] In one embodiment, the co-occurrence threshold is set to 0.40, and the co-occurrence gain coefficient is set to 1.15. These parameters are set to balance the effectiveness of the association strength between item categories with the dominance of the recognition results. In other embodiments, these parameters can be adjusted according to the actual application scenario and the distribution characteristics of item categories.

[0140] Through the above steps, the co-occurrence probability matrix of item categories is used to correct the association between the areas of items to be verified and those to be manually reviewed. The confidence level is adjusted or the ranking of candidate categories is optimized by calculating the co-occurrence relationship between candidate categories and confirmed categories in the scene. The regularity of item pairing in the scene is utilized to improve the accuracy and reliability of the recognition results.

[0141] S6: Update the item feature database based on the final recognition results.

[0142] Beneficial effects: The application effectively solves the identification problem under complex lighting and item occlusion scenes by combining RGB images and depth images for lighting normalization processing and region segmentation, using depth information to determine the occlusion relationship and weighting fusion of features. The identification accuracy and system robustness are improved by reliability evaluation of the identification result and correlation correction using the co-occurrence relationship of item categories. The system has adaptive ability and can continuously optimize the identification performance by dynamically updating the item feature library according to the identification result. The overall scheme organically combines lighting processing, occlusion analysis, multi-modal feature fusion, reliability evaluation and correlation correction, significantly improving the accuracy, reliability and practicality of warehouse item intelligent identification.

[0143] As shown in Figure 4 The application provides a warehouse item intelligent identification and classification system based on image processing, which comprises:

[0144] A lighting normalization module is configured to acquire an RGB image and a depth image of a scene to be identified, and perform lighting normalization processing on the RGB image based on the depth image.

[0145] An occlusion-aware region segmentation module is configured to perform region segmentation based on the depth image and the RGB image to obtain a plurality of item regions, determine the occlusion relationship between the item regions using depth information, and analyze the occlusion conditions of the item regions.

[0146] A partition feature fusion module is configured to perform spatial partitioning on the item regions and extract multi-modal features, weight fuse the partition features according to the occlusion conditions, and obtain a feature representation vector.

[0147] A hierarchical matching and identification module is configured to match and identify each item region with an item feature library based on barcode information and the feature representation vector, perform reliability evaluation on the identification result, and output a candidate identification result and a reliability mark.

[0148] A correlation correction decision module is configured to use the co-occurrence relationship between item categories to correct the candidate identification result of each item region according to the reliability mark, and determine a final identification result.

[0149] A feature library dynamic updating module is configured to update the item feature library according to the final identification result.

[0150] Those skilled in the art can clearly understand the specific working process of the system and modules described above for the convenience and brevity of description. The corresponding process in the foregoing method embodiments can be referred to, and will not be repeated here.

[0151] The embodiment of the present application further provides a computer device, comprising a memory and a processor, the memory stores a computer program; the computer program is executed by the processor, so that the processor executes the steps of the warehouse article intelligent identification and classification method based on image processing of any one of the above embodiments.

[0152] The embodiment of the present application further provides a computer readable storage medium, which stores a computer program / instruction, and the computer program / instruction is executed by a processor to realize the steps of the warehouse article intelligent identification and classification method based on image processing of any one of the above embodiments.

[0153] In several embodiments provided in the present application, it should be understood that the disclosed system and method can be implemented in other manners. For example, the above-described unit embodiments are merely illustrative, and the division of units can be different, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed coupling or direct coupling or communication connection between units can be indirect coupling or communication connection through some interfaces, devices or units, and can be electrical, mechanical or other forms.

[0154] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they can be located in one place, or distributed on multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.

[0155] The above, the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that the technical solutions recorded in the foregoing embodiments can still be modified, or some technical features can be replaced by equivalents; and these modifications or replacements do not make the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. An image processing-based intelligent identification and classification method for warehouse goods, characterized in that, The method comprises the following steps: obtaining an RGB image and a depth image of a scene to be recognized, and performing illumination normalization on the RGB image based on the depth image; performing region segmentation based on the depth image and the RGB image to obtain a plurality of object regions, determining the occlusion relationship between the object regions by using depth information, and analyzing the occlusion condition of each object region; spatially partitioning each object region and extracting multi-modal features, weighting and fusing the features of each partition according to the occlusion condition to obtain a feature representation vector; matching and recognizing each object region and an object feature library based on barcode information and the feature representation vector, evaluating the reliability of the recognition result, and outputting a candidate recognition result and a reliability mark; according to the reliability mark, using the co-occurrence relationship between the object categories to correct the candidate recognition result of each object region, and determining the final recognition result; updating the object feature library according to the final recognition result; the use of the co-occurrence relationship between the object categories to correct the candidate recognition result of each object region comprises: obtaining an object category co-occurrence probability matrix; for the object region to be associated and verified, calculating the co-occurrence probability of the candidate category and other recognized categories in the same scene, and adjusting the confidence according to the co-occurrence probability; for the object region to be manually reviewed, if there are multiple candidate categories, sorting the candidate categories according to the co-occurrence probability, and taking the category with the highest co-occurrence probability as the recommended category; summarizing and outputting the final recognition category, confidence and reliability mark of each object region. 2.The image processing-based warehouse item intelligent recognition and classification method according to claim 1, characterized in that, the illumination normalization of the RGB image based on the depth image comprises: filtering the depth image according to the depth confidence information and the neighborhood depth value, and repairing the depth missing area; performing global correction on the RGB image, converting the corrected RGB image to the LAB color space, and separating to obtain the luminance component and the color component; dividing the luminance component into a plurality of depth layers based on the depth image, and performing adaptive histogram equalization on each depth layer; keeping the color component unchanged, recombining the processed luminance component and the color component, and converting back to the RGB color space to obtain the illumination normalized RGB image; performing local contrast enhancement on the illumination normalized RGB image, detecting and repairing the mirror reflection area. 3.The image processing-based warehouse item intelligent recognition and classification method according to claim 1, characterized in that, the analysis of the occlusion condition of each object region comprises: performing initial region segmentation on the depth image based on depth continuity to obtain preliminary segmentation regions; refining the boundaries of the preliminary segmentation regions based on the color and edge information of the RGB image to obtain finely segmented object regions; identifying the contact boundaries between the object regions, determining the occlusion relationship between adjacent object regions based on the depth information at the contact boundaries, and constructing an occlusion relationship graph; locating the occlusion boundary positions of each object region according to the occlusion relationship graph and the contact boundaries, and labeling the occlusion boundary types; generating a visible part mask for each object region according to the occlusion boundary positions and the object regions, and determining the occlusion metric information based on the visible part mask. 4.The image processing-based warehouse item intelligent recognition and classification method according to claim 1, characterized in that, the weighting and fusion of the features of each partition according to the occlusion condition comprises: spatially partitioning each object region according to the occlusion condition, and determining the visible region range of each partition based on the visible part mask; Barcode detection is performed on each partition, and if a barcode is detected, barcode information is extracted; Visual features, geometric features, depth features, and semantic features are extracted for each partition; According to the position of the occlusion boundary, the partitions adjacent to the occlusion boundary are marked as boundary partitions; The features of each partition are weighted and fused based on the boundary marking and the occlusion severity level to obtain the feature representation vector of each item region. 5.The image processing-based warehouse item intelligent recognition and classification method according to claim 1, characterized in that, The matching and recognition of each item region with the item feature library includes: According to the barcode detection results and feature library matching of each item region, the item regions are prioritized; Different recognition strategies are used to identify different priority item regions; The confidence of the recognition result is evaluated, and the reliability is judged in combination with the occlusion measurement information, and the candidate category, confidence, and reliability mark of each item region are output. 6.The image processing-based warehouse item intelligent recognition and classification method according to claim 3, characterized in that, The determination of the occlusion measurement information based on the visible part mask includes: A binary mask is generated on each item region according to the position of the occlusion boundary; The proportion of visible pixels in each item region is counted, and the occlusion degree is calculated; The occlusion severity level is divided according to the occlusion degree; The potential key regions of each item region are identified based on image feature analysis, the overlap between the potential key regions and the visible part mask is analyzed, and the key region visibility analysis is generated; The occlusion measurement information of each item region is generated by integrating the occlusion degree, the occlusion severity level, and the key region visibility analysis.

7. An image processing-based intelligent identification and classification system for warehouse goods, characterized in that, It includes: An illumination normalization module for obtaining an RGB image and a depth image of a scene to be recognized, and performing illumination normalization on the RGB image based on the depth image; An occlusion-aware region segmentation module for segmenting the regions based on the depth image and the RGB image to obtain multiple item regions, determining the occlusion relationship between the item regions using depth information, and analyzing the occlusion of each item region; A partition feature fusion module for spatial partitioning of each item region and extraction of multi-modal features, and weighted fusion of each partition feature according to the occlusion condition to obtain a feature representation vector; A hierarchical matching and recognition module for matching and recognizing each item region with an item feature library based on barcode information and feature representation vectors, and reliability evaluation of the recognition results to output candidate recognition results and reliability marks; An association correction decision module for associating and correcting the candidate recognition results of each item region based on the reliability marks and the co-occurrence relationship between item categories to determine the final recognition results; A feature library dynamic updating module for updating the item feature library based on the final recognition results; The association correction of the candidate recognition results of each item region based on the co-occurrence relationship between item categories includes: An item category co-occurrence probability matrix is obtained; For the item regions to be associated and verified, the co-occurrence probability of the candidate category with other recognized categories in the same scene is calculated, and the confidence is adjusted according to the co-occurrence probability; For the item regions to be manually reviewed, if there are multiple candidate categories, the candidate categories are sorted according to the co-occurrence probability, and the category with the highest co-occurrence probability is recommended as the recommended category; The final recognition category, confidence, and reliability mark of each item region are output.

8. A computer device, comprising: An image processing-based warehousing article intelligent identification and classification method comprises a memory and a processor, the memory stores a computer program, and the computer program is executed by the processor to make the processor execute the steps of the image processing-based warehousing article intelligent identification and classification method according to any one of claims 1-6.

9. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed to implement the steps of the image processing-based warehousing article intelligent identification and classification method according to any one of claims 1-6.

Images