Warehouse article intelligent identification and classification method and system based on image processing, and related equipment

Abstract

The invention discloses an intelligent identification and classification method and system for stored goods based on image processing and related equipment, and relates to the technical field of computer vision, and the method comprises the steps: obtaining an RGB image and a depth image of a to-be-identified scene; carrying out region segmentation, judging a shielding relationship between article regions by utilizing depth information, and analyzing shielding conditions of the article regions; spatial partitioning is carried out on each article area, multi-modal features are extracted, and weighted fusion is carried out on the features of each partition according to the shielding condition; performing matching identification on each article region and an article feature library, performing reliability evaluation on an identification result, and outputting a candidate identification result and a reliability mark; and performing correlation correction by utilizing a co-occurrence relationship among article categories, and determining a final recognition result. And updating the article feature library according to the final recognition result. According to the invention, through a depth information-based shielding sensing mechanism and a priority-driven identification strategy, the identification precision of the stored articles in the shielding scene is improved.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a method, system and related equipment for intelligent identification and classification of warehouse goods based on image processing. Background Technology

[0002] With the rapid development of e-commerce logistics and smart warehousing, the demand for automatic identification and classification of stored goods is becoming increasingly urgent. Traditional manual identification methods are inefficient and prone to errors, while automatic identification schemes based on barcode scanning have insufficient recognition rates when barcodes are damaged or obscured. Computer vision technology offers a new solution for the identification of stored goods, achieving automatic classification of items through image acquisition, feature extraction, and pattern matching, and has become an important research direction in this field.

[0003] Existing image recognition methods face several challenges in practical applications. The complex and variable lighting conditions in warehouse environments affect image acquisition quality and recognition stability. Stacked items often obstruct each other, increasing the difficulty of item segmentation and feature extraction. Existing methods often rely on single types of image information and features, limiting their adaptability to complex scenes. Furthermore, recognition systems typically lack mechanisms for evaluating and providing feedback on the accuracy of results, making it difficult to guarantee the reliability of recognition. Summary of the Invention

[0004] This invention provides an intelligent identification and classification method for warehouse goods based on image processing, which solves the technical problems of unstable image quality under complex lighting conditions, difficulty in identifying obscured objects, and insufficient reliability of identification results in the prior art.

[0005] The first aspect of this invention provides an intelligent identification and classification method for warehouse goods based on image processing, comprising:

[0006] Acquire the RGB and depth images of the scene to be identified, and perform illumination normalization processing on the RGB image based on the depth image;

[0007] Multiple object regions are obtained by region segmentation based on depth images and RGB images. The occlusion relationship between object regions is determined by using depth information, and the occlusion situation of each object region is analyzed.

[0008] The spatial regions of each item are divided into partitions and multimodal features are extracted. The features of each partition are weighted and fused according to the occlusion situation to obtain the feature representation vector.

[0009] Based on barcode information and feature representation vectors, each item region is matched and identified with the item feature database. The reliability of the identification results is evaluated, and candidate identification results and reliability labels are output.

[0010] Based on the reliability label, the candidate identification results of each item region are correlated and corrected by using the co-occurrence relationship between item categories to determine the final identification result;

[0011] The item feature database is updated based on the final identification results.

[0012] Optionally, illumination normalization processing is performed on the RGB image based on the depth image, including:

[0013] The depth image is filtered based on depth confidence information and neighborhood depth values ​​to repair areas with missing depth.

[0014] Perform global correction on the RGB image, convert the corrected RGB image to the LAB color space, and separate the luminance component and color component;

[0015] The brightness component is divided into multiple depth layers based on the depth image, and adaptive histogram equalization is performed on each depth layer.

[0016] Keeping the color components unchanged, the processed luminance components and color components are recombined and converted back to the RGB color space to obtain the RGB image after illumination normalization.

[0017] Local contrast enhancement is performed on the RGB image after illumination normalization, and specular reflection areas are detected and repaired.

[0018] Optionally, analyze the occlusion of each item area, including:

[0019] Initial region segmentation is performed on the depth image based on depth continuity to obtain preliminary segmented regions;

[0020] The boundaries of the initially segmented regions are refined based on the color edge information of the RGB image to obtain finely segmented object regions;

[0021] Identify the contact boundaries between each object region, determine the occlusion relationship between adjacent object regions based on the depth information at the contact boundaries, and construct an occlusion relationship map.

[0022] Based on the occlusion relationship diagram and contact boundary, locate the occlusion boundary position of each item area and label the occlusion boundary type;

[0023] Based on the occlusion boundary location and the object region, a visible part mask for each object region is generated, and occlusion measurement information is determined based on the visible part mask.

[0024] Optionally, the features of each region are weighted and fused according to the occlusion situation, including:

[0025] Based on the occlusion situation, the area of ​​each item is spatially partitioned, and the visible area range of each partition is determined based on the visible part mask;

[0026] Barcode detection is performed on each partition; if a barcode is detected, the barcode information is extracted.

[0027] Visual features, geometric features, depth features, and semantic features are extracted from each partition;

[0028] Based on the location of the occlusion boundary, the partitions adjacent to the occlusion boundary are marked as boundary partitions;

[0029] The features of each partition are weighted and fused based on boundary markers and occlusion severity levels to obtain the feature representation vector of each item region.

[0030] Optionally, each item region is matched and identified against an item feature database, including:

[0031] The item areas are prioritized based on the barcode detection results and feature library matching of each item area.

[0032] Different identification strategies are used to identify items in areas of different priorities;

[0033] The confidence level of the recognition results is evaluated, and the reliability is judged by combining the occlusion measurement information. The candidate category, confidence level and reliability label of each item region are output.

[0034] Optionally, the candidate identification results for each item region are correlated and corrected using the co-occurrence relationships between item categories, including:

[0035] Obtain the co-occurrence probability matrix of item categories;

[0036] For the item region to be verified for association, calculate the co-occurrence probability of the candidate category with other identified categories in the same scene, and adjust the confidence level according to the co-occurrence probability;

[0037] For areas of items that have been manually reviewed, if there are multiple candidate categories, the candidate categories are sorted according to their co-occurrence probability, and the category with the highest co-occurrence probability is recommended.

[0038] The final identification category, confidence level, and reliability label for each item region are summarized and output.

[0039] Optionally, occlusion measurement information is determined based on the visible part mask, including:

[0040] Generate binary masks for each item region based on the location of the occlusion boundary;

[0041] Calculate the percentage of visible pixels in each item's region and determine the degree of occlusion.

[0042] The severity of occlusion is classified according to the degree of obstruction.

[0043] Based on image feature analysis, potential key regions of each object area are identified, and the overlap between potential key regions and visible area masks is analyzed to generate key region visibility analysis.

[0044] By integrating occlusion level, occlusion severity level, and key area visibility analysis, occlusion measurement information for each item area is generated.

[0045] A second aspect of the present invention provides an intelligent identification and classification system for warehouse goods based on image processing, comprising:

[0046] The illumination normalization module is used to acquire the RGB image and depth image of the scene to be identified, and to perform illumination normalization processing on the RGB image based on the depth image;

[0047] The occlusion perception region segmentation module is used to segment multiple object regions based on depth images and RGB images, determine the occlusion relationship between object regions using depth information, and analyze the occlusion situation of each object region.

[0048] The partition feature fusion module is used to spatially partition each item region and extract multimodal features. Based on the occlusion situation, the features of each partition are weighted and fused to obtain a feature representation vector.

[0049] The hierarchical matching and recognition module is used to match and recognize each item region with the item feature library based on barcode information and feature representation vectors, evaluate the reliability of the recognition results, and output candidate recognition results and reliability labels.

[0050] The association correction decision module is used to correct the candidate identification results of each item region based on the reliability label and the co-occurrence relationship between item categories, and to determine the final identification result.

[0051] The feature library dynamic update module is used to update the item feature library based on the final recognition results.

[0052] A computer device provided in a third aspect of the present invention includes a memory and a processor. The memory stores a computer program. When the computer program is executed by the processor, the processor performs the steps of the image processing-based intelligent identification and classification method for warehouse goods as described above.

[0053] The fourth aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed, performs the steps as described in any of the above claims.

[0054] Beneficial Effects: This invention effectively solves the recognition challenges in complex lighting and object occlusion scenarios by combining RGB and depth images for illumination normalization and region segmentation, utilizing depth information to determine occlusion relationships, and performing weighted feature fusion. By evaluating the reliability of the recognition results and using co-occurrence relationships of item categories for association correction, the accuracy and robustness of the system are improved. Dynamically updating the item feature library based on the recognition results enables the system to adapt and continuously optimize recognition performance. The overall solution organically combines illumination processing, occlusion analysis, multimodal feature fusion, reliability assessment, and association correction, significantly improving the accuracy, reliability, and practicality of intelligent identification of warehouse items. Attached Figure Description

[0055] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0056] Figure 1 This is a flowchart illustrating an image processing-based intelligent identification and classification method for warehouse goods, provided in an embodiment of the present invention.

[0057] Figure 2 This is a flowchart illustrating the depth-guided illumination normalization process of an image-process-based intelligent identification and classification method for warehouse goods, provided in an embodiment of the present invention.

[0058] Figure 3 This is a flowchart of the partition feature fusion process for an image processing-based intelligent identification and classification method for warehouse goods, provided in an embodiment of the present invention.

[0059] Figure 4 This is a structural block diagram of an image processing-based intelligent identification and classification system for warehouse goods, provided as an embodiment of the present invention. Detailed Implementation

[0060] This invention provides an intelligent identification and classification method for warehouse goods based on image processing, which solves the technical problems of unstable image quality under complex lighting conditions, difficulty in identifying obscured objects, and insufficient reliability of identification results in the prior art.

[0061] To make the objectives, features, and advantages of this invention more apparent and understandable, the technical solutions of the embodiments of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the embodiments described below are only some embodiments of this invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0062] like Figure 1 As shown, the present invention provides an intelligent identification and classification method for warehouse goods based on image processing, comprising:

[0063] S1: Obtain the RGB image and depth image of the scene to be identified, and perform illumination normalization processing on the RGB image based on the depth image.

[0064] Specifically, the flowchart for depth-guided illumination normalization is as follows: Figure 2 As shown, it includes:

[0065] S1.1: Obtain the RGB image, depth image, and depth confidence map of the scene to be identified.

[0066] In this image, the pixel positions of the RGB image and the depth image are in one-to-one correspondence.

[0067] S1.2: Filter the depth image based on depth confidence information and neighborhood depth values ​​to repair areas with missing depth.

[0068] In one embodiment, low-confidence pixel regions are marked, isolated abnormal depth points are detected and removed, edge-preserving spatial filtering is applied to the depth image to smooth noise, and interpolation is used to repair small areas of missing depth.

[0069] S1.3: Perform global correction on the RGB image, convert the corrected RGB image to the LAB color space, and separate the luminance component and color component.

[0070] Specifically, global correction includes white balance processing and exposure correction.

[0071] S1.4: Based on the depth image, the brightness component is divided into multiple depth layers, and adaptive histogram equalization is performed on each depth layer.

[0072] Furthermore, the depth value distribution of the depth image is analyzed to determine the depth range and divide it into N depth layers; according to the depth value of each pixel in the depth image, each pixel of the luminance component is assigned to the corresponding depth layer; adaptive histogram equalization is performed on the luminance pixels of each depth layer to obtain the equalized luminance value of each depth layer; according to the depth layer affiliation of each pixel, the equalized luminance value is used to replace the original luminance value to generate the processed luminance component.

[0073] In one embodiment, the value of N is determined based on the scene depth range, preferably 3 to 10.

[0074] In another embodiment, adaptive histogram equalization employs a contrast-limited adaptive histogram equalization algorithm, setting a contrast limit threshold to avoid over-enhancement.

[0075] S1.5: Keeping the color components unchanged, recombine the processed luminance components and color components and convert them back to the RGB color space to obtain the RGB image after illumination normalization.

[0076] S1.6: Perform local contrast enhancement on the RGB image after illumination normalization, and detect and repair specular reflection areas.

[0077] In a preferred embodiment, local contrast enhancement is performed on dark areas of the RGB image after illumination normalization to improve detail visibility, while specular reflection areas are detected and repaired based on neighboring pixels. Specular reflection areas are identified through brightness threshold detection, appearing as locally overexposed bright areas.

[0078] Through the above processing, this step adjusts the brightness of objects at different depth positions based on the depth-layered illumination normalization method, which solves the problem that traditional global illumination correction cannot take into account foreground and background objects, and makes similar objects present consistent visual characteristics in different spatial positions.

[0079] S2: Based on the depth image and RGB image, perform region segmentation to obtain multiple object regions, use depth information to determine the occlusion relationship between object regions, and analyze the occlusion situation of each object region.

[0080] Specifically, S2 includes:

[0081] S2.1: Perform initial region segmentation on the depth image based on depth continuity to obtain preliminary segmented regions.

[0082] S2.2: Based on the color edge information of the RGB image, the boundaries of the initially segmented region are refined to obtain finely segmented object regions.

[0083] In one embodiment, color edges within the initially segmented region are detected. When the difference in average depth values ​​between the regions on both sides of the color edge is less than the average depth gradient at the boundary of the region, and the color edge penetrates the region, the region is further segmented along the color edge to avoid merging multiple independent items with similar depths into a single region.

[0084] S2.3: Identify the contact boundaries between each item area, determine the occlusion relationship between adjacent item areas based on the depth information at the contact boundaries, and construct an occlusion relationship map.

[0085] Specifically, the shared boundary segments between finely segmented object regions are detected; the depth difference and depth gradient on both sides of the boundary segment are extracted, and the front-to-back relationship of adjacent regions is determined based on the direction of the depth gradient, with regions with smaller depth values ​​identified as occluders; the reliability of the occlusion determination of the boundary segments is evaluated, and unreliable boundaries with excessively small average depth gradient values ​​or low consistency in the direction of the depth gradient are filtered out; based on reliable occlusion relationships, a directed occlusion relationship graph between object regions is constructed.

[0086] S2.4: Based on the occlusion relationship diagram and contact boundary, locate the occlusion boundary position of each item area and label the occlusion boundary type.

[0087] Furthermore, based on the occlusion relationship diagram, the occlusion relationship pairs of each item region are determined. For each occlusion relationship pair, the occlusion boundary position of the occluded item is marked on the corresponding contact boundary segment. The spatial distribution of all occlusion boundaries in each item region is analyzed to determine the orientation of the occlusion boundaries. Based on the orientation distribution characteristics of the occlusion boundaries, the occlusion types are divided into unilateral occlusion, multilateral occlusion, and encircling occlusion. For each item region, all occlusion boundary positions and occlusion boundary types are integrated to generate a complete occlusion boundary description.

[0088] Among them, unilateral occlusion refers to the occlusion boundary being concentrated in one direction of the object area, multilateral occlusion refers to the occlusion boundary being distributed in two or more discontinuous directions, and encircling occlusion refers to the occlusion boundary being distributed around the perimeter of the object area.

[0089] S2.5: Generate the visible part mask of each item area based on the occlusion boundary position and item area, and determine the occlusion measurement information based on the visible part mask.

[0090] Preferably, a binary mask is generated on each item region based on the occlusion boundary position, and the occluded part and the visible part are marked respectively; the proportion of visible pixels in each item region is counted to calculate the degree of occlusion; the occlusion severity level is divided according to the degree of occlusion; potential key regions of each item region are identified based on image feature analysis, including regions with complex textures, regions with dense edges, and regions with high contrast; the overlap between potential key regions and the visible part mask is analyzed to generate key region visibility analysis; the occlusion degree, occlusion severity level, and key region visibility analysis are integrated to generate occlusion measurement information for each item region.

[0091] In one embodiment, edge-dense regions are identified by gradient magnitude statistics, texture-complex regions are identified by local variance analysis, and high-contrast regions are identified by brightness-contrast calculation. These regions typically contain key identification features such as barcodes, text, and brand logos.

[0092] In another embodiment, occlusion less than 30% is marked as mild occlusion, occlusion between 30% and 70% is marked as moderate occlusion, and occlusion greater than 70% is marked as severe occlusion. In other embodiments, the above thresholds can be adjusted according to the actual application scenario.

[0093] Based on the above processing, region segmentation is performed by fusing depth continuity and color edge information, avoiding the erroneous merging of items with similar depth and the erroneous segmentation of items with similar color in single-modal segmentation. By constructing a directed occlusion relationship graph based on the depth information of the contact boundary, and generating the visible part mask and occlusion measurement information of each item region, the occlusion type, occlusion degree and key area visibility of each item region are clarified, providing accurate occlusion analysis results for subsequent feature weighting and priority division.

[0094] S3: Spatial partitioning is performed on each object region and multimodal features are extracted. The features of each partition are weighted and fused according to the occlusion situation to obtain the feature representation vector.

[0095] Specifically, the flowchart for partition feature fusion is as follows: Figure 3 As shown, it includes:

[0096] S3.1: Divide the spatial area of ​​each item into partitions based on the occlusion situation, and determine the visible area range of each partition based on the visible part mask.

[0097] In one embodiment, a partitioning strategy for each item region is determined based on the occlusion severity level in the occlusion metric information: item regions with a mild occlusion severity level are finely partitioned, while item regions with a moderate or severe occlusion severity level are coarsely partitioned. Fine and coarse partitioning are implemented using different grid densities. The item regions are then divided into grids according to the partitioning strategy. The number of visible pixels in each grid partition is calculated based on the visible part mask. Invalid partitions with a number of visible pixels less than the effective pixel threshold are filtered out. The effective pixel threshold is set to 500 pixels, which is determined based on the minimum input size requirement of the feature extraction algorithm. The retained effective partitions are numbered, and a list of effective partitions for each item region and the visible area range of each partition are output.

[0098] S3.2: Perform barcode detection on each partition. If a barcode is detected, extract the barcode information.

[0099] Specifically, barcode detection and decoding are performed on the visible areas of each partition; if decoding is successful, the barcode content and detection confidence are recorded, and the detection confidence is output by the barcode recognition algorithm; if the same barcode is detected in multiple partitions of the same item area, the highest detection confidence is taken as the final confidence; if different barcodes are detected in multiple partitions of the same item area, they are marked as barcode abnormal; the barcode information, detection confidence, or no barcode mark for each item area are output.

[0100] In one embodiment, barcode detection uses the ZBar or ZXing algorithm, and the detection confidence is determined by the decoding quality score output by the algorithm, with a value ranging from 0 to 1.

[0101] S3.3: Extract visual features, geometric features, depth features, and semantic features from each partition.

[0102] Among them, visual features include color features and texture features, geometric features include shape features, depth features include depth statistical features and depth geometric features, and semantic features include text features and logo features.

[0103] S3.4: Based on the location of the occlusion boundary, mark the partitions adjacent to the occlusion boundary as boundary partitions.

[0104] S3.5: Based on boundary markers and occlusion severity levels, the features of each partition are weighted and fused to obtain the feature representation vector of each item region.

[0105] Preferably, fusion weights are assigned to the features of each partition: the feature weights of boundary partitions are lower than those of non-boundary partitions; the weights of various features are adjusted according to the occlusion severity level of the item region: when the occlusion severity level is mild, the weights of visual and geometric features are higher; when the occlusion severity level is moderate or severe, the weights of depth features are higher; the weights of various features are normalized; and the weighted features of all valid partitions within each item region are fused to generate the feature representation vector of that item region.

[0106] By adopting the above processing, the partitioning strategy is dynamically adjusted according to the severity of occlusion. Combined with multimodal feature extraction and differential weighted fusion, the fine features of lightly occluded objects are fully utilized, while the spatial features such as depth are mainly utilized for heavily occluded objects. This avoids feature redundancy or loss caused by a uniform processing strategy and improves the feature representation quality of objects with different degrees of occlusion.

[0107] S4: Based on barcode information and feature representation vectors, match and identify each item region with the item feature library, evaluate the reliability of the identification results, and output candidate identification results and reliability labels.

[0108] Specifically, S4 includes:

[0109] S4.1: Obtain the item feature database.

[0110] It should be noted that the item feature library stores barcode information, feature representation vectors, key feature area annotations, physical dimensions, and category labels for each category of items.

[0111] In one embodiment, the process of establishing an item feature library includes: collecting barcode information for each category of items and establishing a mapping relationship between the barcode information and category labels; collecting standard images for each category of items and extracting feature representation vectors; labeling key feature regions for each category of items, including trademark regions, pattern regions, or text regions; recording the physical dimensions of each category of items; and establishing an index and storing it in the item feature library. The feature library can be established offline and supports dynamic updates and expansion.

[0112] S4.2: Prioritize the item areas based on the barcode detection results and feature library matching results of each item area.

[0113] Specifically, based on the barcode detection confidence level and the barcode's matching status in the feature library, the item areas are divided into three priorities: the first priority is the item areas where the barcode detection confidence level is higher than the first preset threshold and the corresponding category is successfully matched in the item feature library; the second priority is the item areas where the barcode detection confidence level is higher than the first preset threshold but the corresponding category is not matched in the item feature library; the third priority is the item areas where the barcode detection confidence level is between the second preset threshold and the first preset threshold; and the fourth priority is the item areas where no barcode is detected or the barcode detection confidence level is lower than the second preset threshold.

[0114] In one embodiment, the first preset threshold is set to 0.85, representing a standard for clear and reliable barcodes; the second preset threshold is set to 0.50, representing the minimum standard for usable barcodes. The setting of these thresholds comprehensively considers the reliability of barcode detection technology and the actual distribution of barcode quality in a warehousing scenario. In other embodiments, the above thresholds can be adjusted according to the accuracy of the barcode detection equipment and the quality of the item's barcode.

[0115] S4.3: Different identification strategies are used to identify items of different priorities.

[0116] Specifically, barcode matching is performed on the first priority item area: the product category corresponding to the barcode is queried in the item feature library, and the product category and barcode detection confidence are output.

[0117] Furthermore, barcode information is output and visual matching is performed on the second priority item area: the barcode information and unmatched markers in the feature library are output, and the similarity between the feature representation vector of the item area and the feature vector of each commodity category in the item feature library is calculated. The candidate category list and the similarity of each candidate category are output as auxiliary reference information to assist manual judgment of commodity category or detection of barcode recognition errors.

[0118] Furthermore, barcode matching and visual verification are performed on the third priority item area: the product category corresponding to the barcode is queried in the item feature library; if the match is successful, the standard feature vector of the product category is obtained, the visual similarity between the standard feature vector and the feature representation vector of the item area is calculated, and the product category, barcode detection confidence, and visual similarity are output; if the match fails, the barcode information and a match failure mark are output, and visual matching is performed simultaneously, the similarity between the feature representation vector of the item area and the feature vector of each product category in the item feature library is calculated, and a list of candidate categories and the similarity of each candidate category are output.

[0119] Furthermore, visual matching is performed on the fourth priority item region: the feature representation vector of the item region is compared with the feature vector of each commodity category in the item feature library to calculate the similarity, and the candidate category list and the similarity of each candidate category are output.

[0120] In one embodiment, similarity is calculated using cosine similarity or Euclidean distance.

[0121] S4.4: Evaluate the confidence level of the recognition results, combine the occlusion measurement information to judge the reliability, and output the candidate category, confidence level and reliability label of each item region.

[0122] Specifically, for the identification results of the first priority item area, the barcode detection confidence level is used as the final confidence level; the reliability is judged according to the occlusion severity level: if the occlusion severity level is mild, the reliability is marked as directly confirmable; if the occlusion severity level is moderate or severe, the reliability is marked as pending association verification.

[0123] Furthermore, for the identification results of the second priority item area, the barcode information is used as the main identification result, the barcode detection confidence is used as the final confidence, and the reliability is marked as pending manual review (it is recommended to update the feature library); the visual matching result is also output as auxiliary reference information.

[0124] Furthermore, the identification results for the third priority item region are processed according to the barcode matching situation: if the barcode is successfully matched in the item feature library, the final confidence score is calculated based on the barcode detection confidence score and visual similarity; the reliability is judged based on visual similarity: if the visual similarity is higher than the verification threshold, the reliability is marked as directly confirmable; if the visual similarity is lower than the verification threshold but not lower than the minimum verification threshold, the reliability is marked as pending association verification; if the visual similarity is lower than the minimum verification threshold, it is marked as feature anomaly, and the reliability is marked as pending manual review.

[0125] Furthermore, if the barcode fails to match in the item feature library, the confidence level of the candidate categories for visual matching is evaluated: the difference between the highest similarity and the second highest similarity is calculated; if the highest similarity is not lower than the similarity threshold and the difference is not lower than the minimum discrimination threshold, the highest similarity is used as the confidence level and the reliability is marked as pending association verification; if the highest similarity is lower than the similarity threshold or the difference is less than the minimum discrimination threshold, the reliability is marked as pending manual review.

[0126] Furthermore, for the visual matching results of the fourth priority item region, the severity of occlusion is first assessed: if the occlusion severity is severe, it is directly marked as requiring manual review; if the occlusion severity is mild or moderate, the difference between the highest and second-highest similarity is calculated for further evaluation: when the occlusion severity is mild, if the highest similarity is not lower than the high-confidence similarity threshold and the difference is greater than the discrimination threshold, the highest similarity is used as the confidence level, and the reliability is marked as directly confirmable; if the highest similarity is lower than the high-confidence similarity threshold but not lower than the similarity threshold, and the difference is not lower than the minimum discrimination threshold, the reliability is marked as requiring association verification; otherwise, the reliability is marked as requiring manual review. When the occlusion severity is moderate, if the highest similarity is not lower than the similarity threshold and the difference is greater than the discrimination threshold, the highest similarity is used as the confidence level, and the reliability is marked as requiring association verification; otherwise, the reliability is marked as requiring manual review.

[0127] Furthermore, the identification results of each item region are aggregated, and candidate categories, confidence levels, and reliability labels are output.

[0128] In one embodiment, the verification threshold is set to 0.75; the minimum verification threshold is set to 0.50; the high-confidence similarity threshold is set to 0.75; the similarity threshold is set to 0.60; the discrimination threshold is set to 0.15; and the minimum discrimination threshold is set to 0.08. These thresholds comprehensively consider the differences between barcode reliability and visual matching reliability, the impact of occlusion on recognition, and the appearance similarity characteristics of stored goods. In other embodiments, these thresholds can be adjusted according to the characteristics of the item category and business needs.

[0129] By using the above method, the item regions are prioritized based on the barcode detection confidence and feature library matching. Differentiated recognition strategies are adopted for different priorities, and the reliability of the recognition results is evaluated by combining occlusion measurement information. This achieves synergistic complementarity between barcode recognition and visual matching, improving recognition accuracy. At the same time, the reliability markers distinguish between item regions that can be directly confirmed, those that need to be associated and verified, and those that need to be manually reviewed, providing a basis for subsequent association correction and manual intervention.

[0130] S5: Based on the reliability label, the candidate identification results of each item region are correlated and corrected by utilizing the co-occurrence relationship between item categories to determine the final identification result.

[0131] Specifically, S5 includes:

[0132] S5.1: Obtain the co-occurrence probability matrix of item categories;

[0133] It should be noted that the co-occurrence probability matrix of item categories stores the co-occurrence probability between items of each category. The co-occurrence probability represents the probability that two items of the same category will appear simultaneously in the same scene.

[0134] S5.2: For the item region to be verified for association, calculate the co-occurrence probability of the candidate category with other identified categories in the same scene, and adjust the confidence level according to the co-occurrence probability.

[0135] Specifically, the identification category of the item region in the same scene that is marked as directly verifiable is obtained as the reference category; if the reference category is empty, the co-occurrence relationship adjustment is skipped and the original identification result is maintained; otherwise, the co-occurrence probability of the candidate category of the item region to be associated and verified and each reference category is calculated, and the highest co-occurrence probability is taken as the co-occurrence score; if the co-occurrence score is not lower than the co-occurrence threshold, the confidence is multiplied by the co-occurrence gain coefficient to improve it, and the reliability label is modified to be directly verifiable; if the co-occurrence score is lower than the co-occurrence threshold, the original confidence and reliability label are maintained.

[0136] S5.3: For areas of items to be manually reviewed, if there are multiple candidate categories, the candidate categories are sorted according to their co-occurrence probability, and the category with the highest co-occurrence probability is selected as the recommended category.

[0137] Specifically, obtain the reference category; if the reference category is empty, skip the sorting optimization and keep the original candidate category order; otherwise, calculate the co-occurrence score of each candidate category with the reference category; re-sort the candidate categories according to the co-occurrence score; if the category with the highest co-occurrence score is not lower than the co-occurrence threshold and the category originally ranked first in similarity, modify the reliability label to pending association verification; otherwise, keep the original reliability label and output the sorted candidate category list for manual reference.

[0138] S5.4: Summarize and output the final identification category, confidence level, and reliability label for each item region.

[0139] In one embodiment, the co-occurrence threshold is set to 0.40, and the co-occurrence gain coefficient is set to 1.15. These parameters are set to balance the effectiveness of the association strength between item categories with the dominance of the recognition results. In other embodiments, these parameters can be adjusted according to the actual application scenario and the distribution characteristics of item categories.

[0140] Through the above steps, the co-occurrence probability matrix of item categories is used to correct the association between the areas of items to be verified and those to be manually reviewed. The confidence level is adjusted or the ranking of candidate categories is optimized by calculating the co-occurrence relationship between candidate categories and confirmed categories in the scene. The regularity of item pairing in the scene is utilized to improve the accuracy and reliability of the recognition results.

[0141] S6: Update the item feature database based on the final recognition results.

[0142] Beneficial Effects: This invention effectively solves the recognition challenges in complex lighting and object occlusion scenarios by combining RGB and depth images for illumination normalization and region segmentation, utilizing depth information to determine occlusion relationships, and performing weighted feature fusion. By evaluating the reliability of the recognition results and using co-occurrence relationships of item categories for association correction, the accuracy and robustness of the system are improved. Dynamically updating the item feature library based on the recognition results enables the system to adapt and continuously optimize recognition performance. The overall solution organically combines illumination processing, occlusion analysis, multimodal feature fusion, reliability assessment, and association correction, significantly improving the accuracy, reliability, and practicality of intelligent identification of warehouse items.

[0143] like Figure 4 As shown, the present invention provides an intelligent identification and classification system for warehouse goods based on image processing, comprising:

[0144] The illumination normalization module is used to acquire the RGB image and depth image of the scene to be identified, and to perform illumination normalization processing on the RGB image based on the depth image;

[0145] The occlusion perception region segmentation module is used to segment multiple object regions based on depth images and RGB images, determine the occlusion relationship between object regions using depth information, and analyze the occlusion situation of each object region.

[0146] The partition feature fusion module is used to spatially partition each item region and extract multimodal features. Based on the occlusion situation, the features of each partition are weighted and fused to obtain a feature representation vector.

[0147] The hierarchical matching and recognition module is used to match and recognize each item region with the item feature library based on barcode information and feature representation vectors, evaluate the reliability of the recognition results, and output candidate recognition results and reliability labels.

[0148] The association correction decision module is used to correct the candidate identification results of each item region based on the reliability label and the co-occurrence relationship between item categories, and to determine the final identification result.

[0149] The feature library dynamic update module is used to update the item feature library based on the final recognition results.

[0150] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process of the system and modules described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0151] This invention also provides a computer device, including a memory and a processor, wherein the memory stores a computer program; when the computer program is executed by the processor, the processor performs the steps of the image processing-based intelligent identification and classification method for warehouse goods as described in any of the above embodiments.

[0152] This invention also provides a computer-readable storage medium storing a computer program / instructions thereon, which, when executed by a processor, implements the steps of the image processing-based intelligent identification and classification method for warehouse goods as described in any of the above embodiments.

[0153] In the several embodiments provided in this application, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0154] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0155] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An image processing-based intelligent identification and classification method for warehouse goods, characterized in that, The method comprises the following steps: Obtain the RGB image and the depth image of the scene to be recognized, and perform illumination normalization on the RGB image based on the depth image; Perform region segmentation on the depth image and the RGB image to obtain a plurality of object regions, determine the occlusion relationship between the object regions based on the depth information, and analyze the occlusion of each object region; Divide each object region into sub-regions and extract multi-modal features, and perform weighted fusion on the features of each sub-region according to the occlusion to obtain a feature representation vector; Match and recognize each object region with the object feature library based on the barcode information and the feature representation vector, evaluate the reliability of the recognition result, and output the candidate recognition result and the reliability label; According to the reliability label, the co-occurrence relationship between the object categories is used to correct the candidate recognition result of each object region to determine the final recognition result; Update the object feature library according to the final recognition result. 2.The image processing-based warehouse item intelligent recognition and classification method according to claim 1, characterized in that, The illumination normalization of the RGB image based on the depth image comprises the following steps: Filter the depth image according to the depth confidence information and the neighborhood depth value, and repair the depth missing area; Perform global correction on the RGB image, convert the corrected RGB image to the LAB color space, and separate to obtain the luminance component and the color component; Divide the luminance component into a plurality of depth layers based on the depth image, and perform adaptive histogram equalization on each depth layer; Keep the color component unchanged, recombine the processed luminance component and the color component, and convert back to the RGB color space to obtain the illumination normalized RGB image; Perform local contrast enhancement on the illumination normalized RGB image, detect and repair the mirror reflection area. 3.The image processing-based warehouse item intelligent recognition and classification method according to claim 1, characterized in that, The analysis of the occlusion of each object region comprises the following steps: Perform initial region segmentation on the depth image based on the depth continuity to obtain preliminary segmentation regions; Refine the boundaries of the preliminary segmentation regions based on the color and edge information of the RGB image to obtain finely segmented object regions; Identify the contact boundaries between the object regions, determine the occlusion relationship between the adjacent object regions based on the depth information at the contact boundaries, and construct an occlusion relationship graph; Locate the occlusion boundary positions of each object region according to the occlusion relationship graph and the contact boundaries, and label the occlusion boundary types; Generate the visible part mask of each object region according to the occlusion boundary positions and the object regions, and determine the occlusion metric information based on the visible part mask. 4.The image processing-based warehouse item intelligent recognition and classification method according to claim 1, characterized in that, The weighted fusion of the features of each sub-region according to the occlusion comprises the following steps: Divide each object region into sub-regions according to the occlusion, and determine the visible region range of each sub-region based on the visible part mask; Detect the barcode of each sub-region, and extract the barcode information if the barcode is detected; Extract the visual features, geometric features, depth features and semantic features of each sub-region; According to the occlusion boundary positions, mark the sub-regions adjacent to the occlusion boundary as boundary sub-regions; Perform weighted fusion on the features of each sub-region based on the boundary label and the occlusion severity level to obtain the feature representation vector of each object region. 5.The image processing based warehouse item intelligent recognition and classification method according to claim 1, characterized in that, The matching and recognition of each object region with the object feature library comprises the following steps: According to the barcode detection result of each object region and the matching situation of the feature library, divide the object regions according to the priority. Different recognition strategies are adopted for different priority item regions for recognition; The confidence of the recognition result is evaluated, and the reliability is judged in combination with the occlusion measurement information, and the candidate categories, confidence and reliability markers of each item region are output. 6.The image processing based warehouse item intelligent recognition and classification method according to claim 1, characterized in that, The association correction of the candidate recognition result of each item region based on the co-occurrence relationship between item categories includes: An item category co-occurrence probability matrix is obtained; For the item region to be associated and verified, the co-occurrence probability of the candidate category and other recognized categories in the same scene is calculated, and the confidence is adjusted according to the co-occurrence probability; For the item region to be manually reviewed, if there are multiple candidate categories, the candidate categories are sorted according to the co-occurrence probability, and the category with the highest co-occurrence probability is taken as the recommended category; The final recognition category, confidence and reliability marker of each item region are output. 7.The image processing-based warehouse item intelligent recognition and classification method according to claim 3, characterized in that, The occlusion measurement information is determined based on the visible part mask, including: A binary mask is generated on each item region according to the occlusion boundary position; The occlusion degree is calculated by counting the proportion of visible pixels in each item region; The occlusion severity level is divided according to the occlusion degree; The potential key region of each item region is analyzed based on image feature analysis, the overlap between the potential key region and the visible part mask is analyzed, and the key region visibility analysis is generated; The occlusion measurement information of each item region is generated by integrating the occlusion degree, the occlusion severity level and the key region visibility analysis.

8. An image processing-based intelligent identification and classification system for warehouse goods, characterized in that, It includes: An illumination normalization module for obtaining an RGB image and a depth image of a scene to be recognized, and performing illumination normalization processing on the RGB image based on the depth image; An occlusion-aware region segmentation module for performing region segmentation based on the depth image and the RGB image to obtain a plurality of item regions, determining the occlusion relationship between the item regions using depth information, and analyzing the occlusion of each item region; A partition feature fusion module for spatial partitioning each item region and extracting multi-modal features, and weighting and fusing each partition feature according to the occlusion to obtain a feature representation vector; A hierarchical matching and recognition module for matching and recognizing each item region with an item feature library based on barcode information and the feature representation vector, evaluating the reliability of the recognition result, and outputting the candidate recognition result and the reliability marker; An association correction decision module for associating and correcting the candidate recognition result of each item region based on the reliability marker and the co-occurrence relationship between item categories to determine the final recognition result; A feature library dynamic updating module for updating the item feature library according to the final recognition result.

9. A computer device, comprising: The computer program is executed to realize the steps of the image processing-based warehouse item intelligent recognition and classification method according to any one of claims 1-7.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed to realize the steps of the image processing-based warehouse item intelligent recognition and classification method according to any one of claims 1-7.

Images