Goods Quantity Inventory Method, Device, Equipment and Medium Based on Cross-Validation

Abstract

The present invention discloses a method, device, equipment and medium for inventorying the quantity of goods based on cross-validation. The method includes: numbering the goods in the stack to be inventoried and collecting images of the goods in the stack; classifying and identifying the stack type of the pre-pickup goods images to establish a position status matrix; performing positioning detection on the pre-pickup goods images to generate occlusion information and confidence scores; obtaining the depth information of the goods images during transportation, updating the position status matrix to generate occlusion information and confidence scores; performing positioning detection on the post-delivery goods images, updating the position status matrix to generate occlusion information and confidence scores; performing cross-validation in combination with the occlusion information to generate target occlusion information, performing cross-validation in combination with the confidence scores to generate target confidence scores, and generating an inventory result of the quantity of goods in the stack according to the target confidence scores and the target occlusion information. The present invention can improve the inventory accuracy and reliability of the quantity of goods in the stack.

Description

Technical Field

[0001] The present invention relates to the technical field of warehouse management, and particularly to a method, device, equipment and medium for inventorying the quantity of goods based on cross-validation. Background Art

[0002] In the daily operation of an automated high-rise warehouse, inventorying goods is an important task. In the traditional method, for the goods stored in the automated high-rise warehouse, when an inventory operation needs to be performed, the goods usually have to be taken down one by one for inventorying, which is obviously time-consuming and laborious. In this regard, the prior art borrows the ability of the stacker to take pictures of the storage locations from the side during driving, and obtains the goods information of the storage locations by performing corresponding processing on the taken pictures, so as to realize the automatic inventory operation and improve the efficiency of goods inventorying.

[0003] Currently, when processing pictures for automatic inventorying, a simple image classification algorithm is generally adopted. However, this kind of inventorying has many defects. For example, this kind of inventorying method is only applicable to an automated high-rise warehouse where each storage location stores a single product specification and the goods on the pallet are stacked according to the stack type rules, and its applicability to other storage scenarios is relatively low. For another example, for non-full pallets, the recognition error rate is relatively high. Moreover, when using a single picture for shooting and calculating, it is affected by the shooting conditions and the unstable characteristics of the AI algorithm, which will also lead to an increase in the recognition error rate. Therefore, how to improve the accuracy and reliability of goods inventorying is a problem that needs to be solved by those skilled in the art. Summary of the Invention

[0004] Embodiments of the present invention provide a method, device, computer equipment and storage medium for inventorying the quantity of goods based on cross-validation, aiming to improve the inventorying accuracy and reliability of the quantity of goods in the stack.

[0005] In a first aspect, embodiments of the present invention provide a method for inventorying the quantity of goods based on cross-validation, including:

[0006] Pre-numbering the goods in the stack to be inventoried according to a preset numbering system, and collecting images of the goods in the stack; wherein, the images of the goods in the stack include pre-picking goods images, in-transit goods images and post-placement goods images;

[0007] Using a classification model to perform stack type classification and recognition on the pre-picking goods images, determining the corresponding numbering system according to the results of the stack type classification and recognition, and then establishing a position status matrix in combination with the numbering system;

[0008] Using a target detection model to perform a first positioning detection on the pre-picking goods images, and generating first occlusion information and first confidence scores for each numbered position in the pre-picking goods images according to the results of the first positioning detection;

[0009] The depth information of the in-transit cargo image is obtained by using a depth estimation algorithm, and the position status matrix is first updated according to the depth information. Moreover, the spatial relationship of the in-stack cargo is inferred based on the depth information. Then, the second occlusion information and the second confidence score for each numbered position in the in-transit cargo image are generated by combining the position status matrix after the first update and the result of the spatial relationship inference.

[0010] The second positioning detection of the post-placement cargo image is performed by using an object detection model, and the position status matrix is second updated according to the result of the second positioning detection. Then, the third occlusion information and the third confidence score for each numbered position in the post-placement cargo image are generated according to the position status matrix after the second update.

[0011] The occlusion information cross-validation is performed by combining the first occlusion information, the second occlusion information, and the third occlusion information, and the target occlusion information is generated. Moreover, the cross-validation is performed by combining the first confidence score, the second confidence score, and the third confidence score, and the target confidence score is generated. Then, the in-stack cargo quantity inventory result is generated according to the target confidence score and the target occlusion information.

[0012] In a second aspect, an embodiment of the present invention provides a cargo quantity inventory device based on cross-validation, including:

[0013] A number collection unit, configured to pre-number the in-stack cargo to be inventoried according to a preset numbering system and collect in-stack cargo images; wherein, the in-stack cargo images include pre-pickup cargo images, in-transit cargo images, and post-placement cargo images.

[0014] A classification and establishment unit, configured to perform stack type classification and recognition on the pre-pickup cargo image by using a classification model, determine the corresponding numbering system according to the result of the stack type classification and recognition, and then establish a position status matrix in combination with the numbering system.

[0015] A first generation unit, configured to perform a first positioning detection on the pre-pickup cargo image by using an object detection model and generate the first occlusion information and the first confidence score for each numbered position in the pre-pickup cargo image according to the result of the first positioning detection.

[0016] A second generation unit, configured to obtain the depth information of the in-transit cargo image by using a depth estimation algorithm, first update the position status matrix according to the depth information, and infer the spatial relationship of the in-stack cargo according to the depth information. Then, the second occlusion information and the second confidence score for each numbered position in the in-transit cargo image are generated by combining the position status matrix after the first update and the result of the spatial relationship inference.

[0017] A third generation unit, configured to perform a second positioning detection on the post-delivery goods image by using a target detection model, update the position status matrix according to the result of the second positioning detection, and then generate third occlusion information and third confidence scores for each numbered position in the post-delivery goods image according to the position status matrix after the second update;

[0018] A verification and inventory unit, configured to perform cross-verification of occlusion information by combining the first occlusion information, the second occlusion information, and the third occlusion information, and generate target occlusion information, and perform cross-verification by combining the first confidence score, the second confidence score, and the third confidence score, and generate a target confidence score, and then generate an in-stack goods quantity inventory result according to the target confidence score and the target occlusion information.

[0019] In a third aspect, an embodiment of the present invention provides a computer device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, the method for inventorying goods quantity based on cross-verification as described in the first aspect is implemented.

[0020] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the method for inventorying goods quantity based on cross-verification as described in the first aspect is implemented.

[0021] An embodiment of the present invention proposes a method, a device, a computer device, and a storage medium for inventorying goods quantity based on cross-verification. The method first numbers the goods in the stack to be inventoried and collects the goods images before picking, during transportation, and after delivery. Then, the stack type is classified and identified through a classification model, a position status matrix is established, and a target detection model is used to perform positioning detection on the goods image to generate occlusion information and confidence scores. At the same time, a depth estimation algorithm is used to obtain depth information, update the position status matrix, and perform spatial relationship inference. Finally, target occlusion information and a target confidence score are generated through cross-verification, so as to generate an in-stack goods quantity inventory result, which can improve the inventory accuracy and reliability of the goods in the stack. Description of the Drawings

[0022] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings in the following description are some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0023] Figure 1Schematic flowchart of a method for inventorying the quantity of goods based on cross - validation provided by an embodiment of the present invention;

[0024] Figure 2 Schematic block diagram of a device for inventorying the quantity of goods based on cross - validation provided by an embodiment of the present invention;

[0025] Figure 3 First example diagram of a method for inventorying the quantity of goods based on cross - validation provided by an embodiment of the present invention;

[0026] Figure 4 Second example diagram of a method for inventorying the quantity of goods based on cross - validation provided by an embodiment of the present invention;

[0027] Figure 5 Third example diagram of a method for inventorying the quantity of goods based on cross - validation provided by an embodiment of the present invention;

[0028] Figure 6 Fourth example diagram of a method for inventorying the quantity of goods based on cross - validation provided by an embodiment of the present invention;

[0029] Figure 7 Fifth example diagram of a method for inventorying the quantity of goods based on cross - validation provided by an embodiment of the present invention. Detailed implementation manners

[0030] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0031] It should be understood that when used in this specification and the appended claims, the terms "include" and "comprise" indicate the presence of the described features, wholes, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components, and / or their combinations.

[0032] It should also be understood that the terms used in this specification of the present invention are merely for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in this specification of the present invention and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an", and "the" are intended to include the plural forms.

[0033] It should be further understood that the term "and / or" used in the specification and appended claims of the present invention refers to any combination and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0034] Please refer to the following Figure 1 , Figure 1 which is a schematic flowchart of a method for inventorying the quantity of goods based on cross-validation provided for an embodiment of the present invention, specifically including: steps S101 to S106.

[0035] Step S101: Pre-number the goods in the stack to be inventoried according to a preset numbering system, and collect images of the goods in the stack; wherein, the images of the goods in the stack include pre-picking goods images, in-transit goods images, and post-delivery goods images;

[0036] Step S102: Use a classification model to classify and identify the stack type of the pre-picking goods image, determine the corresponding numbering system according to the result of the stack type classification and identification, and then establish a position status matrix in combination with the numbering system;

[0037] Step S103: Use a target detection model to perform a first positioning detection on the pre-picking goods image, and generate first occlusion information and first confidence scores for each numbered position in the pre-picking goods image according to the result of the first positioning detection;

[0038] Step S104: Use a depth estimation algorithm to obtain the depth information of the in-transit goods image, perform a first update on the position status matrix according to the depth information, and perform a spatial relationship inference on the goods in the stack according to the depth information, and then generate second occlusion information and second confidence scores for each numbered position in the in-transit goods image in combination with the first updated position status matrix and the result of the spatial relationship inference;

[0039] Step S105: Use a target detection model to perform a second positioning detection on the post-delivery goods image, perform a second update on the position status matrix according to the result of the second positioning detection, and then generate third occlusion information and third confidence scores for each numbered position in the post-delivery goods image according to the second updated position status matrix;

[0040] Step S106: Perform occlusion information cross-validation by combining the first occlusion information, second occlusion information, and third occlusion information, and generate target occlusion information, and perform cross-validation by combining the first confidence score, second confidence score, and third confidence score, and generate a target confidence score, and then generate an inventory result of the quantity of goods in the stack according to the target confidence score and the target occlusion information.

[0041] In this embodiment, first, the goods in the stack are numbered, and images before picking, during transportation, and after discharging are collected. Then, the classification model and the object detection model are used to classify and identify the stack type and perform positioning detection on the pre-picking goods image, and a position status matrix is established based on this, and then the first occlusion information and the first confidence score are generated. Next, the depth information of the goods image during transportation is obtained through the depth estimation algorithm, the position status matrix is updated for the first time, and the second occlusion information and the second confidence score are generated. After that, the second positioning detection is performed on the goods image after discharging, the position status matrix is updated for the second time, and the third occlusion information and the third confidence score are generated. Then, the target occlusion information and the target confidence score are generated through the cross-validation of the occlusion information and the cross-validation of the confidence score, respectively, so as to generate the inventory result of the goods in the stack, which can improve the inventory accuracy and reliability of the goods in the stack in this way.

[0042] Through the method for inventorying the goods in the stack described in this embodiment, on the premise of being compatible with the whole-stack recognition, the correct rate of the quantity count for non-whole-stack storage locations can be greatly improved, and the misjudgment results brought by the AI algorithm can be excluded. Moreover, when the AI algorithm produces incorrect results, the errors can be automatically detected and identified. In particular, in order to facilitate taking photos of two different sides of the goods as much as possible when the inbound task is not a full stack, when the upper-level system allocates storage locations, the non-full stack can be placed on the shelves on different sides of this stacker and the picking platform (as the starting point of the stacker's inbound task) as much as possible. Since the situation of non-full stacks occurs less frequently, this requirement can usually be met by the upper-level system.

[0043] In a practical application scenario, in order to achieve image acquisition, the hardware related to the inventory of goods in the high-bay warehouse is configured, and the shooting process is preset. Specifically, in terms of basic configuration, two cameras are fixedly installed on the top of the stacker's load platform (at a position higher than the goods), and the angles are not adjustable. The two cameras are respectively oriented in the front and rear directions of the stacker (i.e., the goods location direction) to adapt to two-way access operations. At the same time, this installation method will not affect the goods loading and unloading operations. The shooting process is divided into three stages. When shooting before picking up goods, after the stacker aligns with the goods location, according to the picking direction, the camera facing the goods location is used to shoot the front of the goods. The front camera is used for forward picking, and the rear camera is used for backward picking, and the photo can present the side and top of the goods, which can be used by the classification model to judge the stack type of the goods. When shooting during transportation, the stacker uses the fork to transport the goods into the stacker. At this time, two opposite cameras are used to shoot the top of the goods located inside the stacker. Due to the limited shooting angle, when the stack is full, each photo can only capture a part of the goods at the top. By shooting with two cameras respectively, the top can be completely covered, and at this time the goods are completely off the shelf without occlusion, which is convenient for identification. When shooting after putting down the goods, after the stacker puts down the goods at the target goods location, if the goods location for putting down and the goods location for picking up are on different sides of the stacker, after putting down the goods, the camera on the opposite side of the load platform is used to shoot the side of the goods to be picked up. This photo will present one side and the top of the goods; if they are on the same side, there is no need to shoot.

[0044] In terms of image quality requirements, it is necessary to ensure that the speed of the stacker is within the range of the adapted shutter speed before shooting. At the same time, a high-speed shutter is adopted to avoid motion blur, and automatic exposure technology is used to ensure image clarity. It should be especially noted that when the warehouse uses multi-depth shelves, in order to simplify the algorithm, before picking up goods and after putting down the goods, it can be changed to shoot when the goods pass through the goods location with the shallowest depth, so that it is expected to take photos of the goods at different depth goods locations at the same position.

[0045] In another practical application scenario, combined with Figures 3 - 7 , when numbering the goods in the stack to be inventoried according to the preset numbering system, the basic numbering principles are as follows:

[0046] (1) All possible goods positions within each stack type have unique numbers;

[0047] (2) The numbering starts from 1 and increases continuously;

[0048] (3) The numbering order follows the rules from bottom to top, from front to back, and from left to right;

[0049] (4) The numbering system needs to support the correspondence of positions at different angles. For example Figures 3 - 5 shown in the top view of the first layer, the top view of the second layer, and the top view of the third layer.

[0050] The standard stack type numbering schemes are:

[0051] (1) 30-piece stack pattern (6 layers and 5 columns):

[0052] Layer 1: positions 1 - 5;

[0053] Layer 2: positions 6 - 10;

[0054] Layer 3: positions 11 - 15;

[0055] Layer 4: positions 16 - 20;

[0056] Layer 5: positions 21 - 25;

[0057] Layer 6: positions 26 - 30;

[0058] It is characterized by 5 positions per layer, facilitating layer identification;

[0059] (2) 28-piece stack pattern (7 layers and 4 columns):

[0060] Layer 1: positions 1 - 4;

[0061] Layer 2: positions 5 - 8;

[0062] Layer 3: positions 9 - 12;

[0063] Layer 4: positions 13 - 16;

[0064] Layer 5: positions 17 - 20;

[0065] Layer 6: positions 21 - 24;

[0066] Layer 7: positions 25 - 28;

[0067] It is characterized by 4 positions per layer, with a higher vertical stack.

[0068] The rule for the correspondence between the number and the spatial position is:

[0069] (1) Front view:

[0070] Numbering from left to right and from bottom to top;

[0071] Facilitating position identification during front shooting;

[0072] Example: For the 30-piece stack pattern, the first layer is 1, 2, 3, 4, 5 from left to right. Another example is Figure 6 the front view shown;

[0073] (2) Top view:

[0074] Numbering from front to back and from left to right;

[0075] Adapting to the perspective of top shooting during transportation;

[0076] Example: For the 30-piece stack pattern, from front to back and from left to right at the top layer are 26, 27, 28, 29, 30. Another example is Figures 3 - 5 as shown;

[0077] (3) Side view:

[0078] Maintain the same hierarchical relationship as the front view;

[0079] Facilitate the combination of results captured from different angles;

[0080] Example: When looking at the 30-piece stack pattern from front to back, from bottom to top in the first column are 1, 6, 11, 16, 21, 26. Another example is Figure 7 as shown.

[0081] The numbering application strategy is:

[0082] During image recognition, simultaneously output the position number and status (in stock / out of stock / occluded);

[0083] Match and cross-verify the results captured from different angles through the numbers;

[0084] The occluded position can be inferred from the status of adjacent numbered positions;

[0085] For generating the final statistical result of the quantity of goods.

[0086] In one embodiment, the step S102 includes:

[0087] Use the ResNet50 neural network to perform stack pattern classification and recognition on the pre-picking goods image to obtain the result of stack pattern classification and recognition;

[0088] Determine the corresponding stack pattern according to the result of stack pattern classification and recognition, and determine the corresponding numbering system according to the stack pattern;

[0089] Create a corresponding position status array in combination with the stack pattern and numbering system, set the initial state of the position status array, and then obtain the spatial coordinate information of each position in the position status array, thereby establishing the position status matrix.

[0090] In this embodiment, during the stack type recognition and initialization process, first, the front photos taken before picking are used for stack type classification. Through a pre-trained classification model, the input photos can be output as stack type categories, such as 28-piece or 30-piece. The backbone network used may be ResNet50 or EfficientNetB4, and these models have been pre-trained on the ImageNet dataset. To adapt to the current task, we only fine-tune the last few layers and apply data augmentation techniques, such as random rotation, brightness adjustment, and contrast adjustment, to improve the generalization ability of the model. This process is crucial for determining the location numbering system. Next, to establish the location status matrix, the corresponding number of location status arrays are created according to the stack type, and all initial states are set to "unknown". In addition, the spatial coordinate information of each location is recorded, and an adjacent relationship graph between locations is established to provide basic data support for subsequent picking and management.

[0091] In the actual application scenario, taking the numbering example of a 30-piece - neat stack type, the numbering form is as follows:

[0092] Layer 6:

[26]

[27]

[28]

[29]

[30] ← top layer

[0093] Layer 5:

[21]

[22]

[23]

[24]

[25]

[0094] Layer 4:

[16]

[17]

[18]

[19]

[20]

[0095] Layer 3:

[11]

[12]

[13]

[14]

[15]

[0096] Layer 2:

[06]

[07]

[08]

[09]

[10]

[0097] Layer 1:

[01]

[02]

[03]

[04]

[05] ← bottom layer

[0098] Status representation:

[0099] Unknown state (U);

[0100] Stocked state (A);

[0101] Out-of-stock state (B);

[0102] Occluded state (C);

[0103] Then an example of a status matrix is as follows:

[0104] Layer 6: [26:U] [27:U][28:U] [29:U][30:U]

[0105] Layer 5: [21:U] [22:U][23:U] [24:U][25:U]

[0106] Layer 4: [16:U] [17:U][18:U] [19:U][20:U]

[0107] Layer 3: [11:U] [12:U][13:U] [14:U][15:U]

[0108] Layer 2: [06:A] [07:A][08:C] [09:B][10:B]

[0109] Layer 1: [01:A] [02:A][03:A] [04:B][05:B]

[0110] Position association table taking the goods numbered 13 as an example:

[0111] Position association table = {

[0112] "13": {

[0113] "Vertical association": {

[0114] "Upper layer": "18",

[0115] "Lower layer": "08"

[0116] },

[0117] "Horizontal association": [11, 12, 14, 15],

[0118] "Status": "U",

[0119] "Confidence": 0.95

[0120] }

[0121] }。

[0122] In one embodiment, the step S103 includes:

[0123] Using the target detection model to perform goods positioning detection on each numbered position in the pre-pickup goods image;

[0124] Generating first occlusion information according to the results of the goods positioning detection; wherein, the first occlusion information includes complete occlusion, partial occlusion, and non-occlusion;

[0125] Based on the first occlusion information, determining whether there are goods at the corresponding numbered positions, thereby generating the first confidence score for each numbered position.

[0126] When performing cargo positioning detection in this embodiment, first, image preprocessing is carried out on the cargo image before picking. Specifically, it may include image enhancement, such as contrast adjustment and noise elimination, perspective correction, such as fixed correction based on the camera installation angle, and size standardization, such as unifying the image resolution, etc. Subsequently, position detection is carried out. A target detection model is used to locate each visible cargo. Specific models can select target detection models such as YOLOv8 or Faster R-CNN, and the backbone network is CSPDarknet or ResNet-FPN, etc. In the actual training process, the training strategy is to use the COCO pre-trained model in the first stage, use synthetic data for pre-training in the second stage, and use actual scenario data for fine-tuning in the third stage. The output processing of the model includes: the confidence threshold is 0.75, the NMS threshold is 0.45, and the anchor box size can be optimized according to the actual size of the cargo. Through the target detection model, the standard positions of each layer of cargo can be calculated based on the stack type rule, the detected cargo can be mapped to the numbered positions, and the initial state (with cargo / no cargo / occluded) of each position can be output.

[0127] In one embodiment, the step S104 includes:

[0128] Obtain the depth information of the in-transit cargo image using a monocular depth estimation algorithm;

[0129] Obtain the second occlusion information in the in-transit cargo image according to the depth information;

[0130] Combine the depth information and the stack type to perform spatial relationship inference on the in-transit cargo image, and obtain the result of the spatial relationship inference; wherein, the result of the spatial relationship inference includes the occlusion state of the second occlusion information, the stacking rationality of the in-stack cargo, and the abnormal stacking situation;

[0131] Generate the second confidence score for each numbered position by combining the result of the second occlusion information and the result of the spatial relationship inference.

[0132] When processing the in-transit cargo image in this embodiment, first, image preprocessing is also carried out, and the distribution of the top-layer cargo is focused on. Then, depth information is obtained through depth estimation, and the position state is updated based on the height information and depth information. When updating the position state, the position state of the unoccluded area is preferentially processed, and compared with the recognition result before picking. Then, the position state matrix is updated, and the confidence of the state change is recorded. Here, spatial relationship inference can also be carried out based on the depth information, that is, analyzing the spatial relationship by combining the stack type rule, and processing the state of the occluded position. The stacking rationality can also be verified and abnormal stacking situations can be detected.

[0133] In practical application scenarios, depth information can be obtained based on a monocular depth estimation algorithm. Among them, the main model can adopt the Depth Anything (based on the ViT architecture) model, and the alternative model can adopt the MiDaS v3.1 (suitable for indoor scenarios) model. The model configuration includes: the input resolution is 384×384 or 512×512, the backbone network is ViT-L (DepthAnything) or DPT-Large (MiDaS), and large-scale mixed datasets are used for pre-training. Model inference optimization can use half-precision (FP16) acceleration, the batch size is 1 (for real-time processing), and TensorRT acceleration is supported. In addition, post-processing is performed on the cargo images, which can specifically include depth map normalization, conversion of actual distances based on camera parameters, and detection of depth discontinuities.

[0134] Furthermore, the depth map is converted into a three-dimensional point cloud for (relative) height comparison and occlusion judgment. Specifically, first, basic parameter configuration is performed, including camera internal parameter settings:

[0135] Horizontal focal length (focal_length_x): If the camera internal parameters cannot be obtained, the default value of 470.4 is used;

[0136] Vertical focal length (focal_length_y): If the camera internal parameters cannot be obtained, the default value of 470.4 is used;

[0137] Image center point: If the camera internal parameters cannot be obtained, take the midpoint of the image size (the exact center).

[0138] It also includes depth estimation configuration:

[0139] Maximum depth value: Based on the camera hardware capabilities, it can be configured, and the default value is 20 meters.

[0140] Secondly, the processing process includes:

[0141] Depth map generation;

[0142] Use the depth estimation model to process the image;

[0143] Generate a depth map with the same proportion as the original image;

[0144] Obtain depth values based on the depth map, and this depth value represents the distance from each pixel point to the camera.

[0145] Thirdly, perform point cloud coordinate calculation:

[0146] (1) Establish an image plane grid coordinate system:

[0147] Grid coordinate calculation:

[0148] x_grid = (pixel_x - width / 2) / focal_length_x;

[0149] y_grid = (pixel_y - height / 2) / focal_length_y;

[0150] (2)Three - dimensional coordinate transformation:

[0151] Spatial coordinate calculation:

[0152] X = x_grid * depth_value;

[0153] Y = y_grid * depth_value;

[0154] Z = depth_value;

[0155] Thus, the point cloud coordinates P(X, Y, Z) are obtained. After obtaining the transformed three - dimensional point cloud P(X, Y, Z), plane detection can be performed to find an approximate horizontal plane.

[0156] The plane detection method is as follows:

[0157] (1)Principal Component Analysis (PCA):

[0158] Calculate the covariance matrix of the point cloud;

[0159] Extract three principal directions (eigenvectors);

[0160] Among them, the direction corresponding to the largest eigenvalue in the eigenvector usually represents the vertical direction, because the change of the cargo stack is the largest in the vertical direction.

[0161] The core steps of PCA are as follows: For the point cloud P(x, y, z), calculate the 3x3 covariance matrix C: C[i, j] = E[(P[i] - μ[i])(P[j] - μ[j])], where i, j ∈ {x, y, z} and μ is the mean.

[0162] In addition, three eigenvalues (λ1 ≥ λ2 ≥ λ3) can be obtained by solving the characteristic equation, and the corresponding eigenvectors (v1, v2, v3) are calculated. Among them, v1 usually corresponds to the vertical direction (the direction of the largest change), and v2 and v3 form an approximate horizontal plane. Then, direction verification and optimization are carried out. First, the gravity direction alignment is performed. Assuming that the direction of the largest eigenvalue (v1) is close to the vertical direction, the angle between v1 and the estimated gravity direction is calculated. Then, the horizontal plane direction optimization is carried out. v2 and v3 should roughly correspond to the front-back and left-right directions of the shelf. The angles between these directions and the known shelf directions are calculated, and rotation adjustment is performed if necessary. The approximate coordinate system is constructed using the finally obtained three directions.

[0163] (2)Prior constraint optimization:

[0164] Utilizing the characteristic of regular stacking of goods, the vertical direction should be approximately parallel to the gravity direction, and the two directions on the horizontal plane should be approximately aligned with the front-back and left-right directions of the shelf.

[0165] Estimate the relative height according to the constructed approximate coordinate system. Project the point cloud onto the obtained horizontal plane to obtain the approximate height. The relative height is obtained according to the missing goods and the height jump (the gap between adjacent regions is greater than a certain threshold).

[0166] In one embodiment, when processing the post-placement goods image, the processing result is mainly used for verification. The same processing flow as the pre-pickup goods image can be used. However, since the stack type has been determined after the pre-pickup goods image is recognized by the classification model, for the post-placement goods image, only the target detection algorithm can be used for positioning detection to focus on verifying the positions with uncertain states and handle the newly generated occlusion situations, so as to generate the final position state. In actual application scenarios, the depth estimation algorithm can also be used to extract the depth information of the pre-pickup goods image and the post-placement goods image respectively, so as to perform more accurate goods inventory according to the depth information.

[0167] In one embodiment, the step S106 includes:

[0168] Perform geometric occlusion analysis by combining the first occlusion information, the second occlusion information, and the third occlusion information to obtain the first occlusion analysis result. Among them, the geometric occlusion analysis includes: calculating the line-of-sight occlusion range based on the fixed camera angle, judging the foreground occlusion using the depth estimation result, establishing an occlusion relationship graph between positions, and marking the completely occluded and partially occluded areas;

[0169] Perform occlusion inference analysis by combining the first occlusion information, the second occlusion information, and the third occlusion information to obtain the second occlusion analysis result. Among them, the occlusion inference analysis includes: the lower layer support rule, the boundary constraint, the layer integrity, and inferring the unknown positions based on the known position states;

[0170] Perform occlusion cross - validation based on the first occlusion analysis result and the second occlusion analysis result, and generate target occlusion information according to the result of the occlusion cross - validation.

[0171] When generating the final inventory result of goods in this embodiment, occlusion judgment will be performed in combination with the first occlusion information, the second occlusion information, and the third occlusion information. Specifically, on the one hand, it is geometric occlusion analysis, including calculating the line - of - sight occlusion range based on the fixed angle of the camera, using the depth estimation result to judge foreground occlusion, establishing an occlusion relationship graph between positions, and marking completely occluded and partially occluded areas; on the other hand, it is stack - type rule inference, including the lower - layer support rule, that is, if there is goods in the upper layer, there must be goods in the lower layer, boundary constraints, that is, it cannot exceed the standard stack - type range, layer integrity, that is, goods in the same layer are usually placed continuously, and inferring unknown positions based on the known position status. Based on the occlusion judgment, the occlusion degree assessment and confirmation can be realized, including complete occlusion - the target position cannot be directly observed, partial occlusion - only part of the area can be observed, and unoccluded. Through the occlusion degree assessment, the visibility percentage of each position can be calculated, so that the confidence of the position status can be updated. In the actual scenario, depth information can be extracted from the goods images before picking, during transportation, and after placing respectively, and then the partial visible information at different angles is used to cross - validate the inference result and calculate the confidence of the inference result. Here, the visibility is defined as: 0, indicating complete occlusion and no target detection result can be obtained; 1, indicating that the target detection identifies "there is". The confidence sources include the confidence output by the target detection model, but the occlusion state (visibility 0) does not calculate the confidence, and the highest confidence value is taken for multiple detections. In addition, when the visibility is 1, according to the target detection result, when the visibility is 0, wait for the detection result at the subsequent visible moment.

[0172] Furthermore, verify the occlusion judgment. First, perform physical constraint verification to ensure that the inference result conforms to the gravity support principle, does not violate the standard stack - type rules, and ensures its physical feasibility, and mark the suspicious inference results. Then, perform confidence assessment. According to the different situations of directly observable positions, partially occluded positions, and completely occluded positions, use the detection confidence, visibility - adjusted confidence, and inference confidence, and set different confidence thresholds.

[0173] In the abnormal handling link, when dealing with inference conflicts, give priority to the direct observation result, and re - evaluate the status of the relevant positions. If necessary, mark them for manual review. For special situations, such as non - standard stacking patterns, occlusions in multi - deep - level shelves, and uncertain stack - type identifications, appropriate handling is carried out, and an abnormal report and suggestions are generated.

[0174] Finally, the output results include status updates and quality assessments. The status update involves updating the position status matrix, recording the basis and confidence of the inference, marking the positions that require manual confirmation, and generating the final quantity statistics. The quality assessment includes evaluating the reliability of the overall recognition, calculating the confidence of key positions, generating a detailed log of the inference process, and providing optimization suggestions.

[0175] In one embodiment, step S106 further includes:

[0176] Assign weights to the first confidence score, the second confidence score, and the third confidence score respectively according to a preset second weight, and obtain the target confidence score through weighted summation;

[0177] Combine the target occlusion information and the target confidence score to determine whether there is goods at the corresponding numbered position, so as to obtain the inventory result of the goods quantity in the stack.

[0178] Specifically, the combining the target occlusion information and the target confidence score to determine whether there is goods at the corresponding numbered position, so as to obtain the inventory result of the goods quantity in the stack, includes:

[0179] When the numbered position meets the first determination condition, it is determined that there is goods; wherein, the first determination condition is: the depth information corresponding to the numbered position shows the existence of an object, the target confidence score is greater than the first preset score, and the overlap degree between the detection frame corresponding to the numbered position and the preset standard position is greater than the first preset threshold;

[0180] When the numbered position meets the second determination condition, it is determined that there is no goods; wherein, the second determination condition is: there is no detection frame in the preset standard position for the depth information corresponding to the numbered position, the depth information shows no protrusion, and the target occlusion information of the numbered position is not occluded;

[0181] When the numbered position meets the third determination condition, it is determined that there is occlusion; wherein, the third determination condition is: the depth information corresponding to the numbered position shows the existence of occlusion, there is a detection frame of other goods in the preset standard position, or the result inferred based on the spatial relationship infers that the numbered position is in an occluded state;

[0182] When the numbered position meets the fourth determination condition, it is determined to be uncertain; wherein, the fourth determination condition is: the overlap degree between the detection frame corresponding to the numbered position and the preset standard position is within the second preset threshold, the target confidence score of the numbered position is within the second preset score, or the target occlusion information of the numbered position is partially occluded;

[0183] Statistically count the quantity of goods in the stack based on the determination result of each numbered position, so as to obtain the inventory result of the goods quantity in the stack.

[0184] In this embodiment, during the process of integrating the results, the recognition results of the three shootings are comprehensively considered, and a confidence-weighted voting mechanism is adopted to handle the situation where the results are inconsistent. Finally, a detailed position status table is generated, thus realizing the goods inventory. In addition, anomaly detection can also be performed based on content such as depth information, checking the status of violations of the stack type rules, identifying physically impossible stacking situations, and marking the positions that require manual confirmation, thereby generating an anomaly report. The corresponding output results include the final status (with goods / without goods) of each numbered position, the confidence score of each status, the marking and description of the abnormal positions, and the list of positions recommended for rechecking.

[0185] In practical applications, first, the basic status is defined. The possible statuses of each position are:

[0186] A: Confirmed with goods (confidence greater than the threshold and directly observed);

[0187] B: Confirmed without goods (confidence greater than the threshold and directly observed an empty space);

[0188] C: Occluded (cannot be directly observed);

[0189] U: Uncertain (confidence lower than the threshold).

[0190] When processing the results of a single shooting, different strategies are adopted to ensure accuracy and efficiency. First, when shooting before picking up the goods, the confidence of the object detection model is used as the basic score, and the weights of the detection results in the edge and overlapping areas are reduced. At the same time, the status and confidence of each position are recorded. When shooting during the transportation process, since the goods are not occluded, the highest weight is given to the results of this stage. In particular, the detection results of the visible area at the top are given the highest confidence, and special attention is paid to the positions that are inconsistent with the previous results to ensure the accurate monitoring of the status of the goods during the entire transportation process. Finally, when shooting after putting down the goods, the same processing method as before picking up the goods is adopted, focusing on verifying the uncertain positions in the previous two results and giving appropriate weights to the new perspective information, which helps to comprehensively understand the status change of the goods.

[0191] When processing the results of multiple shootings, a confidence-weighted method is used to merge these results. For example, the weight of the photos taken during transportation is 0.5, the weight of the photos taken before picking up the goods is 0.3, and the weight of the photos taken after delivering the goods is 0.2. The final confidence level is obtained by multiplying the confidence level of each shooting by the corresponding weight and summing them up. In terms of status determination, the following rules are followed: If there is a conflict between status A and status B in any two shootings, mark this position as abnormal; if a certain position obtains status A in any one shooting and there is no conflict, then finally determine that this position is in status A; if a certain position obtains status B in any one shooting and there is no conflict, then finally determine that this position is in status B; if a certain position only obtains status C, it is necessary to enter the occlusion processing process; if the statuses are inconsistent but the confidence levels of all statuses are relatively low, mark this position as status U.

[0192] When dealing with conflicts, first, it is necessary to detect level conflicts, which involve contradictions between different shooting results of the same position. The methods to solve this problem include checking the confidence levels of each detection and referring to the status of adjacent positions. If the problem still cannot be solved, it needs to be marked for manual review. Second, stack pattern rule conflicts also need to be concerned about, which include situations that violate the physical stacking rules, such as having goods on the upper layer but no goods on the lower layer, or the detection results exceeding the standard stack pattern range. For these situations, the system should automatically adjust to a reasonable state or mark it as abnormal for further processing.

[0193] In practical applications, the basic confidence level can be defined first, which includes the probability value output by the object detection model, the clarity of the object in the image, and whether there is partial occlusion. Subsequently, considering several adjustment factors, including perspective adaptability (adjusted according to the shooting angle), distance attenuation (the farther the object is from the camera, the lower the confidence level), and occlusion impact (the higher the occlusion degree, the lower the confidence level), to obtain the final confidence level. The calculation method of the final confidence level is the basic confidence level multiplied by the product of all adjustment factors. To ensure the reliability of the results, a minimum confidence level threshold can be set, such as 0.6. Any result below this threshold will be marked as status U to indicate uncertainty. Further, when outputting the final inventory results, the final status (A / B / C / U) of each numbered position, the comprehensive confidence level of each status, and the abnormal mark and reason description, etc. can be output. At the same time, it is also possible to recommend rechecking the positions with low confidence levels, detection conflicts, and / or violations of stack pattern rules, and generate a manual recheck list.

[0194] In a specific embodiment, when determining the position status, the following determination strategy can be specifically adopted:

[0195] (1) Conditions for determining the presence of goods (A) (all of the following conditions need to be met):

[0196] AND (

[0197] The overlap degree between the detection box and the standard position is > 70%

[0198] The detection confidence is > 0.75

[0199] Depth estimation shows that there is an object at this position )

[0201] (2)No goods (B) determination conditions (all of the following conditions need to be met):

[0202] AND (

[0203] There are no detection boxes within the field of view at this position

[0204] The depth map shows that there are no obvious protrusions in this area

[0205] This position is completely visible (unoccluded) )

[0207] (3)Occlusion (C) determination conditions (meeting any of the following conditions):

[0208] OR (

[0209] An occlusion object is detected in front through the depth map

[0210] The expected field of view at this position is occupied by other goods detection boxes

[0211] It is inferred that this position should be in the occlusion area according to the stack type rules )

[0213] (4)Uncertain (U) determination conditions (meeting any of the following conditions):

[0214] OR (

[0215] The overlap degree between the detection box and the position is between 30% - 70%

[0216] The detection confidence is between 0.6 - 0.75

[0217] Partially occluded but still partially visible

[0218] Do not meet the determination conditions of any of the above A / B / C states )

[0220] Among them, when the conditions of multiple states are met simultaneously, the following priority is processed:

[0221] (1)Occlusion (C): If the occlusion condition is met, it is preferentially determined as occlusion;

[0222] (2)Goods available (A): If all the conditions for goods available are met and it is not determined as occluded;

[0223] (3) Out of stock (B): If all out-of-stock conditions are met and it is not determined to be occluded or in stock;

[0224] (4) Uncertain (U): All other cases.

[0225] For the processing method of the final state being uncertain for the three types of photo determinations, specifically:

[0226] (1) Trigger conditions. Specifically, it includes that the position is in an uncertain (U) state in all three shootings or remains uncertain after result merging, and the definite state cannot be inferred through occlusion processing;

[0227] (2) Processing strategies. It includes automatic processing attempts, such as checking the status of adjacent positions, applying stack type rule constraints, and analyzing the common status of this storage location in historical data, etc. If the automatic processing fails, this position can be marked as "pending manual review" and marked as the highest priority in the exception report. At the same time, data recording is carried out, including recording the specific reasons leading to uncertainty, recording all relevant detection data and images for subsequent algorithm optimization and model improvement.

[0228] The following takes a complete operation of stack type A with 30 pieces as an example to illustrate the processing process:

[0229] (1) Initial scenario, a picking operation is carried out for a storage location with 24 pieces of goods (the 6th layer is vacant);

[0230] (2) Shooting results before picking. Front view results:

[0231] Layers 1 - 4 (positions 1 - 20): Visibility 1, target detection confidence 0.85 - 0.92;

[0232] The 5th layer (positions 21 - 25): Visibility 0, completely occluded by the front device;

[0233] The 6th layer (positions 26 - 30): Visibility 1, target detection confirms out of stock, confidence 0.88 - 0.90.

[0234] (2) Shooting results during transportation. Front camera shooting:

[0235] The 5th layer: Visibility 1, target detection confidence 0.95;

[0236] The 6th layer: Visibility 1, confirmed out of stock, confidence 0.96.

[0237] Rear camera shooting:

[0238] The 5th layer: Visibility 1, target detection confidence 0.94;

[0239] Layer 6: Visibility 1, out of stock confirmed, confidence level 0.95.

[0240] (3)Results of shooting after goods release. Observation from the side:

[0241] Layers 1 - 4: Visibility 1, object detection confidence level 0.82 - 0.89;

[0242] Layer 5: Visibility 1, object detection confidence level 0.91;

[0243] Layer 6: Visibility 1, out of stock confirmed, confidence level 0.92.

[0244] (4)Process of result merging. Taking Layer 5 as an example:

[0245] Before picking up the goods: Visibility 0, no confidence level;

[0246] During transportation: Visibility 1, confidence levels 0.95 (front camera) and 0.94 (rear camera);

[0247] After goods release: Visibility 1, confidence level 0.91;

[0248] Then the merged calculation is as follows:

[0249] Status determination: When visible, goods are detected, determined to have goods;

[0250] Final confidence level: Take the highest confidence level 0.95.

[0251] Regarding the occlusion processing among them, taking the occlusion situation of Layer 5 as an example:

[0252] (1)Occlusion analysis. Status of Layer 5 during shooting before picking up the goods:

[0253] Occlusion source: Stacker structure; Occlusion determination: Positions 21 - 25: Visibility is 0 for all; Completely occluded, unable to obtain object detection results;

[0254] (2)Inference process.

[0255] (a)Geometric relationship analysis:

[0256] It is known that all positions 16 - 20 of Layer 4 have goods (visibility 1, confidence level > 0.85); It is known that all positions 26 - 30 of Layer 6 have no goods (visibility 1, confidence level > 0.88); Wait for the direct observation results obtained during transportation shooting;

[0257] (b)Observation during transportation:

[0258] Visibility becomes 1; Obtain object detection confidence levels 0.94 - 0.95; Direct observation confirms there are goods, no need for inference;

[0259] (3)Verification and confirmation.

[0260] During transportation, both cameras directly observed the goods. After the goods were unloaded, direct observation was carried out again for confirmation, and all direct observation results were consistent;

[0261] (4)Result output,

[0262] Status: Confirmed with goods;

[0263] Confidence level: 0.95 (using the highest confidence level);

[0264] Visibility record: [0, 1, 1] (before picking up the goods, during transportation, after unloading the goods).

[0265] Figure 2 It is a schematic block diagram of a device 200 for inventory counting of goods based on cross - validation provided by an embodiment of the present invention. The device 200 includes:

[0266] A number acquisition unit 201, configured to pre - number the goods in the stack to be counted according to a preset numbering system and acquire the images of the goods in the stack; wherein, the images of the goods in the stack include the images of the goods before picking up, during transportation, and after unloading;

[0267] A classification and establishment unit 202, configured to use a classification model to perform stack type classification and recognition on the images of the goods before picking up, determine the corresponding numbering system according to the result of the stack type classification and recognition, and then establish a position status matrix in combination with the numbering system;

[0268] A first generation unit 203, configured to use a target detection model to perform a first positioning detection on the images of the goods before picking up, and generate first occlusion information and first confidence scores for each numbered position in the images of the goods before picking up according to the result of the first positioning detection;

[0269] A second generation unit 204, configured to use a depth estimation algorithm to obtain the depth information of the images of the goods during transportation, perform a first update on the position status matrix according to the depth information, and perform spatial relationship inference on the goods in the stack according to the depth information, and then generate second occlusion information and second confidence scores for each numbered position in the images of the goods during transportation in combination with the first - updated position status matrix and the result of the spatial relationship inference;

[0270] A third generation unit 205, configured to use a target detection model to perform a second positioning detection on the images of the goods after unloading, perform a second update on the position status matrix according to the result of the second positioning detection, and then generate third occlusion information and third confidence scores for each numbered position in the images of the goods after unloading according to the second - updated position status matrix;

[0271] The verification and inventory unit 206 is used to perform cross-verification of occlusion information by combining the first occlusion information, the second occlusion information, and the third occlusion information, and generate target occlusion information, and perform cross-verification by combining the first confidence score, the second confidence score, and the third confidence score, and generate a target confidence score, and then generate an inventory result of the number of goods in the stack according to the target confidence score and the target occlusion information.

[0272] In one embodiment, the classification and establishment unit 202 includes:

[0273] The classification and recognition unit is used to perform stack type classification and recognition on the pre-picking goods image by using the ResNet50 neural network to obtain the result of stack type classification and recognition;

[0274] The recognition and determination unit is used to determine the corresponding stack type according to the result of stack type classification and recognition, and determine the corresponding numbering system according to the stack type;

[0275] The matrix establishment unit is used to create a corresponding position status array by combining the stack type and the numbering system, set the initial state of the position status array, and then obtain the position space coordinate information of each position in the position status array, so as to establish the position status matrix.

[0276] In one embodiment, the first generation unit 203 includes:

[0277] The positioning and detection unit is used to perform goods positioning and detection on each numbered position in the pre-picking goods image by using a target detection model;

[0278] The first occlusion generation unit is used to generate first occlusion information according to the result of goods positioning and detection; wherein, the first occlusion information includes complete occlusion, partial occlusion, and non-occlusion;

[0279] The first score generation unit is used to judge whether there is a good on the corresponding numbered position based on the first occlusion information, so as to generate the first confidence score of each numbered position.

[0280] In one embodiment, the second generation unit 204 includes:

[0281] The information acquisition unit is used to obtain the depth information of the in-transit goods image by using a monocular depth estimation algorithm;

[0282] The second occlusion generation unit is used to obtain the second occlusion information in the in-transit goods image according to the depth information;

[0283] A spatial inference unit for inferring the spatial relationship of the in-transit cargo image by combining the depth information and the stack type, and obtaining the result of the spatial relationship inference; wherein, the result of the spatial relationship inference includes the occlusion state of the second occlusion information, the stacking rationality of the in-stack cargo, and the stack abnormal situation;

[0284] A second score generation unit for generating a second confidence score for each numbered position by combining the result of the second occlusion information and the result of the spatial relationship inference.

[0285] In one embodiment, the verification and inventory unit 206 includes:

[0286] A geometric occlusion analysis unit for performing geometric occlusion analysis by combining the first occlusion information, the second occlusion information, and the third occlusion information, and obtaining a first occlusion analysis result; wherein, the geometric occlusion analysis includes: calculating the line-of-sight occlusion range based on the fixed camera angle, judging foreground occlusion using the depth estimation result, establishing an occlusion relationship graph between positions, and marking the completely occluded and partially occluded areas;

[0287] An occlusion inference analysis unit for performing occlusion inference analysis by combining the first occlusion information, the second occlusion information, and the third occlusion information, and obtaining a second occlusion analysis result; wherein, the occlusion inference analysis includes: lower layer support rule, boundary constraint, hierarchical integrity, and inferring unknown positions based on known position states;

[0288] An occlusion cross-validation unit for performing occlusion cross-validation based on the first occlusion analysis result and the second occlusion analysis result, and generating target occlusion information according to the result of the occlusion cross-validation.

[0289] In one embodiment, the verification and inventory unit 206 further includes:

[0290] An assignment and weighting unit for assigning weights to the first confidence score, the second confidence score, and the third confidence score respectively according to a preset second weight, and obtaining the target confidence score by weighted summation;

[0291] A goods determination unit for determining whether there is goods at the corresponding numbered position by combining the target occlusion information and the target confidence score, so as to obtain the inventory result of the in-stack goods quantity.

[0292] In one embodiment, the goods determination unit includes:

[0293] The first determination unit is configured to determine that there is goods when the numbered position meets the first determination condition; wherein, the first determination condition is that: the depth information corresponding to the numbered position shows the existence of an object, the target confidence score is greater than the first preset score, and the overlap degree between the detection frame corresponding to the numbered position and the preset standard position is greater than the first preset threshold;

[0294] The second determination unit is configured to determine that there is no goods when the numbered position meets the second determination condition; wherein, the second determination condition is that: there is no detection frame in the preset standard position for the depth information corresponding to the numbered position, the depth information shows no protrusion, and the target occlusion information of the numbered position is not occluded;

[0295] The third determination unit is configured to determine that it is occluded when the numbered position meets the third determination condition; wherein, the third determination condition is that: the depth information corresponding to the numbered position shows the existence of occlusion, there is a detection frame of other goods in the preset standard position, or the result inferred based on the spatial relationship infers that the numbered position is in an occluded state;

[0296] The fourth determination unit is configured to determine that it is uncertain when the numbered position meets the fourth determination condition; wherein, the fourth determination condition is that: the overlap degree between the detection frame corresponding to the numbered position and the preset standard position is within the second preset threshold, the target confidence score of the numbered position is within the second preset score, or the target occlusion information of the numbered position is partially occluded;

[0297] The quantity statistics unit is configured to statistically count the quantity of goods in the stack based on the determination result of each numbered position, so as to obtain the inventory result of the quantity of goods in the stack.

[0298] Since the embodiments of the device part correspond to the embodiments of the method part, for the embodiments of the device part, please refer to the description of the embodiments of the method part, and will not be elaborated here for the time being.

[0299] The embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed, the steps provided in the above embodiments can be implemented. The storage medium may include: various media such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disc that can store program codes.

[0300] The embodiment of the present invention further provides a computer device, which may include a memory and a processor. When the processor calls the computer program stored in the memory, the steps provided in the above embodiments can be implemented. Of course, the computer device may further include various network interfaces, power supplies and other components.

[0301] The various embodiments in the specification are described in a progressive manner. Each embodiment focuses on the differences from other embodiments, and the same or similar parts among the embodiments can be referred to each other. For the system disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the description of the method part. It should be noted that for those of ordinary skill in the art in this technical field, without departing from the principle of this application, several improvements and modifications can be made to this application, and these improvements and modifications also fall within the protection scope of the claims of this application.

[0302] It should also be noted that in this specification, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements, but also includes other elements not expressly listed, or also includes elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "including a..." does not exclude the existence of another identical element in the process, method, article or device including the said element.

Claims

1. A method for inventorying the quantity of goods based on cross-validation, characterized in that, Including: Pre-number the goods in the stack to be inventoried according to a preset numbering system in advance, and collect images of the goods in the stack; wherein, the images of the goods in the stack include pre-picking goods images, in-transit goods images, and post-placement goods images; Use a classification model to perform stack type classification and recognition on the pre-picking goods images, determine the corresponding numbering system according to the results of the stack type classification and recognition, and then establish a position status matrix in combination with the numbering system; Use an object detection model to perform the first positioning detection on the pre-picking goods images, and generate the first occlusion information and the first confidence score for each numbered position in the pre-picking goods images according to the results of the first positioning detection; Use a depth estimation algorithm to obtain the depth information of the in-transit goods images, perform the first update on the position status matrix according to the depth information, and perform spatial relationship inference on the goods in the stack according to the depth information, and then generate the second occlusion information and the second confidence score for each numbered position in the in-transit goods images in combination with the results of the first updated position status matrix and the spatial relationship inference; Use an object detection model to perform the second positioning detection on the post-placement goods images, perform the second update on the position status matrix according to the results of the second positioning detection, and then generate the third occlusion information and the third confidence score for each numbered position in the post-placement goods images according to the position status matrix after the second update; Combine the first occlusion information, the second occlusion information, and the third occlusion information for occlusion information cross-verification, and generate target occlusion information, and combine the first confidence score, the second confidence score, and the third confidence score for cross-verification, and generate a target confidence score, and then generate an inventory result of the quantity of goods in the stack according to the target confidence score and the target occlusion information; The combining the first occlusion information, the second occlusion information, and the third occlusion information for occlusion information cross-verification, and generating target occlusion information includes: Combining the first occlusion information, the second occlusion information, and the third occlusion information for geometric occlusion analysis to obtain a first occlusion analysis result; wherein, the geometric occlusion analysis includes: calculating the line-of-sight occlusion range based on the fixed camera angle, using the depth estimation result to judge foreground occlusion, establishing an occlusion relationship graph between positions, and marking completely occluded and partially occluded areas; Combining the first occlusion information, the second occlusion information, and the third occlusion information for occlusion inference analysis to obtain a second occlusion analysis result; wherein, the occlusion inference analysis includes: lower layer support rule, boundary constraint, hierarchical integrity, and inferring unknown positions according to known position status; Perform occlusion cross-verification based on the first occlusion analysis result and the second occlusion analysis result, and generate target occlusion information according to the results of the occlusion cross-verification.

2. The method for checking the quantity of goods based on cross-validation according to claim 1, wherein The using a classification model to perform stack type classification and recognition on the pre-picking goods images, determining the corresponding numbering system according to the results of the stack type classification and recognition, and establishing a position status matrix according to the results of the stack type classification and recognition includes: Use the ResNet50 neural network to perform stack type classification and recognition on the pre-pickup goods image to obtain the result of stack type classification and recognition; Determine the corresponding stack type according to the result of stack type classification and recognition, and determine the corresponding numbering system according to the stack type; Create a corresponding position status array in combination with the stack type and numbering system, set the initial state of the position status array, and then obtain the position space coordinate information of each position in the position status array, thereby establishing the position status matrix.

3. The method for inventorying the quantity of goods based on cross-validation according to claim 1, wherein The first positioning detection of the pre-pickup goods image is performed using the object detection model, and the first occlusion information and the first confidence score of each numbered position in the pre-pickup goods image are generated according to the result of the first positioning detection, including: Use the object detection model to perform goods positioning detection on each numbered position in the pre-pickup goods image; Generate the first occlusion information according to the result of the goods positioning detection; wherein, the first occlusion information includes complete occlusion, partial occlusion, and unoccluded; Based on the first occlusion information, judge whether there is a good on the corresponding numbered position, thereby generating the first confidence score of each numbered position.

4. The method for inventorying the quantity of goods based on cross-validation according to claim 1, wherein The depth information of the in-transit goods image is obtained using the depth estimation algorithm, and the position status matrix is first updated according to the depth information, and the spatial relationship of the goods in the stack is inferred according to the depth information, and then the second occlusion information and the second confidence score of each numbered position in the in-transit goods image are generated in combination with the first-updated position status matrix and the result of the spatial relationship inference, including: Use the monocular depth estimation algorithm to obtain the depth information of the in-transit goods image; Obtain the second occlusion information in the in-transit goods image according to the depth information; Perform spatial relationship inference on the in-transit goods image in combination with the depth information and the stack type to obtain the result of the spatial relationship inference; wherein, the result of the spatial relationship inference includes the occlusion state of the second occlusion information, the stacking rationality of the goods in the stack, and the stack abnormality; Generate the second confidence score of each numbered position in combination with the result of the second occlusion information and the result of the spatial relationship inference.

5. The method for checking the quantity of goods based on cross-validation according to claim 1, wherein The first occlusion information, the second occlusion information, and the third occlusion information are combined for occlusion information cross-verification, and the target occlusion information is generated, and the first confidence score, the second confidence score, and the third confidence score are combined for cross-verification, and the target confidence score is generated, and then the in-stack goods quantity inventory result is generated according to the target confidence score and the target occlusion information, further including: Assign weights to the first confidence score, the second confidence score, and the third confidence score respectively according to the preset second weight, and obtain the target confidence score by weighted summation; Combine the target occlusion information and the target confidence score to determine whether there is a good on the corresponding numbered position, thereby obtaining the in-stack goods quantity inventory result.

6. The method for inventorying the quantity of goods based on cross-validation according to claim 5, wherein, The combination of the target occlusion information and the target confidence score to determine whether there is a good on the corresponding numbered position, thereby obtaining the in-stack goods quantity inventory result, includes: When the numbered position meets the first determination condition, it is determined that there is goods; wherein, the first determination condition is: the depth information corresponding to the numbered position shows the existence of an object, the target confidence score is greater than the first preset score, and the overlap degree between the detection frame corresponding to the numbered position and the preset standard position is greater than the first preset threshold; When the numbered position meets the second determination condition, it is determined that there is no goods; wherein, the second determination condition is: there is no detection frame in the preset standard position for the depth information corresponding to the numbered position, the depth information shows no protrusion, and the target occlusion information of the numbered position is not occluded; When the numbered position meets the third determination condition, it is determined that there is occlusion; wherein, the third determination condition is: the depth information corresponding to the numbered position shows the existence of occlusion, there is a detection frame of other goods in the preset standard position, or it is inferred based on the result of spatial relationship inference that the numbered position is in an occluded state; When the numbered position meets the fourth determination condition, it is determined to be uncertain; wherein, the fourth determination condition is: the overlap degree between the detection frame corresponding to the numbered position and the preset standard position is within the second preset threshold, the target confidence score of the numbered position is within the second preset score, or the target occlusion information of the numbered position is partially occluded; Based on the determination results of each numbered position, the number of goods in the stack is counted to obtain the inventory result of the number of goods in the stack.

7. A device for inventorying the quantity of goods based on cross-validation, characterized in that, Including: A number collection unit for pre-numbering the goods in the stack to be inventoried according to a preset numbering system and collecting images of the goods in the stack; wherein, the images of the goods in the stack include images of the goods before picking, images of the goods during transportation, and images of the goods after placing; A classification and establishment unit for using a classification model to classify and identify the stack type of the image of the goods before picking, determining the corresponding numbering system according to the result of the stack type classification and identification, and then establishing a position status matrix in combination with the numbering system; A first generation unit for using a target detection model to perform a first positioning detection on the image of the goods before picking, and generating the first occlusion information and the first confidence score of each numbered position in the image of the goods before picking according to the result of the first positioning detection; A second generation unit for using a depth estimation algorithm to obtain the depth information of the image of the goods during transportation, performing a first update on the position status matrix according to the depth information, and performing a spatial relationship inference on the goods in the stack according to the depth information, and then generating the second occlusion information and the second confidence score of each numbered position in the image of the goods during transportation in combination with the position status matrix after the first update and the result of the spatial relationship inference; A third generation unit for using a target detection model to perform a second positioning detection on the image of the goods after placing, performing a second update on the position status matrix according to the result of the second positioning detection, and then generating the third occlusion information and the third confidence score of each numbered position in the image of the goods after placing according to the position status matrix after the second update; A verification inventory unit is used to perform cross-verification of occlusion information by combining the first occlusion information, the second occlusion information, and the third occlusion information, and generate target occlusion information, and perform cross-verification by combining the first confidence score, the second confidence score, and the third confidence score, and generate a target confidence score, and then generate an in-stack goods quantity inventory result according to the target confidence score and the target occlusion information; The verification inventory unit includes: A geometric occlusion analysis unit is used to perform geometric occlusion analysis by combining the first occlusion information, the second occlusion information, and the third occlusion information to obtain a first occlusion analysis result; wherein, the geometric occlusion analysis includes: calculating a line-of-sight occlusion range based on a fixed camera angle, using a depth estimation result to judge foreground occlusion, establishing an occlusion relationship graph between positions, and marking completely occluded and partially occluded areas; An occlusion inference analysis unit is used to perform occlusion inference analysis by combining the first occlusion information, the second occlusion information, and the third occlusion information to obtain a second occlusion analysis result; wherein, the occlusion inference analysis includes: lower layer support rules, boundary constraints, hierarchical integrity, and inferring unknown positions based on known position states; An occlusion cross-verification unit is used to perform occlusion cross-verification based on the first occlusion analysis result and the second occlusion analysis result, and generate target occlusion information according to the result of the occlusion cross-verification.

8. A computer device, characterized in that, It includes a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, it implements the cross-verification-based goods quantity inventory method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, A computer program is stored on the computer-readable storage medium. When the computer program is executed by a processor, it implements the cross-verification-based goods quantity inventory method according to any one of claims 1 to 6.

Images