Cargo quantity checking method and device based on cross validation, equipment and medium

Abstract

The invention discloses a cargo quantity checking method and device based on cross validation, equipment and a medium, and the method comprises the steps: numbering cargoes to be checked in a stack, and collecting the images of the cargoes in the stack; carrying out stack type classification and identification on the goods image before taking the goods, and establishing a position state matrix; carrying out positioning detection on the goods image before the goods are taken, and generating shielding information and a confidence score; acquiring depth information of the image of the cargo in carrying, updating the position state matrix, and generating shielding information and a confidence score; carrying out positioning detection on the cargo image after cargo placement, updating the position state matrix, and generating shielding information and a confidence score; and performing cross validation in combination with the shielding information, generating target shielding information, performing cross validation in combination with the confidence score, generating a target confidence score, and generating an in-stack cargo quantity checking result according to the target confidence score and the target shielding information. According to the invention, the checking precision and the checking reliability of the number of goods in the stack can be improved.

Description

Technical Field

[0001] The present invention relates to the technical field of warehouse management, and in particular to a method, device, equipment and medium for counting the quantity of goods based on cross-validation. Background Art

[0002] In the daily operation of high-bay warehouses, inventory counting is an important task. Traditionally, when it is necessary to perform inventory counting on goods stored in high-bay warehouses, the goods can only be taken down and counted one by one, which is obviously time-consuming and labor-intensive. In this regard, the existing technology uses the ability of the stacker to take pictures of the cargo space from the side during driving, and obtains the cargo information of the cargo space by processing the photos taken accordingly, thereby realizing automatic inventory counting operations and improving the efficiency of cargo counting.

[0003] At present, when automatically counting and processing pictures, a simple image classification algorithm is generally used. However, this type of inventory has many defects. For example, this inventory method is only suitable for high-bay warehouses where each cargo space stores a single specification and the goods on the pallets are stacked according to the stacking rules. It is less applicable to other storage scenarios. For another example, for pallets that are not full, the recognition error rate is high. In addition, when using a single picture for calculation, it will be affected by the shooting conditions and the unstable characteristics of the AI ​​algorithm, which will also lead to an increase in the recognition error rate. Therefore, how to improve the accuracy and reliability of cargo inventory is a problem that technicians in this field need to solve. Summary of the invention

[0004] The embodiments of the present invention provide a method, device, computer equipment and storage medium for counting the number of goods based on cross-validation, aiming to improve the counting accuracy and reliability of the number of goods in a stack.

[0005] In a first aspect, an embodiment of the present invention provides a method for counting the quantity of goods based on cross-validation, comprising: The goods in the stack to be counted are numbered in advance according to a preset numbering system, and images of the goods in the stack are collected; wherein the images of the goods in the stack include images of the goods before picking up the goods, images of the goods in transit, and images of the goods after releasing the goods; Using the classification model to classify and identify the stack type of the cargo image before picking up the cargo, and determining the corresponding numbering system according to the result of the stack type classification and identification, and then establishing a position state matrix in combination with the numbering system; Performing a first positioning detection on the cargo image before pickup using a target detection model, and generating first occlusion information and a first confidence score for each numbered position in the cargo image before pickup according to a result of the first positioning detection; Using a depth estimation algorithm to obtain depth information of the image of the goods in transit, and performing a first update on the position state matrix according to the depth information, and performing spatial relationship inference on the goods in the stack according to the depth information, and then combining the position state matrix after the first update and the result of the spatial relationship inference to generate second occlusion information and a second confidence score for each numbered position in the image of the goods in transit; Performing a second positioning detection on the cargo image after cargo release by using the target detection model, and performing a second updating on the position state matrix according to the result of the second positioning detection, and then generating third occlusion information and a third confidence score for each numbered position in the cargo image after cargo release according to the second updated position state matrix; The first occlusion information, the second occlusion information and the third occlusion information are combined to perform cross-validation of the occlusion information and generate target occlusion information, and the first confidence score, the second confidence score and the third confidence score are combined to perform cross-validation and generate a target confidence score, and then an inventory result of the quantity of goods in the stack is generated based on the target confidence score and the target occlusion information.

[0006] In a second aspect, an embodiment of the present invention provides a device for counting the number of goods based on cross-validation, comprising: A number collection unit is used to pre-number the goods in the stack to be counted according to a preset numbering system, and collect images of the goods in the stack; wherein the images of the goods in the stack include images of the goods before picking up, images of the goods in transit, and images of the goods after releasing; A classification establishment unit, used for classifying and identifying the type of goods before picking up the goods by using a classification model, and determining a corresponding numbering system according to the result of the classification and identification of the type of goods, and then establishing a position state matrix in combination with the numbering system; a first generating unit, configured to perform a first positioning detection on the cargo image before pickup using a target detection model, and generate first occlusion information and a first confidence score for each numbered position in the cargo image before pickup according to a result of the first positioning detection; a second generating unit, configured to obtain depth information of the image of the goods in transit by using a depth estimation algorithm, and to perform a first update on the position state matrix according to the depth information, and to perform a spatial relationship inference on the goods in the stack according to the depth information, and then to generate second occlusion information and a second confidence score for each numbered position in the image of the goods in transit by combining the position state matrix after the first update and the result of the spatial relationship inference; a third generating unit, configured to perform a second positioning detection on the cargo image after cargo release by using the target detection model, and perform a second updating on the position state matrix according to the result of the second positioning detection, and then generate third occlusion information and a third confidence score for each numbered position in the cargo image after cargo release according to the position state matrix after the second updating; A verification and inventory unit is used to cross-validate the occlusion information in combination with the first occlusion information, the second occlusion information and the third occlusion information, and generate target occlusion information, and to cross-validate the first confidence score, the second confidence score and the third confidence score, and generate a target confidence score, and then generate an inventory result of the quantity of goods in the stack based on the target confidence score and the target occlusion information.

[0007] In a third aspect, an embodiment of the present invention provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the cross-validation-based cargo quantity inventory method as described in the first aspect is implemented.

[0008] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the method for counting the quantity of goods based on cross-validation as described in the first aspect is implemented.

[0009] The embodiment of the present invention proposes a method, device, computer equipment and storage medium for counting the number of goods based on cross-validation. The method first numbers the goods in the stack to be counted and collects the images of the goods before picking up, during transportation and after releasing the goods. Then, the stack type classification and recognition are performed through the classification model, the position state matrix is ​​established, and the target detection model is used to locate the goods image to generate occlusion information and confidence score. At the same time, the depth estimation algorithm is used to obtain the depth information, update the position state matrix, and perform spatial relationship inference. Finally, the target occlusion information and the target confidence score are generated through cross-validation, thereby generating the inventory result of the number of goods in the stack, so as to improve the inventory accuracy and reliability of the goods in the stack. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying any creative work.

[0011] Figure 1 A schematic diagram of a flow chart of a method for counting the quantity of goods based on cross-validation provided by an embodiment of the present invention; Figure 2 A schematic block diagram of a device for counting the number of goods based on cross-validation provided by an embodiment of the present invention; Figure 3 A first exemplary diagram of a method for counting the number of goods based on cross-validation provided by an embodiment of the present invention; Figure 4 A second exemplary diagram of a method for counting the number of goods based on cross-validation provided by an embodiment of the present invention; Figure 5 A third exemplary diagram of a method for counting the number of goods based on cross-validation provided by an embodiment of the present invention; Figure 6 A fourth example diagram of a method for counting the number of goods based on cross-validation provided by an embodiment of the present invention; Figure 7 A fifth example diagram of a method for counting the number of goods based on cross-validation provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0012] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0013] It should be understood that when used in this specification and the appended claims, the terms "include" and "comprises" indicate the presence of described features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.

[0014] It should also be understood that the terms used in this specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the specification of the present invention and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0015] It should be further understood that the term "and / or" used in the present description and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

[0016] See below Figure 1 , Figure 1A schematic flow chart of a method for counting the quantity of goods based on cross-validation provided in an embodiment of the present invention specifically includes: steps S101 to S106.

[0017] Step S101, numbering the goods in the stack to be counted in advance according to a preset numbering system, and collecting images of the goods in the stack; wherein the images of the goods in the stack include images of the goods before picking up the goods, images of the goods in transit, and images of the goods after releasing the goods; Step S102, using the classification model to classify and identify the stack type of the cargo image before picking up the cargo, and determining the corresponding numbering system according to the result of the stack type classification and identification, and then establishing a position state matrix in combination with the numbering system; Step S103: performing a first positioning detection on the cargo image before pickup using the target detection model, and generating first occlusion information and a first confidence score for each numbered position in the cargo image before pickup according to the result of the first positioning detection; Step S104, using a depth estimation algorithm to obtain depth information of the image of the goods in transit, and performing a first update on the position state matrix according to the depth information, and performing spatial relationship inference on the goods in the stack according to the depth information, and then combining the position state matrix after the first update and the result of the spatial relationship inference to generate second occlusion information and a second confidence score for each numbered position in the image of the goods in transit; Step S105: performing a second positioning detection on the cargo image after cargo release by using the target detection model, and performing a second updating on the position state matrix according to the result of the second positioning detection, and then generating third occlusion information and a third confidence score for each numbered position in the cargo image after cargo release according to the position state matrix after the second updating; Step S106, cross-validating the occlusion information in combination with the first occlusion information, the second occlusion information and the third occlusion information, and generating target occlusion information, and cross-validating the first confidence score, the second confidence score and the third confidence score, and generating a target confidence score, and then generating an inventory result of the quantity of goods in the stack based on the target confidence score and the target occlusion information.

[0018] In the present embodiment, first, the goods in the stack are numbered and the images before picking up, during transportation and after placing the goods are collected. Then, the classification model and the target detection model are used to perform stack type classification and location detection on the goods image before picking up the goods, and a position state matrix is ​​established to thereby generate the first occlusion information and the first confidence score. Next, the depth information of the goods image in transportation is obtained by a depth estimation algorithm, the position state matrix is ​​updated for the first time, and the second occlusion information and the second confidence score are generated. Afterwards, the goods image after placing the goods is positioned for the second time, the position state matrix is ​​updated for the second time, and the third occlusion information and the third confidence score are generated. Then, the target occlusion information and the target confidence score are generated respectively by cross-validation of the occlusion information and cross-validation of the confidence score, thereby generating the inventory result of the goods quantity in the stack, so that the inventory accuracy and reliability of the goods in the stack can be improved.

[0019] Through the method for counting goods in the stack, this embodiment can greatly improve the accuracy of counting the number of cargo positions that are not full stacks, and can eliminate the misjudgment results caused by the AI ​​algorithm, while being compatible with the identification of the whole stack. And when the AI ​​algorithm produces an erroneous result, the error can be automatically detected and identified. In particular, in order to facilitate the warehousing task to take photos of two different sides as much as possible when the stack is not full, the upper system can try to place the non-full stack on the shelf on the different side of the stacker and the pickup platform (as the starting point of the stacker warehousing task) when allocating cargo positions. Since the situation of non-full stacks occurs less frequently, this requirement can usually be met by the upper system.

[0020] In an actual application scenario, in order to realize image acquisition, the hardware related to the inventory of goods in the high-bay warehouse is configured, and the shooting process is pre-set. Specifically, in the basic configuration, two cameras are fixedly installed on the top of the stacker loading platform (the position is higher than the goods), and the angle cannot be adjusted. The two cameras face the front and rear directions of the stacker (i.e. the direction of the cargo position) to adapt to two-way storage and retrieval operations. At the same time, this installation method will not affect the loading and unloading operations of goods. The shooting process is divided into three stages. When shooting before picking up goods, after the stacker is aligned with the cargo position, according to the direction of picking up goods, the front camera is used for forward picking, and the rear camera is used for backward picking. The photo can show the side and top of the goods, which can be used for the classification model to determine the type of goods stacking. When shooting during transportation, the stacker uses a fork to carry the goods to the inside of the stacker. At this time, two opposite cameras are used to shoot the top of the goods in the stacker. Due to the limited shooting angle, each photo can only capture part of the goods on the top when the stack is full, while two cameras can capture the top completely, and the goods are completely off the shelf without obstruction, making it easy to identify. When shooting after placing the goods, after the stacker puts the goods at the target location, if the placing location and the picking location are on different sides of the stacker, after placing the goods, use the camera on the opposite side of the loading platform to shoot the side of the goods to be picked up. This photo will show one side and the top of the goods; if they are on the same side, there is no need to shoot.

[0021] In terms of image quality requirements, it is necessary to ensure that the stacker speed is within the range of shutter speed before shooting, and use a high-speed shutter to avoid motion blur, and use automatic exposure technology to ensure image clarity. It is particularly important to note that when the warehouse uses multi-depth shelves, in order to simplify the algorithm, before picking up and after putting the goods, it can be changed to shooting when the goods pass through the shallowest depth of the goods, so that it is expected that photos of goods at different depths can be taken at the same position.

[0022] In another practical application scenario, combined with Figure 3-Figure 7 When numbering the goods in the stack to be counted according to the preset numbering system, the basic principles of numbering are as follows: (1) All possible cargo locations within each stack have unique numbers; (2) The numbers are incremented starting from 1; (3) The numbering sequence follows the rules from bottom to top, from front to back, and from left to right; (4) The numbering system needs to support position correspondence at different angles. For example Figure 3-Figure 5 Shown are the first layer top view, the second layer top view and the third layer top view.

[0023] The standard stack numbering schemes are: (1) 30 pieces stack (6 layers and 5 rows): Level 1: Positions 1-5; Level 2: Positions 6-10; Level 3: Positions 11-15; Level 4: Positions 16-20; Level 5: Positions 21-25; Level 6: Positions 26-30; Features: 5 positions per layer, easy to identify the layers; (2) 28-piece stack (7 layers, 4 rows): Level 1: Positions 1-4; Level 2: Positions 5-8; Level 3: Positions 9-12; Level 4: Positions 13-16; Level 5: Positions 17-20; Level 6: Positions 21-24; 7th floor: positions 25-28; Features 4 positions per layer, allowing for vertical stacking for higher heights.

[0024] The corresponding rules between numbers and spatial positions are as follows: (1) Front view: Number from left to right, bottom to top; Facilitates position identification when shooting from the front; For example: The first layer of a 30-piece stack is 1, 2, 3, 4, 5 from left to right. Figure 6 The front view shown; (2) Top view: Numbered from front to back, left to right; Adapt to the perspective of top shooting during transportation; For example: The top layer of a 30-piece stack is 26, 27, 28, 29, 30 from front to back and from left to right. Figure 3-Figure 5 As shown; (3) Side view: Keep the same hierarchy as the front view; It is convenient to merge the results of shooting at different angles; For example: Looking from the front to the back of a 30-piece stack, the first row from bottom to top is 1, 6, 11, 16, 21, 26. Figure 7 shown.

[0025] The number application strategy is: During image recognition, the location number and status (available / out of stock / occluded) are output simultaneously. The shooting results at different angles are matched and cross-validated by numbering; The occlusion position can be inferred from the state of the adjacent numbered positions; Used to generate the final cargo quantity statistics results.

[0026] In one embodiment, the step S102 includes: Using the ResNet50 neural network to classify and identify the stack type of the cargo image before picking up the cargo, and obtain the result of the stack type classification and identification; Determine the corresponding stack type according to the result of the stack type classification and identification, and determine the corresponding numbering system according to the stack type; The corresponding position state array is created in combination with the stack type and the numbering system, and the initial state of the position state array is set, and then the spatial coordinate information of each position in the position state array is obtained, so as to establish the position state matrix.

[0027] In the present embodiment, in the stack type identification and initialization process, first utilize the frontal photo taken before picking up to carry out stack type classification.Through the pre-trained classification model, the photo of input can be output as a stack type category, for example 28 or 30 pieces.The backbone network adopted may be ResNet50 or EfficientNetB4, and these models have been pre-trained on the ImageNet data set.In order to adapt to the current task, we only fine-tune the last few layers, and apply data enhancement techniques, such as random rotation, brightness adjustment and contrast adjustment, to improve the generalization ability of the model.This process is crucial for determining the position numbering system.Next, in order to establish the position state matrix, create the position state array of the corresponding number according to the stack type, and set all initial states to "unknown".In addition, record the spatial coordinate information of each position, and establish the adjacent relationship diagram between the positions, for subsequent picking up and management provide basic data support.

[0028] In actual application scenarios, taking the numbering of 30 pieces in a neat stack as an example, the numbering format is as follows: Level 6:

[26]

[27]

[28]

[29]

[30] ← Top floor Level 5:

[21]

[22]

[23]

[24]

[25] Level 4:

[16]

[17]

[18]

[19]

[20] Layer 3:

[11]

[12]

[13]

[14]

[15] Layer 2:

[06]

[07]

[08]

[09]

[10] Layer 1:

[01]

[02]

[03]

[04]

[05] ← Bottom layer Status indicates: Unknown status (U); Available status (A); Out of stock status (B); Occlusion state (C); Then an example of a state matrix is ​​as follows: Layer 6: [26:U] [27:U][28:U] [29:U][30:U] Layer 5: [21:U] [22:U][23:U] [24:U][25:U] Layer 4: [16:U] [17:U][18:U] [19:U][20:U] Layer 3: [11:U] [12:U][13:U] [14:U][15:U] Layer 2: [06:A] [07:A] [08:C] [09:B] [10:B] Layer 1: [01:A] [02:A][03:A] [04:B][05:B] Take the location association table of cargo No. 13 as an example: Location association table = { "13": { "Vertical association": { "Upper": "18", "Lower level": "08" }, "Horizontal association": [11,12,14,15], "Status": "U", "Confidence": 0.95 } }.

[0029] In one embodiment, the step S103 includes: Using the target detection model to perform cargo location detection on each numbered position in the cargo image before pickup; Generate first occlusion information according to the result of cargo positioning detection; wherein the first occlusion information includes complete occlusion, partial occlusion and no occlusion; Based on the first occlusion information, it is determined whether there is cargo at the corresponding numbered position, thereby generating a first confidence score for each numbered position.

[0030] When the cargo positioning detection is performed in this embodiment, the cargo image before picking up is first preprocessed, which may specifically include image enhancement, such as contrast adjustment and noise elimination, perspective correction, such as fixed correction based on the camera installation angle, and size standardization, such as unified image resolution. Then, position detection is performed, and the target detection model is used to locate each visible cargo. The specific model can select target detection models such as YOLOv8 or Faster R-CNN, and the backbone network is CSPDarknet or ResNet-FPN. In the actual training process, the training strategy is to use the COCO pre-training model in the first stage, pre-train with synthetic data in the second stage, and fine-tune with actual scene data in the third stage. The output processing of the model includes: the confidence threshold is 0.75, the NMS threshold is 0.45, and the anchor frame size can be optimized according to the actual size of the cargo. Through the target detection model, the standard position of each layer of cargo can be calculated based on the stacking rule, and the detected cargo can be mapped to the numbered position, and the initial state of each position (cargo / no cargo / occlusion) can be output.

[0031] In one embodiment, the step S104 includes: Acquiring depth information of the image of the cargo in transit using a monocular depth estimation algorithm; Acquire second occlusion information in the image of the cargo in transit according to the depth information; Inferring the spatial relationship of the cargo image in transit in combination with the depth information and the stack type to obtain a result of the spatial relationship inference; wherein the result of the spatial relationship inference includes the occlusion state of the second occlusion information, the rationality of stacking of the cargo in the stack, and the abnormality of stacking; A second confidence score for each numbered position is generated by combining the result of the second occlusion information and the result of the spatial relationship inference.

[0032] When processing the image of the goods in transit, this embodiment first performs image preprocessing and focuses on the distribution of the top layer of goods. Then, the depth information is obtained through depth estimation, and the position status is updated based on the height information and depth information. When updating the position status, the position status of the unobstructed area is processed first, and the recognition results before picking up the goods are compared. Then, the position status matrix is ​​updated, and the confidence of the state change is recorded. Here, spatial relationship inference can also be performed based on depth information, that is, spatial relationship analysis is combined with stacking rules, and the state of the obstructed position is processed. The rationality of the stacking can also be verified and abnormal stacking can be detected.

[0033] In actual application scenarios, depth information can be obtained based on a monocular depth estimation algorithm, where the main model can use the Depth Anything (based on the ViT architecture) model, and the alternative model can use the MiDaS v3.1 (suitable for indoor scenes) model. The model configuration includes: input resolution of 384×384 or 512×512, backbone network of ViT-L (DepthAnything) or DPT-Large (MiDaS), and pre-training using a large-scale mixed data set. Model inference optimization can use half-precision (FP16) acceleration, batch size of 1 (real-time processing), and support TensorRT acceleration. In addition, post-processing of cargo images can include depth map normalization, actual distance conversion based on camera parameters, and depth discontinuity detection.

[0034] Furthermore, the depth map is converted into a 3D point cloud for (relative) height comparison and occlusion judgment. Specifically, basic parameter configuration is first performed, including the camera intrinsic parameter settings: Horizontal focal length (focal_length_x): If the camera intrinsic parameters cannot be obtained, the default value of 470.4 is used; Vertical focal length (focal_length_y): If the camera intrinsic parameters cannot be obtained, the default value of 470.4 is used; Image center point: If the camera intrinsic parameters cannot be obtained, take the position of half the image size (the exact center).

[0035] Also includes depth estimation configuration: Maximum depth value: configurable based on the camera hardware capability, with a default value of 20 meters.

[0036] Secondly, the processing flow includes: Depth map generation; Process the image using a depth estimation model; Generate a depth map that is proportional to the original image; Get the depth value based on the depth map, which represents the distance from each pixel to the camera.

[0037] Again, calculate the point cloud coordinates: (1) Establish the image plane grid coordinate system: Grid coordinate calculation: x_grid = (pixel_x - width / 2) / focal_length_x; y_grid = (pixel_y - height / 2) / focal_length_y; (2) Three-dimensional coordinate transformation: Space coordinate calculation: X = x_grid * depth_value; Y = y_grid * depth_value; Z = depth_value; Thus, the point cloud coordinates P(X, Y, Z) are obtained. After obtaining the transformed three-dimensional point cloud P(X, Y, Z), plane detection can be performed to find an approximate horizontal plane.

[0038] The plane detection method is: (1) Principal Component Analysis (PCA): Calculate the covariance matrix of the point cloud; Extract three main directions (eigenvectors); Among them, the direction corresponding to the maximum eigenvalue in the eigenvector usually represents the vertical direction, because the cargo stacking changes the most in the vertical direction.

[0039] The core steps of PCA are as follows: For the point cloud P(x,y,z), calculate the 3x3 covariance matrix C: C[i,j] = E[(P[i]-μ[i])(P[j]- μ[j])], where i,j∈{x,y,z} and μ is the mean.

[0040] In addition, by solving the characteristic equation, three eigenvalues ​​(λ1 ≥ λ2 ≥ λ3) can be obtained, and the corresponding eigenvectors (v1, v2, v3) can be calculated, where v1 usually corresponds to the vertical direction (the direction of maximum change), and v2 and v3 form an approximate horizontal plane. Then the direction is verified and optimized. First, the gravity direction is aligned. Assuming that the maximum eigenvalue direction (v1) is close to the vertical direction, the angle between v1 and the estimated gravity direction is calculated, and then the horizontal plane direction is optimized. v2 and v3 should roughly correspond to the front and back and left and right directions of the shelf. The angles between these directions and the known shelf directions are calculated, and rotation adjustments are made when necessary. The approximate coordinate system is constructed using the three directions finally obtained.

[0041] (2) Prior Constrained Optimization: Taking advantage of the characteristics of regular stacking of goods, the vertical direction should be roughly parallel to the direction of gravity, and the two directions on the horizontal plane should be roughly aligned with the front and back and left and right directions of the shelf.

[0042] The relative height is estimated based on the constructed approximate coordinate system, and the point cloud is projected onto the obtained horizontal plane to obtain the approximate height. The relative height is obtained based on the method of missing goods and height jump (the difference between adjacent areas is greater than a certain threshold).

[0043] In one embodiment, when processing the cargo image after the goods are placed, the processing result is mainly used for verification, and the same processing flow as the cargo image before the goods are picked up can be used. However, since the cargo image before the goods are picked up has determined the stack type after being identified by the classification model, the cargo image after the goods are placed can only be positioned and detected by the target detection algorithm to focus on verifying the position of the uncertain state and handle the newly generated occlusion situation, thereby generating the final position state. In actual application scenarios, the depth estimation algorithm can also be used to extract the depth information of the cargo image before the goods are picked up and the cargo image after the goods are placed, so as to perform a more accurate inventory of the goods based on the depth information.

[0044] In one embodiment, the step S106 includes: Performing geometric occlusion analysis in combination with the first occlusion information, the second occlusion information, and the third occlusion information to obtain a first occlusion analysis result; wherein the geometric occlusion analysis includes: calculating the line of sight occlusion range based on a fixed angle of the camera, determining foreground occlusion using a depth estimation result, establishing an occlusion relationship diagram between positions, and marking fully occluded and partially occluded areas; Performing occlusion inference analysis in combination with the first occlusion information, the second occlusion information and the third occlusion information to obtain a second occlusion analysis result; wherein the occlusion inference analysis includes: lower layer support rules, boundary constraints, layer integrity and inferring unknown positions according to known position states; Occlusion cross-validation is performed based on the first occlusion analysis result and the second occlusion analysis result, and target occlusion information is generated according to the result of the occlusion cross-validation.

[0045] When generating the final goods inventory result, this embodiment combines the first occlusion information, the second occlusion information and the third occlusion information for occlusion judgment. Specifically, on the one hand, it is geometric occlusion analysis, including calculating the line of sight occlusion range based on the camera fixed angle, judging the foreground occlusion by using the depth estimation result, establishing the occlusion relationship diagram between positions, and marking the fully occluded and partially occluded areas; on the other hand, it is the inference of the stacking rules, including the lower layer support rule, that is, if there is goods on the upper layer, there must be goods on the lower layer, the boundary constraint, that is, it cannot exceed the standard stacking range, the level integrity, that is, the goods on the same layer are usually placed continuously, and the unknown position is inferred according to the known position state. Based on the occlusion judgment, the occlusion degree assessment and confirmation can be realized, including complete occlusion-the target position cannot be directly observed, partial occlusion-only part of the area can be observed, and no occlusion. Through the occlusion degree assessment, the visibility percentage of each position can be calculated, so that the confidence of the position state can be updated. In the actual scene, the depth information can be extracted from the cargo images before picking up, during transportation and after placing the cargo, and then the partial visible information at different angles is used to cross-validate the inference results and calculate the confidence of the inference results. Here, the visibility is defined as: 0, indicating complete occlusion, and no target detection result can be obtained; 1, indicating that the target detection is recognized. The confidence source includes the confidence output by the target detection model, but the confidence is not calculated for the occlusion state (visibility 0), and the highest confidence value is taken for multiple detections. In addition, when the visibility is 1, the target detection result is used, and when the visibility is 0, the detection result of the subsequent visible moment is waited.

[0046] Furthermore, the occlusion judgment is verified. First, physical constraint verification is performed to ensure that the inference results comply with the gravity support principle, do not violate the standard pile shape rules, and ensure their physical feasibility, and mark suspicious inference results. Then, confidence evaluation is performed. According to the different situations of direct observation position, partial occlusion position and complete occlusion position, the detection confidence and visibility are used to adjust the confidence and inference confidence, and different confidence thresholds are set.

[0047] In the exception handling stage, direct observation results are given priority when handling inference conflicts, and the status of related locations is re-evaluated and marked for manual review when necessary. For special cases, such as non-standard stacking patterns, occlusion of multi-deep shelves, and uncertain stack type identification, appropriate processing is carried out and exception reports and suggestions are generated.

[0048] Finally, the output results include status updates and quality assessments. Status updates involve updating the position state matrix, recording the inference basis and confidence, marking locations that require manual confirmation, and generating final quantity statistics. Quality assessments include evaluating the reliability of overall recognition, calculating confidence in key locations, generating detailed logs of the inference process, and providing optimization suggestions.

[0049] In one embodiment, the step S106 further includes: Assigning weights to the first confidence score, the second confidence score, and the third confidence score respectively according to a preset second weight, and obtaining the target confidence score by weighted summation; The target occlusion information and the target confidence score are combined to determine whether there is any cargo at the corresponding numbered position, thereby obtaining an inventory result of the cargo quantity in the stack.

[0050] Specifically, the step of combining the target occlusion information and the target confidence score to determine whether there is any cargo at the corresponding numbered position, thereby obtaining an inventory result of the cargo quantity in the stack, includes: When the numbered position meets the first determination condition, it is determined that there is a product; wherein the first determination condition is: the depth information corresponding to the numbered position shows that there is an object, the target confidence score is greater than a first preset score, and the overlap between the detection frame corresponding to the numbered position and the preset standard position is greater than a first preset threshold; When the numbered position meets the second determination condition, it is determined that there is no stock; wherein the second determination condition is: the depth information corresponding to the numbered position does not have a detection frame within the preset standard position, the depth information shows no protrusion, and the target occlusion information of the numbered position is not blocked; When the number position meets the third determination condition, it is determined to be blocked; wherein the third determination condition is: the depth information corresponding to the number position shows that there is blockage, there is a detection frame of other goods in the preset standard position, or it is inferred that the number position is blocked based on the result of spatial relationship inference; When the numbered position meets the fourth determination condition, it is determined to be uncertain; wherein the fourth determination condition is: the overlap between the detection frame corresponding to the numbered position and the preset standard position is within a second preset threshold, the target confidence score of the numbered position is within a second preset score, or the target occlusion information of the numbered position is partially occluded; The number of goods in the stack is counted based on the determination result of each number position, thereby obtaining the inventory result of the number of goods in the stack.

[0051] In this embodiment, in the process of integrating the results, the recognition results of the three shots are combined, and a confidence-weighted voting mechanism is used to handle inconsistent results, and finally a detailed position status table is generated, thereby realizing the inventory of goods. In addition, anomaly detection can be performed based on depth information and other content, check the state that violates the stacking rules, identify physically impossible stacking situations, and mark the positions that require manual confirmation, so as to generate an abnormality report. The corresponding output results include the final state of each numbered position (goods / no goods), the confidence score of each state, the mark and description of the abnormal position, and a list of locations recommended for review.

[0052] In practical applications, we first define the basic states of each position: A: Confirmed to be in stock (confidence is greater than the threshold and directly observed); B: Confirmed out of stock (confidence is greater than the threshold and the empty position is directly observed); C: occluded (cannot be directly observed); U: Uncertain (confidence is below the threshold).

[0053] When processing the results of a single shot, different strategies are adopted to ensure accuracy and efficiency. First, when shooting before picking up the goods, the confidence of the target detection model is used as the base score, and the detection results of the edge and overlapping areas are reduced in weight, while the status and confidence of each position are recorded. When shooting during transportation, the results of this stage are given the highest weight because the goods are not obstructed. In particular, the detection results of the top visible area are given the highest confidence, and special attention is paid to the positions that are inconsistent with the previous results to ensure that the status of the goods is accurately monitored throughout the transportation process. Finally, when shooting after releasing the goods, the same processing method as before picking up the goods is adopted, focusing on verifying the uncertain positions in the previous two results, and giving appropriate weight to the new perspective information, which helps to fully understand the status changes of the goods.

[0054] When processing multiple shooting results, the confidence weighting method is used to merge these results. For example, the weight of the photos taken during transportation is 0.5, the weight of the photos taken before picking up the goods is 0.3, and the weight of the photos after releasing the goods is 0.2. The final confidence is obtained by multiplying the confidence of each shot by the corresponding weight and summing them. In terms of state determination, the following rules are followed: if there is a conflict between the A and B states in any two shots, the position is marked as abnormal; if a position obtains the A state in any shot and there is no conflict, then the position is finally determined to be in the A state; if a position obtains the B state in any shot and there is no conflict, then the position is finally determined to be in the B state; if a position only obtains the C state, it needs to enter the occlusion processing process; if the states are inconsistent but the confidence of all states is low, the position is marked as the U state.

[0055] When dealing with conflicts, the first thing to do is to detect level conflicts, which involve the contradictions between different shooting results at the same location. Solutions to this problem include checking the confidence of each detection and referring to the status of adjacent locations. If the problem still cannot be solved, it needs to be marked for manual review. Secondly, the stacking rule conflicts also need attention, which include violations of physical stacking rules, such as no goods on the lower layer but goods on the upper layer, or the detection results exceed the standard stacking range. For these situations, the system should automatically adjust to a reasonable state, or mark them as abnormal for further processing.

[0056] In practical applications, the basic confidence can be defined first, which includes the probability value output by the target detection model, the clarity of the target in the image, and whether there is partial occlusion. Subsequently, several adjustment factors are considered, including perspective adaptation (adjusted according to the shooting angle), distance attenuation (the farther the target is from the camera, the lower the confidence), and occlusion effect (the higher the degree of occlusion, the lower the confidence), to obtain the final confidence. The final confidence is calculated by multiplying the basic confidence by the product of all adjustment factors. In order to ensure the reliability of the results, a minimum confidence threshold can be set, such as 0.6, and any result below this threshold will be marked as U state to indicate uncertainty. Further, when outputting the final inventory results, the final state (A / B / C / U) of each numbered position, the comprehensive confidence of each state, and the abnormal mark and reason description can be output. At the same time, it is also possible to recommend a review of the positions with low confidence, detection conflicts and / or violations of the stacking rules, and generate a manual review list.

[0057] In a specific embodiment, when determining the position state, the following determination strategy may be specifically adopted: (1) In-stock (A) determination conditions (all of the following conditions must be met): AND ( The overlap between the detection frame and the standard position is > 70% Detection confidence>0.75 Depth estimation shows that there is an object at this location ) (2) Out of stock (B) determination conditions (all of the following conditions must be met): AND ( There is no detection frame within the field of view of this location The depth map shows no obvious protrusions in this area The location is fully visible (no obstructions) ) (3) Occlusion (C) determination conditions (meet any of the following conditions): OR ( Detecting an occlusion in front through the depth map The expected field of view of this location is occupied by other cargo detection frames According to the stacking rules, it is inferred that this position should be within the shielding area. ) (4) Uncertain (U) judgment conditions (meet any of the following conditions): OR ( The overlap between the detection frame and the position is between 30% and 70%. The detection confidence is between 0.6-0.75 Partially obscured but still partially visible Does not meet any of the above A / B / C status criteria ) When the conditions of multiple states are met at the same time, they are processed according to the following priority: (1) Occlusion (C): If the occlusion condition is met, it is prioritized as occlusion; (2) Available (A): If all available conditions are met and the product is not considered blocked; (3) Out of stock (B): If all out of stock conditions are met and the item is not judged as blocked or in stock; (4) Uncertain (U): All other situations.

[0058] For the three types of photos that are determined to be in an uncertain state, the following processing methods are available: (1) Trigger conditions. Specifically, the position is in an uncertain (U) state in all three shots or is still in an uncertain state after the results are merged, and the determined state cannot be inferred through occlusion processing; (2) Processing strategy. This includes automatic processing attempts, such as checking the status of adjacent locations, applying stacking rule constraints, and analyzing the common status of the cargo location in historical data. If the automatic processing fails, the location can be marked as "pending manual review" and marked as the highest priority in the exception report. At the same time, data recording is performed, including recording the specific reasons for the uncertainty and recording all relevant detection data and images for subsequent algorithm optimization and model improvement.

[0059] The following is an example of a complete operation of 30 pieces of stack type A to illustrate the processing process: (1) Initial scenario: 24 items are stored in a certain cargo space (the sixth floor is vacant) for picking; (2) Photographing results before picking up the goods. Frontal observation results: Layers 1-4 (positions 1-20): visibility 1, target detection confidence 0.85-0.92; Layer 5 (position 21-25): visibility 0, completely blocked by the equipment in front; Layer 6 (position 26-30): visibility 1, target detection confirms no goods, confidence 0.88-0.90.

[0060] (2) Shooting results during transportation. Shot by the front camera: Layer 5: visibility 1, target detection confidence 0.95; Level 6: Visibility 1, confirmed out of stock, confidence 0.96.

[0061] Rear camera shooting: Layer 5: visibility 1, target detection confidence 0.94; Level 6: Visibility 1, confirmed out of stock, confidence 0.95.

[0062] (3) Shooting results after releasing the goods. Side observation: Layers 1-4: visibility 1, target detection confidence 0.82-0.89; Layer 5: visibility 1, target detection confidence 0.91; Level 6: Visibility 1, confirmed out of stock, confidence 0.92.

[0063] (4) Result merging process. Take layer 5 as an example: Before picking up the goods: visibility 0, no confidence; During transport: visibility 1, confidence 0.95 (front camera) and 0.94 (rear camera); After releasing the goods: visibility 1, confidence 0.91; Then the combined calculation is: Status determination: Goods are detected when visible and are determined to be in stock; Final confidence level: Take the highest confidence level of 0.95.

[0064] For the occlusion processing, take the occlusion of layer 5 as an example: (1) Occlusion analysis. The state of the fifth layer when taking the photo before picking up the goods: Occlusion source: stacker structure; Occlusion judgment: Positions 21-25: visibility is 0; completely blocked, no target detection results can be obtained; (2) Inference process. (a) Geometric relationship analysis: It is known that positions 16-20 on the 4th floor are all stocked (visibility 1, confidence>0.85); it is known that positions 26-30 on the 6th floor are all empty (visibility 1, confidence>0.88); it is necessary to wait for direct observation results by taking photos during transportation; (b) Observation during transport: Visibility becomes 1; target detection confidence is 0.94-0.95; direct observation confirms the presence of goods without inference; (3) Verification and confirmation.

[0065] The cargo was directly observed by both cameras during transportation, and directly observed again after the cargo was released, and all direct observation results were consistent; (4) Result output, Status: Confirmed to be in stock; Confidence level: 0.95 (the highest confidence level is used); Visibility record: [0,1,1] (before picking up, during transportation, after releasing).

[0066] Figure 2A schematic block diagram of a device 200 for counting the number of goods based on cross-validation provided in an embodiment of the present invention, the device 200 includes: The number collection unit 201 is used to pre-number the goods in the stack to be counted according to a preset numbering system, and collect images of the goods in the stack; wherein the images of the goods in the stack include images of the goods before picking up, images of the goods in transit, and images of the goods after releasing; A classification establishment unit 202 is used to classify and identify the stack type of the goods image before picking up the goods using a classification model, and determine the corresponding numbering system according to the result of the stack type classification and identification, and then establish a position state matrix in combination with the numbering system; The first generating unit 203 is used to perform a first positioning detection on the cargo image before pickup using the target detection model, and generate first occlusion information and a first confidence score for each numbered position in the cargo image before pickup according to the result of the first positioning detection; The second generating unit 204 is used to obtain the depth information of the image of the goods in transit by using a depth estimation algorithm, and to perform a first update on the position state matrix according to the depth information, and to perform a spatial relationship inference on the goods in the stack according to the depth information, and then to generate second occlusion information and a second confidence score for each numbered position in the image of the goods in transit by combining the position state matrix after the first update and the result of the spatial relationship inference; The third generating unit 205 is used to perform a second positioning detection on the cargo image after the cargo is released by using the target detection model, and perform a second updating on the position state matrix according to the result of the second positioning detection, and then generate third occlusion information and a third confidence score for each numbered position in the cargo image after the cargo is released according to the position state matrix after the second updating; The verification and inventory unit 206 is used to cross-validate the occlusion information in combination with the first occlusion information, the second occlusion information and the third occlusion information, and generate target occlusion information, and to cross-validate the first confidence score, the second confidence score and the third confidence score, and generate a target confidence score, and then generate an inventory result of the quantity of goods in the stack based on the target confidence score and the target occlusion information.

[0067] In one embodiment, the classification establishment unit 202 includes: A classification and recognition unit, used for classifying and recognizing the type of the goods before picking up the goods by using a ResNet50 neural network to obtain a result of the classification and recognition of the type of the goods; An identification and determination unit, used to determine the corresponding stack type according to the result of the stack type classification and identification, and determine the corresponding numbering system according to the stack type; The matrix establishment unit is used to create a corresponding position state array in combination with the stack type and numbering system, set the initial state of the position state array, and then obtain the spatial coordinate information of each position in the position state array to establish the position state matrix.

[0068] In one embodiment, the first generating unit 203 includes: A positioning detection unit, used for performing cargo positioning detection on each numbered position in the cargo image before picking up the cargo using a target detection model; A first occlusion generating unit, configured to generate first occlusion information according to a result of cargo positioning detection; wherein the first occlusion information includes complete occlusion, partial occlusion and no occlusion; The first score generating unit is used to determine whether there is a product at the corresponding numbered position based on the first occlusion information, so as to generate a first confidence score for each numbered position.

[0069] In one embodiment, the second generating unit 204 includes: An information acquisition unit, configured to acquire depth information of the image of the cargo being transported using a monocular depth estimation algorithm; A second occlusion generating unit, configured to obtain second occlusion information in the image of the cargo in transit according to the depth information; A spatial inference unit is used to perform spatial relationship inference on the image of the cargo in transit in combination with the depth information and the stack type to obtain a result of spatial relationship inference; wherein the result of spatial relationship inference includes an occlusion state of the second occlusion information, a stacking rationality of the cargo in the stack, and a stacking abnormality; The second score generating unit is used to generate a second confidence score for each numbered position by combining the result of the second occlusion information and the result of the spatial relationship inference.

[0070] In one embodiment, the verification inventory unit 206 includes: A geometric occlusion analysis unit, configured to perform geometric occlusion analysis on the first occlusion information, the second occlusion information and the third occlusion information to obtain a first occlusion analysis result; wherein the geometric occlusion analysis includes: calculating the line of sight occlusion range based on a fixed camera angle, determining foreground occlusion using a depth estimation result, establishing an occlusion relationship diagram between positions, and marking fully occluded and partially occluded areas; An occlusion inference analysis unit is used to perform an occlusion inference analysis on the basis of the first occlusion information, the second occlusion information and the third occlusion information to obtain a second occlusion analysis result; wherein the occlusion inference analysis includes: lower layer support rules, boundary constraints, layer integrity and inference of unknown positions according to known position states; The occlusion cross-validation unit is used to perform occlusion cross-validation based on the first occlusion analysis result and the second occlusion analysis result, and generate target occlusion information according to the result of the occlusion cross-validation.

[0071] In one embodiment, the verification inventory unit 206 further includes: a weighting unit, configured to respectively weight the first confidence score, the second confidence score, and the third confidence score according to a preset second weight, and obtain the target confidence score by weighted summation; The goods-in-stock determination unit is used to determine whether there is goods at the corresponding numbered position based on the target occlusion information and the target confidence score, so as to obtain the inventory result of the quantity of goods in the stack.

[0072] In one embodiment, the in-stock determination unit includes: A first determination unit is configured to determine that there is stock when the numbered position satisfies a first determination condition; wherein the first determination condition is: the depth information corresponding to the numbered position shows that there is an object, the target confidence score is greater than a first preset score, and the overlap between the detection frame corresponding to the numbered position and the preset standard position is greater than a first preset threshold; A second determination unit is configured to determine that there is no stock when the numbered position meets a second determination condition; wherein the second determination condition is that: the depth information corresponding to the numbered position does not have a detection frame within a preset standard position, the depth information shows no protrusion, and the target occlusion information of the numbered position is not blocked; A third determination unit is configured to determine that the number position is blocked when the number position meets a third determination condition; wherein the third determination condition is: the depth information corresponding to the number position shows that there is blockage, there is a detection frame of other goods in the preset standard position, or it is inferred that the number position is blocked based on the result of spatial relationship inference; A fourth determination unit is used to determine that the numbered position is uncertain when the numbered position meets a fourth determination condition; wherein the fourth determination condition is: the overlap between the detection frame corresponding to the numbered position and the preset standard position is within a second preset threshold, the target confidence score of the numbered position is within a second preset score, or the target occlusion information of the numbered position is partial occlusion; The quantity counting unit is used to count the quantity of goods in the stack based on the determination result of each number position, so as to obtain the inventory result of the quantity of goods in the stack.

[0073] Since the embodiments of the apparatus part correspond to the embodiments of the method part, please refer to the description of the embodiments of the method part for the embodiments of the apparatus part, which will not be repeated here.

[0074] The embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed, the steps provided in the above embodiment can be implemented. The storage medium may include: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program codes.

[0075] The embodiment of the present invention also provides a computer device, which may include a memory and a processor, wherein a computer program is stored in the memory, and when the processor calls the computer program in the memory, the steps provided in the above embodiment may be implemented. Of course, the computer device may also include various network interfaces, power supplies and other components.

[0076] The various embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same and similar parts between the various embodiments can be referred to each other. For the system disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of this application, several improvements and modifications can be made to the present application, and these improvements and modifications also fall within the scope of protection of the claims of this application.

[0077] It should also be noted that, in this specification, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprises", "comprising" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the statement "comprising a ..." does not exclude the presence of other identical elements in the process, method, article or device including the element.

Claims

1. A method for counting the number of goods based on cross-validation, characterized in that: include: The goods in the stack to be counted are numbered in advance according to a preset numbering system, and images of the goods in the stack are collected; wherein the images of the goods in the stack include images of the goods before picking up the goods, images of the goods in transit, and images of the goods after releasing the goods; Using the classification model to classify and identify the stack type of the cargo image before picking up the cargo, and determining the corresponding numbering system according to the result of the stack type classification and identification, and then establishing a position state matrix in combination with the numbering system; Performing a first positioning detection on the cargo image before pickup using a target detection model, and generating first occlusion information and a first confidence score for each numbered position in the cargo image before pickup according to a result of the first positioning detection; Using a depth estimation algorithm to obtain depth information of the image of the goods in transit, and performing a first update on the position state matrix according to the depth information, and performing spatial relationship inference on the goods in the stack according to the depth information, and then combining the position state matrix after the first update and the result of the spatial relationship inference to generate second occlusion information and a second confidence score for each numbered position in the image of the goods in transit; Performing a second positioning detection on the cargo image after cargo release by using the target detection model, and performing a second updating on the position state matrix according to the result of the second positioning detection, and then generating third occlusion information and a third confidence score for each numbered position in the cargo image after cargo release according to the second updated position state matrix; The first occlusion information, the second occlusion information and the third occlusion information are combined to perform cross-validation of the occlusion information and generate target occlusion information, and the first confidence score, the second confidence score and the third confidence score are combined to perform cross-validation and generate a target confidence score, and then an inventory result of the quantity of goods in the stack is generated based on the target confidence score and the target occlusion information.

2. The method for counting the number of goods based on cross-validation according to claim 1, characterized in that: The method of using the classification model to classify and identify the stack type of the cargo image before picking up the cargo, determining the corresponding numbering system according to the result of the stack type classification and identification, and establishing a position state matrix according to the result of the stack type classification and identification, includes: Using the ResNet50 neural network to classify and identify the stack type of the cargo image before picking up the cargo, and obtain the result of the stack type classification and identification; Determine the corresponding stack type according to the result of the stack type classification and identification, and determine the corresponding numbering system according to the stack type; The corresponding position state array is created in combination with the stack type and the numbering system, and the initial state of the position state array is set, and then the spatial coordinate information of each position in the position state array is obtained, so as to establish the position state matrix.

3. The method for counting the number of goods based on cross-validation according to claim 1, characterized in that: The method of performing a first positioning detection on the cargo image before picking up the cargo by using the target detection model, and generating first occlusion information and a first confidence score for each numbered position in the cargo image before picking up the cargo according to the result of the first positioning detection, includes: Using the target detection model to perform cargo location detection on each numbered position in the cargo image before pickup; Generate first occlusion information according to the result of cargo positioning detection; wherein the first occlusion information includes complete occlusion, partial occlusion and no occlusion; Based on the first occlusion information, it is determined whether there is cargo at the corresponding numbered position, thereby generating a first confidence score for each numbered position.

4. The method for counting the number of goods based on cross-validation according to claim 1, characterized in that: The method of obtaining depth information of the image of the cargo in transit by using a depth estimation algorithm, updating the position state matrix for the first time according to the depth information, and inferring the spatial relationship of the cargo in the stack according to the depth information, and then combining the position state matrix after the first update and the result of the spatial relationship inference to generate second occlusion information and a second confidence score for each numbered position in the image of the cargo in transit, comprises: Acquiring depth information of the image of the cargo in transit using a monocular depth estimation algorithm; Acquire second occlusion information in the image of the cargo in transit according to the depth information; Inferring the spatial relationship of the cargo image in transit in combination with the depth information and the stack type to obtain a result of the spatial relationship inference; wherein the result of the spatial relationship inference includes the occlusion state of the second occlusion information, the rationality of stacking of the cargo in the stack, and the abnormality of stacking; A second confidence score for each numbered position is generated by combining the result of the second occlusion information and the result of the spatial relationship inference.

5. The method for counting the number of goods based on cross-validation according to claim 1, characterized in that: The combining the first occlusion information, the second occlusion information and the third occlusion information to perform occlusion information cross-verification and generate target occlusion information includes: Performing geometric occlusion analysis in combination with the first occlusion information, the second occlusion information, and the third occlusion information to obtain a first occlusion analysis result; wherein the geometric occlusion analysis includes: calculating the line of sight occlusion range based on a fixed angle of the camera, determining foreground occlusion using a depth estimation result, establishing an occlusion relationship diagram between positions, and marking fully occluded and partially occluded areas; Performing occlusion inference analysis in combination with the first occlusion information, the second occlusion information and the third occlusion information to obtain a second occlusion analysis result; wherein the occlusion inference analysis includes: lower layer support rules, boundary constraints, layer integrity and inferring unknown positions according to known position states; Occlusion cross-validation is performed based on the first occlusion analysis result and the second occlusion analysis result, and target occlusion information is generated according to the result of the occlusion cross-validation.

6. The method for counting the number of goods based on cross-validation according to claim 5, characterized in that: The method further comprises: performing cross-validation of the occlusion information by combining the first occlusion information, the second occlusion information and the third occlusion information, and generating target occlusion information; performing cross-validation by combining the first confidence score, the second confidence score and the third confidence score, and generating a target confidence score; and then generating an inventory result of the quantity of goods in the stack according to the target confidence score and the target occlusion information. Assigning weights to the first confidence score, the second confidence score, and the third confidence score respectively according to a preset second weight, and obtaining the target confidence score by weighted summation; The target occlusion information and the target confidence score are combined to determine whether there is any cargo at the corresponding numbered position, thereby obtaining an inventory result of the cargo quantity in the stack.

7. The method for counting the number of goods based on cross-validation according to claim 6, characterized in that: The step of combining the target occlusion information and the target confidence score to determine whether there is any cargo at the corresponding numbered position, thereby obtaining an inventory result of the cargo quantity in the stack, includes: When the numbered position meets the first determination condition, it is determined that there is a product; wherein the first determination condition is: the depth information corresponding to the numbered position shows that there is an object, the target confidence score is greater than a first preset score, and the overlap between the detection frame corresponding to the numbered position and the preset standard position is greater than a first preset threshold; When the numbered position meets the second determination condition, it is determined that there is no stock; wherein the second determination condition is: the depth information corresponding to the numbered position does not have a detection frame within the preset standard position, the depth information shows no protrusion, and the target occlusion information of the numbered position is not blocked; When the number position meets the third determination condition, it is determined to be blocked; wherein the third determination condition is: the depth information corresponding to the number position shows that there is blockage, there is a detection frame of other goods in the preset standard position, or it is inferred that the number position is blocked based on the result of spatial relationship inference; When the numbered position meets the fourth determination condition, it is determined to be uncertain; wherein the fourth determination condition is: the overlap between the detection frame corresponding to the numbered position and the preset standard position is within a second preset threshold, the target confidence score of the numbered position is within a second preset score, or the target occlusion information of the numbered position is partially occluded; The number of goods in the stack is counted based on the determination result of each number position, thereby obtaining the inventory result of the number of goods in the stack.

8. A device for counting the number of goods based on cross-validation, characterized in that: include: A number collection unit is used to pre-number the goods in the stack to be counted according to a preset numbering system, and collect images of the goods in the stack; wherein the images of the goods in the stack include images of the goods before picking up, images of the goods in transit, and images of the goods after releasing; A classification establishment unit, used for classifying and identifying the type of goods before picking up the goods by using a classification model, and determining a corresponding numbering system according to the result of the classification and identification of the type of goods, and then establishing a position state matrix in combination with the numbering system; a first generating unit, configured to perform a first positioning detection on the cargo image before pickup using a target detection model, and generate first occlusion information and a first confidence score for each numbered position in the cargo image before pickup according to a result of the first positioning detection; a second generating unit, configured to obtain depth information of the image of the goods in transit by using a depth estimation algorithm, and to perform a first update on the position state matrix according to the depth information, and to perform a spatial relationship inference on the goods in the stack according to the depth information, and then to generate second occlusion information and a second confidence score for each numbered position in the image of the goods in transit by combining the position state matrix after the first update and the result of the spatial relationship inference; a third generating unit, configured to perform a second positioning detection on the cargo image after cargo release by using the target detection model, and perform a second updating on the position state matrix according to the result of the second positioning detection, and then generate third occlusion information and a third confidence score for each numbered position in the cargo image after cargo release according to the position state matrix after the second updating; A verification and inventory unit is used to cross-validate the occlusion information in combination with the first occlusion information, the second occlusion information and the third occlusion information, and generate target occlusion information, and to cross-validate the first confidence score, the second confidence score and the third confidence score, and generate a target confidence score, and then generate an inventory result of the quantity of goods in the stack based on the target confidence score and the target occlusion information.

9. A computer device, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the cross-validation-based cargo quantity inventory counting method as described in any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the cross-validation-based cargo quantity inventory counting method according to any one of claims 1 to 7 is implemented.

Images