Cargo management method and device based on depth estimation algorithm, and related components

Abstract

The invention discloses a cargo management method and device based on a depth estimation algorithm, and a related assembly. The method comprises the steps: collecting a cargo image of a to-be-estimated cargo; obtaining depth information of the cargo image based on a depth estimation algorithm; performing cargo area analysis on the cargo according to the depth information to obtain area information; carrying out shielding analysis on the goods in combination with the depth information and the area information to obtain goods shielding information, and carrying out consistency analysis on the goods in combination with the depth information and the area information to obtain goods consistency information; and carrying out cargo quantity estimation based on the cargo shielding information and the cargo consistency information, and outputting an estimation result. According to the embodiment of the invention, the depth information of the cargo image is extracted through the depth estimation algorithm, and the cargo area can be accurately analyzed, so that the cargo quantity can be effectively estimated, and a high-accuracy estimation result is output.

Description

Technical Field

[0001] The present invention relates to the technical field of warehouse management, and in particular to a cargo management method, device and related components based on a depth estimation algorithm. Background Art

[0002] In the automated management of high-bay warehouses, fast and accurate inventory counting of goods is a key link in improving logistics efficiency. Traditional inventory counting methods rely on manual operations, which are time-consuming and prone to errors, especially in environments such as high-bay warehouses with limited space and numerous cargo spaces. With the development of technology, using stackers to take side photos of cargo spaces during movement, and then using image processing technology to infer the type and quantity of cargo, has become an innovative automatic inventory counting method. This method can significantly reduce the need for manual inventory counting and improve the efficiency and accuracy of inventory counting.

[0003] However, traditional 2D image processing technology has limitations when dealing with such problems. First, 2D images lack depth information, which makes it difficult for the algorithm to accurately perceive the spatial position relationship of the goods, especially when dealing with partially full pallets, the recognition error rate is high. Secondly, 2D object detection is weak in dealing with occluded scenes and is prone to missed detections and false detections, especially in high-bay warehouses where there may be complex occlusion relationships between goods. In addition, the spatial reasoning capabilities of 2D image processing technology are limited, physical constraints cannot be verified, and there is a lack of three-dimensional consistency verification, which further limits its application in automatic inventory counting in high-bay warehouses.

[0004] To solve these problems, researchers have tried a variety of methods. For example, some studies have tried to improve the accuracy of recognition by improving image classification algorithms, but this method is still limited in its effect on the recognition of partially full pallets. In addition, some studies have tried to combine deep learning technology and use deep learning models such as convolutional neural networks (CNN) to improve the ability to recognize goods, but these methods still face challenges when dealing with occlusion and spatial position relationships.

[0005] In summary, although the application prospects of automatic inventory counting technology in high-bay warehouses are broad, the current technology still faces many challenges, especially in dealing with complex scenarios and improving recognition accuracy. Therefore, how to ensure the accuracy of cargo recognition in different scenarios is a problem that technicians in this field need to solve. Summary of the invention

[0006] The embodiments of the present invention provide a cargo management method, device, computer equipment and storage medium based on a depth estimation algorithm, aiming to improve the accuracy of cargo quantity recognition.

[0007] In a first aspect, an embodiment of the present invention provides a cargo management method based on a depth estimation algorithm, comprising: Collecting cargo images of cargo to be estimated; Acquiring depth information of the cargo image based on a depth estimation algorithm; Performing cargo area analysis on the cargo according to the depth information to obtain area information; performing an occlusion analysis on the goods in combination with the depth information and the area information to obtain goods occlusion information, and performing a consistency analysis on the goods in combination with the depth information and the area information to obtain goods consistency information; Cargo quantity estimation is performed based on the cargo shielding information and cargo consistency information, and an estimation result is output.

[0008] In a second aspect, an embodiment of the present invention provides a cargo management device based on a depth estimation algorithm, comprising: An image acquisition unit, used for acquiring images of goods to be estimated; A depth estimation unit, configured to obtain depth information of the cargo image based on a depth estimation algorithm; A first analysis unit, configured to perform cargo area analysis on the cargo according to the depth information to obtain area information; A second analysis unit is used to perform an occlusion analysis on the goods in combination with the depth information and the area information to obtain goods occlusion information, and to perform a consistency analysis on the goods in combination with the depth information and the area information to obtain goods consistency information; The quantity estimation unit is used to estimate the quantity of the goods based on the goods shielding information and the goods consistency information, and output an estimation result.

[0009] In a third aspect, an embodiment of the present invention provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the cargo management method based on the depth estimation algorithm as described in the first aspect is implemented.

[0010] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the cargo management method based on the depth estimation algorithm as described in the first aspect is implemented.

[0011] The embodiment of the present invention provides a cargo management method, device, computer equipment and storage medium based on a depth estimation algorithm, the method comprising: collecting cargo images of cargo to be estimated; obtaining depth information of the cargo images based on a depth estimation algorithm; performing cargo area analysis on the cargo according to the depth information to obtain area information; performing occlusion analysis on the cargo in combination with the depth information and the area information to obtain cargo occlusion information, and performing consistency analysis on the cargo in combination with the depth information and the area information to obtain cargo consistency information; estimating the cargo quantity based on the cargo occlusion information and the cargo consistency information, and outputting an estimation result. The embodiment of the present invention extracts depth information from the collected cargo images through a depth estimation algorithm, and can accurately analyze the cargo area. Even if there is occlusion between the cargoes, the occlusion relationship of the cargoes can be accurately identified through the combined analysis of the depth information and the area information. In addition, the embodiment of the present invention further ensures the reliability of the recognition result through consistency analysis, and by comprehensively considering the cargo occlusion information and consistency information, the cargo quantity can be effectively estimated, and a high-accuracy estimation result can be output. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying any creative work.

[0013] Figure 1 A flowchart of a cargo management method based on a depth estimation algorithm provided by an embodiment of the present invention; Figure 2 A schematic diagram of a sub-process of a cargo management method based on a depth estimation algorithm provided by an embodiment of the present invention; Figure 3 A schematic block diagram of a cargo management device based on a depth estimation algorithm provided by an embodiment of the present invention; Figure 4 A sub-schematic block diagram of a cargo management device based on a depth estimation algorithm provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0014] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0015] It should be understood that when used in this specification and the appended claims, the terms "include" and "comprises" indicate the presence of described features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.

[0016] It should also be understood that the terms used in this specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the specification of the present invention and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0017] It should be further understood that the term "and / or" used in the present description and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

[0018] See below Figure 1 , an embodiment of the present invention provides a cargo management method based on a depth estimation algorithm, specifically comprising: steps S101~S105.

[0019] Step S101, collecting cargo images of cargo to be estimated; Step S102: acquiring depth information of the cargo image based on a depth estimation algorithm; Step S103, performing cargo area analysis on the cargo according to the depth information to obtain area information; Step S104: performing an occlusion analysis on the goods in combination with the depth information and the area information to obtain goods occlusion information, and performing a consistency analysis on the goods in combination with the depth information and the area information to obtain goods consistency information; Step S105: Estimating the quantity of goods based on the goods shielding information and the goods consistency information, and outputting the estimation result.

[0020] In this embodiment, first, an image containing the goods to be estimated is collected. Then, the depth information in the image is extracted using a depth estimation algorithm. Then, the goods area is identified by analyzing the depth information to obtain the area information. Furthermore, occlusion analysis and consistency analysis are performed in combination with the depth information and the area information to determine the occlusion situation and consistency characteristics of the goods. Finally, the quantity of goods is estimated based on the occlusion information and consistency information, and the final estimation result is output.

[0021] This embodiment uses a depth estimation algorithm to extract depth information from the collected cargo images, and can accurately analyze the cargo area. Even when there is occlusion between the cargoes, the occlusion relationship of the cargoes can be accurately identified through the combined analysis of the depth information and the area information. In addition, this embodiment further ensures the reliability of the recognition results through consistency analysis. By comprehensively considering the cargo occlusion information and consistency information, it is possible to effectively estimate the cargo quantity and output a highly accurate estimation result. Therefore, the cargo management method provided in this embodiment has important application value in the field of cargo management, and can provide more accurate cargo quantity information for warehousing, logistics and other industries, thereby improving overall operational efficiency and accuracy.

[0022] In actual application scenarios, hardware facilities are configured, such as setting up cameras to capture cargo images. The installation location of the camera is crucial. Two (or more) cameras can be installed on the top of the stacker's cargo platform. The two cameras will be installed opposite to each other and are responsible for photographing the cargo positions on the opposite sides. This installation method has several obvious advantages: the camera will not interfere with the loading and unloading operations of the cargo, and because the camera is far away from the location to be photographed, it is beneficial to keep the captured image unchanged. For the camera parameter requirements, there are the following basic requirements: the resolution needs to be at least 2 million pixels (1920×1080), but 4 million pixels (2560×1440) is recommended for clearer images. The frame rate needs to be at least 30fps to adapt to the movement speed of the stacker. The shutter type should be global shutter to avoid deformation of the image during movement. The dynamic range needs to be at least 70dB to cope with different lighting environments. In addition, the camera should support external triggering to cooperate with the motion control of the stacker. In terms of lens parameters, the choice of focal length should be based on the distance from the camera installation location to the goods (usually 3-5 meters). It is recommended to use a 12-25mm adjustable focal length lens to ensure that the proportion of goods in the picture is not less than 60%. In terms of aperture requirements, the adjustable range of the F value should be F1.4-F16, and the working aperture is recommended to be set between F2.8-F4.0. The depth of field requirement should ensure the clarity of the entire cargo area. In order to control distortion, a low-distortion industrial lens should be used to ensure that the radial distortion is less than 1% and the tangential distortion is less than 0.5%. Finally, the resolution requirement of the lens is that the center resolution must reach or exceed 100lp / mm, and the edge resolution must reach or exceed 80lp / mm.

[0023] The camera calibration process includes several key steps. The first is to read the factory calibration parameters from the camera SDK. These parameters include the intrinsic parameter matrix (focal length, principal point), distortion coefficients (radial, tangential), and resolution parameters. After obtaining this data, it is necessary to verify the validity of the calibration parameters and record the calibration certificate number and date. The next step is installation position calibration. This step involves using the pose estimation tool provided by the camera SDK and performing a quick calibration based on the shelf reference point. During the calibration process, the shelf column can be used as a vertical reference, the edge of the pallet can be used as a horizontal reference, and finally the relative position relationship between the camera and the shelf is recorded.

[0024] Depth mapping simplifies calibration and is another important step in the process. Use the depth mapping function built into the SDK to verify using standard goods. The specific operation is to place the standard goods at a typical working distance, collect a single image for depth verification, and ensure that the depth error is within the allowable range (less than 2 cm). After the verification is completed, record the depth mapping parameters.

[0025] Image quality control is an important part of ensuring that the camera outputs high-quality images. First, use the SDK to automatically adjust parameters, including automatic exposure control (the target grayscale value is set between 160-180), automatic gain control, automatic white balance, and enable anti-shake function. In addition, it is necessary to monitor the image quality in real time through the SDK, including image clarity, contrast, and noise level, and set automatic alarms and adjustment mechanisms to ensure that the image quality always meets the standards.

[0026] In subsequent maintenance management, daily inspections and recalibration trigger conditions need to be managed. Daily inspections include using the SDK's self-diagnosis function to check camera status, monitor image quality indicators, and record operation logs. When recalibration conditions are triggered, it is necessary to replace the camera, respond to abnormal calibration parameters reported by the SDK, respond to obvious image quality degradation, and recalibrate when measurement accuracy exceeds tolerance (verified by standard goods).

[0027] When capturing images of goods, when the stacker passes by the side of the cargo space, the upper software will notify the camera to capture the cargo space. After the capture is completed, the subsequent algorithm system will process the captured information and identify the quantity of goods. Then, this information will be compared with the data in the warehouse management system to realize inventory counting.

[0028] In one embodiment, if Figure 2 As shown, the step S102 includes: steps S201~S205.

[0029] Step S201, preprocessing the cargo image; Step S202: input the preprocessed cargo image into a monocular depth estimation model, and the monocular depth estimation model outputs a corresponding depth image; Step S203, performing relative depth to absolute distance conversion processing on the depth image, and after the processing is completed, performing bilateral filtering denoising and outlier detection processing on the depth image; Step S204: generating a three-dimensional point cloud image from the depth image, and performing point cloud denoising processing on the three-dimensional point cloud image; Step S205: extracting the depth information of the cargo based on the three-dimensional point cloud image after the point cloud noise reduction processing.

[0030] In this embodiment, before performing depth estimation on the cargo image, the input image needs to be preprocessed. For example, the cargo image can be first scaled to 384×384 pixels, which is a balance between computational efficiency and accuracy. Next, the brightness of the cargo image is normalized to the range of [0,1] to meet the requirements of the model. In addition, Gaussian filtering is used for denoising, with a filter kernel size of 3×3 and a standard deviation σ set to 0.5.

[0031] The execution of depth estimation relies on the Depth Anything model (the monocular depth estimation model described above). The input of this model is a preprocessed RGB image, and the output is a 384×384 relative depth map. The depth ranges from 0 (nearest) to 1 (farthest). In order to increase the inference speed, FP16 half-precision calculation can be used, which can increase the speed by 2 times. It can also be optimized by TensorRT to increase the inference speed by another 1.5 times.

[0032] Furthermore, for the depth image output by the Depth Anything model, depth map post-processing is performed, and the depth map post-processing includes depth to absolute distance conversion, bilateral filtering denoising and outlier processing. First, the depth to absolute distance conversion is based on camera parameters and calibration data, and its actual effective range can be 2-6 meters, which can match the actual size of the shelf. Secondly, bilateral filtering denoising can use parameters with spatial σ=3 and value range σ=0.1, which can remove noise while maintaining the edges. Finally, outlier processing can invalidate out-of-range values ​​(for example, less than 2 meters or greater than 6 meters), and can also remove isolated points through 8-neighborhood checks.

[0033] In a specific embodiment, the specific process of converting depth to absolute distance is as follows: Mapping conversion based on camera calibration parameters: 1. First, obtain the camera's intrinsic parameter matrix and distortion coefficient; 2. Establish geometric transformation relationship according to the camera's installation height and pitch angle.

[0034] Assume that the camera is installed at a height of H (usually the shelf height + 0.5 meters), and the camera is tilted downward at an angle of θ (usually 15°). For any point P(x, y) on the image: (1) First, establish the image coordinate system: the origin is at the center of the image, the x-axis is positive to the right, and the y-axis is positive downward; (2) Calculate the sight vector of the point: Horizontal deflection angle α = arctan((x - c x ) / f x ); Vertical deflection angle β = arctan((y - c y ) / f y ); Among them, (c x ,c y ) is the coordinate of the principal point, f x ,f y is the focal length parameter; (3) Spatial geometric relationship: The angle between the sight vector and the vertical direction is φ=θ+β; The horizontal projection angle is kept as α; (4) Actual distance calculation: Vertical distance Dv = H / cos(φ); Horizontal distance Dh = Dv * tan(φ); The straight-line distance in space D = sqrt(Dv² + Dh²).

[0035] 3. Use the principle of similar triangles to map the relative depth value to the actual distance: Assume that the relative depth value range is [0,1], according to the known closest distance (for example, d min =2m) and the farthest distance (e.g. d max =6m). Note: The shortest and longest distance is determined by the camera performance. For any relative depth value d, the corresponding actual distance D is calculated as: D = d min + d*(d max -d min ).

[0036] 4. Correction considering perspective factors: Calculate the angle between the line of sight and the vertical direction based on the pixel position, and use the cosine theorem to correct the depth value (1) Perspective distortion correction: Due to the non-vertical nature of the camera's viewing angle, the same depth plane presents a trapezoidal distribution on the image, so the depth value of the edge area needs to be compensated according to the viewing angle: a. Calculate the distance r from the pixel to the center of the image; b. Get the angle between the point and the optical axis ω = arctan(r / f), where f is the equivalent focal length; c. Apply cosine correction: D 实际 = D 测量 / cos(ω) (2) Correction of perspective effect: For goods stacked on an inclined or stepped surface: a. Detect the normal vector of the local plane; b. Calculate the angle γ between the sight vector and the normal vector; c. Corrected depth value: D 修正 = D 实际 * cos(γ); (3) Edge transition processing: At the edge of the cargo, set the width of the transition zone (usually 2-3 pixels) and perform weighted averaging of depth values ​​within the transition zone to avoid measurement errors caused by sudden depth changes. (4) Correction factor calibration: Use standard test objects to collect data at different viewing angles to establish a viewing angle-correction factor correspondence table, and calculate the correction coefficients for intermediate viewing angles based on interpolation.

[0037] 5. Apply calibration coefficients for fine-tuning: Use a standard test object to obtain the correction factor, and then compensate for the system error based on the correction factor.

[0038] 6. After installation and calibration, verify the conversion accuracy (optional): During the calibration phase, standard test objects can be used to verify the conversion results to ensure that the measurement error is within the range of ±1cm.

[0039] In another specific embodiment, when generating a three-dimensional point cloud image for the depth image, it can be implemented based on the back projection of the camera intrinsic parameters, that is, converting the pixel coordinates into camera coordinates and converting the depth value into spatial distance. Specifically, the mapping transformation from the image plane to the camera coordinate system involves processing each pixel point (u, v) on the depth map. Among them, u represents the horizontal pixel position (0 to image width-1), v represents the vertical pixel position (0 to image height-1), and d represents the absolute depth value corresponding to the pixel. Next, the camera intrinsic parameter matrix is ​​used to perform coordinate transformation to normalize the pixel coordinates to the camera principal point. The specific calculation method is: x' = (u - c x ) / f x , y' = (v - c y ) / f y ; Among them, (c x ,c y ) is the coordinate of the principal point, f x ,f y is the focal length parameter.

[0040] Finally, calculate the three-dimensional space coordinates: X = x' * d, Y = y' * d, Z = d; Through the above steps, the spatial point (X, Y, Z) in the camera coordinate system can be obtained.

[0041] Furthermore, when the three-dimensional point cloud image is subjected to point cloud denoising, it mainly includes statistical outlier removal and voxel downsampling. Specifically, when processing point cloud data, the first step is to remove statistical outliers. This process includes local density analysis, calculating the local neighborhood features for each point P, and the search radius R is set to 2 cm, which is related to the roughness of the cargo surface. Within this radius, the number of neighboring points N is counted, and the average distance D_avg to the neighboring points is calculated. The outlier determination rule includes setting the minimum neighboring point number threshold N_min to 10, calculating the mean μ and standard deviation σ of the average distance in the local area. If the number of neighboring points N is less than N_min, or the absolute value of the difference between the average distance D_avg and the mean μ is greater than 2σ, the point is marked as an outlier. After the first round of obvious outliers are removed, the local statistical features are updated, and secondary optimization is performed to remove edge noise. The next step is voxel downsampling. First, space division is performed, and a 1cm×1cm×1cm three-dimensional grid is established to evenly divide the point cloud space into voxel units. For each voxel unit, the spatial coordinates of all points inside are counted, and the centroid point is calculated as the representative point. At the same time, the point's depth value and reflection intensity and other attributes are saved. In the sampling optimization stage, empty voxels are processed, the empty voxel positions are marked, and the records are retained for future space occupancy analysis.

[0042] In one embodiment, the step S103 includes: Performing spatial alignment processing on the goods in combination with the depth information, and constructing a shelf coordinate system after the spatial alignment processing is completed; Based on the shelf coordinate system, a pallet for carrying the goods is positioned using a deep edge detection algorithm, and pallet features are extracted according to the positioning result; Performing a hierarchical analysis on the goods based on the shelf coordinate system; The space occupation of the goods is analyzed in combination with the pallet characteristics and the result of the layered analysis to determine whether there are goods on each layer of the pallet, and the determination result is set as the area information.

[0043] When the present embodiment uses depth information to obtain regional information, the depth information is first used to perform spatial alignment processing on the goods and establish a shelf coordinate system. Then, the pallet is positioned using a depth edge detection algorithm and the features of the pallet are extracted. Next, the goods are layered and analyzed according to the shelf coordinate system. Finally, the spatial occupancy of the goods is analyzed based on the pallet features and the results of the layered analysis to determine whether there are goods on each layer of the pallet, and this information is recorded as regional information. It is understandable that in actual warehouse management, multiple shelves are set up, and each shelf is set up with multiple layers, and each layer is set up with multiple pallets for storing goods. Therefore, the present embodiment obtains the regional information of the goods through information such as shelves and pallets.

[0044] In a specific embodiment, the spatial alignment processing of the goods is performed in combination with the depth information, and a shelf coordinate system is constructed after the spatial alignment processing is completed, including: Acquire a depth mutation band in the depth image based on the depth information, and extract a target depth mutation band that meets a preset depth value from the depth mutation band; Extracting a linear structure from the three-dimensional point cloud image, and determining a parallel line segment group according to the extracted linear structure; Selecting a first candidate coordinate system horizontal axis and a first candidate coordinate system vertical axis that are perpendicular to each other according to the target depth mutation band; and selecting a second candidate coordinate system horizontal axis and a second candidate coordinate system vertical axis that are perpendicular to each other according to the parallel line segment group; The shelf coordinate system is constructed by combining the first candidate coordinate system horizontal axis and the second candidate coordinate system horizontal axis to obtain the coordinate system horizontal axis, and combining the first candidate coordinate system vertical axis and the second candidate coordinate system vertical axis to obtain the coordinate system vertical axis.

[0045] When performing spatial alignment and establishing the shelf coordinate system, the coordinate system can be established based on the shelf structural features. Specifically, the main reference feature is the shelf horizontal bracket. First, the horizontal bracket features are determined, including the detection of depth map features, such as the detection of depth mutation bands. The width of the mutation band should meet the bracket specifications, usually in the range of 8-12 cm, depending on the bracket specifications and the actual project scene. The depth value should be 2-5 cm higher than the background, also depending on the bracket specifications and the actual project scene. Then perform structural feature verification, which involves detecting linear structures in the point cloud. The RANSAC method can be used to fit the main straight line segments, and two points are selected each time by random sampling to form a straight line hypothesis, and the distance from other points to the hypothesized straight line is calculated (the threshold is 1 cm). The number of supporting points (points with a distance less than the threshold) is counted, and iterative optimization is performed to find the straight line with the most supporting points. After removing the points of the fitted straight line, continue to look for the next main straight line. Screen the straight line segments whose lengths meet the standard span of the shelf, and calculate the parallelism and perpendicularity between the straight line segments. The other aspect involves multi-layer structure analysis, including detecting equally spaced parallel line segments (standard spacing is 1.5 m ± 5 cm), determining the direction of the main parallel line family, and finding line segments perpendicular to the main parallel line family as column candidates. Then, the connection between the transverse support and the column is used as the reference point, the direction of the transverse support determines the horizontal reference, and the vertical relationship of multiple transverse supports determines the Y-axis direction.

[0046] Auxiliary reference features include the vertical edge of the column as an auxiliary confirmation of the Y axis, the horizontal plane of the lateral bracket as the XZ plane reference, and the bracket spacing as a scale reference (standard spacing).

[0047] The horizontal support judgment criteria include satisfying any of the following conditions to be determined as a horizontal support candidate: Condition 1 is that the width of the depth mutation zone is within the range of 8-12 cm and is 2-5 cm higher than the background; Condition 2 is that the length of the straight line segment obtained by RANSAC fitting is greater than 2 meters and the number of support points exceeds 80% of the total number of points; Condition 3 is that more than two parallel and equally spaced (1.5 m ± 5 cm) line segments are detected. The final confirmation condition satisfies at least one of the following conditions: Confirmation condition 1 is that any two of the above three conditions are met at the same time; Confirmation condition 2 is that condition 1 is met, and a parallel line segment is detected at 1.5 m ± 5 cm above and below; Confirmation condition 3 is that condition 2 is met, and a vertical line segment (column) is detected at each end.

[0048] The criteria for judging pillar features include depth features, structural features, and pillar judgment criteria. Among them, the depth feature requires that the width of the depth mutation band is within the range of 15-20 cm, the relative background protrusion is 8-12 cm, and the depth value remains stable in the vertical direction (the change is less than 1 cm). The structural feature requires that the length of the straight line segment obtained by RANSAC fitting is greater than 3 meters (standard shelf height), the number of support points exceeds 85% of the total number of points, and the inclination angle of the straight line segment is less than 5 degrees from the gravity direction. The pillar judgment criteria include preliminary judgment (all of the following conditions are met): the width and protrusion of the depth mutation band meet the requirements, and the length and number of support points of the RANSAC fitting straight line segment meet the threshold; it also includes confirmation judgment, at least one of which is met: confirmation condition 1 is that the horizontal bracket is detected within 1 meter on both sides of the straight line segment; confirmation condition 2 is that another pillar that meets the preliminary judgment is detected at a parallel distance of 2.7 meters ± 10 cm.

[0049] Preferably, the coordinate system is precisely optimized. First, the accuracy of the data is ensured by the least squares fitting method based on multiple reference points. Then, the structure of the coordinate system is further refined by the lateral bracket edge extraction and straight line fitting technology. In order to ensure that the vertical reference is accurate, the vertical reference is established, and the origin is located and the direction of the coordinate axis is determined. In addition, by calculating the coordinate transformation matrix, the rigid body transformation from the camera coordinate system to the shelf coordinate system can be achieved. In order to optimize the correction process and take into account the measurement error, corresponding measures can be taken to ultimately control the transformation accuracy within the range of ±1cm.

[0050] In another practical application, when analyzing the cargo area, the pallet needs to be located first, which involves depth edge detection and pallet feature extraction. Depth edge detection uses the Sobel operator to extract depth discontinuities, setting the depth gradient threshold to greater than 0.5 meters per pixel. Pallet feature extraction focuses on its standard size of 1.2 meters by 1.0 meters, a height of about 15 centimeters, and an allowable error within the range of ±2 centimeters.

[0051] Next, the cargo layering analysis is performed, including vertical and horizontal analysis. In the vertical direction, the bin size is set to 5 cm, ranging from 0 to 2 meters (relative to the pallet surface) through the height histogram calculation. The standard spacing for layer spacing detection is 30 cm ± 2 cm, the minimum spacing is 25 cm, and the maximum spacing is 35 cm. The confirmation of the number of layers is based on the standard stack type (28 or 30 pieces) and verifies the rationality of the total height. In the horizontal direction, the standard spacing for column spacing detection is 25 cm ± 2 cm, based on the average width of the pallet. The column position is determined by numbering from left to right and taking into account the standard layout of the stack type.

[0052] Finally, the spatial occupancy determination is performed, which includes local feature analysis and state determination rules. In the local feature analysis, the point cloud density calculation uses a voxel size of 2 cm × 2 cm × 2 cm, and the density threshold is more than 50 points per voxel. The depth continuity check is performed by local plane fitting, and the residual threshold is set to ± 2 cm. In the state determination rules, the conditions for goods include point cloud density meeting the threshold, good depth continuity, and high compliance with the hierarchical requirements; the conditions for no goods include low point cloud density, discontinuous depth, or direct observation of the background.

[0053] Furthermore, for the depth continuity check, it can specifically include local plane fitting and continuity evaluation. Among them, when performing local plane fitting, it is necessary to first select an inspection window with a size of 10cm×10cm and slide it with a step size of 2cm (the step size is consistent with the voxel size). Next, the RANSAC method is used for plane fitting. This method selects 3 non-collinear points each time through random sampling to calculate the plane equation ax + by + cz + d = 0. Then, the distance from other points to the plane is calculated, and the number of supporting points (that is, the number of points with a distance less than 2cm) is counted. Through iterative optimization, the optimal plane can be found. There are two criteria for judging the quality of the plane: the proportion of supporting points must be greater than 85% to be considered as a valid plane; the angle between the plane normal vector and the line of sight should be less than 60°. In terms of continuity evaluation, it is necessary to compare the planes of adjacent windows to ensure that the angle between the plane normal vectors is less than 15° and the difference in plane distance is less than 2cm. The criteria for judging regional continuity are that the proportion of continuous planes must be greater than 80%, and the maximum discontinuous interval does not exceed 4cm. Finally, the residual threshold is set to ±2cm, which is consistent with the accuracy of the system.

[0054] In one embodiment, the step S104 includes: Using ray tracing technology based on line of sight projection to detect the obstruction of the goods, and determining the degree of obstruction for goods that are obstructed; wherein the degree of obstruction includes complete obstruction, partial folding and no obstruction; Combining the depth information, the area information and the degree of occlusion, determining the state of the completely occluded goods to determine whether the goods actually exist; The results of the occlusion degree and state judgment are summarized as the occlusion information.

[0055] When performing occlusion analysis in this embodiment, it is first necessary to detect the occluded area and infer the state. Occlusion detection can be achieved by a line of sight projection method, that is, projecting the line of sight from the camera position to the target position and checking the depth value of the passing point. Depending on the degree of occlusion, occlusion can be divided into three situations: complete occlusion, partial occlusion, and no occlusion. For example, complete occlusion means that the line of sight is completely blocked, partial occlusion means that more than 50% of the area is still visible, and no occlusion means that more than 90% of the area is visible.

[0056] For fully obstructed locations, state inference is required. First, a support check is performed to verify the state of the lower layer cargo and ensure that the support area is greater than 80%. Next, an adjacent state analysis is performed to check adjacent locations on the same layer and consider standard stacking rules. Finally, physical constraints are applied, including gravity support rules, stability requirements, and standard spacing restrictions.

[0057] In the result verification stage, physical rules need to be verified. Gravity support verification requires that the upper cargo must be supported by the lower layer, and the support area must be sufficient. Stability check needs to ensure that the stacking height is reasonable and the misalignment between layers is within the allowable range. Spacing verification includes horizontal spacing and vertical spacing, for example, 25cm±2cm and 30cm±2cm respectively. Through these steps, the accuracy of occlusion processing and the stability of cargo stacking can be ensured.

[0058] In a specific embodiment, when the ray tracing technology based on line of sight projection is used to detect the occlusion of the goods, firstly, a ray equation is constructed, from the camera position (C x ,C y ,C z ) to the target position (T x ,T y ,T z ), generate the ray parameter equation P(t) = C + t(TC), where t ranges from [0,1]. Then perform sampling detection and set the sampling interval Δt to 0.01, which corresponds to an actual distance of about 1cm. For each sampling point P(t), calculate the theoretical depth dt = ||P(t) - C|| and the actual depth dr (obtained from the depth map), and use this to calculate the depth difference Δd = dr-dt. According to the depth difference Δd, the occlusion situation can be determined: if Δd<-2cm, it is confirmed that there is occlusion, and the position and depth difference are recorded; if |Δd|≤2cm, it is considered that there is no occlusion.

[0059] In order to further analyze the occlusion degree of the target area, grid division is first performed, for example, the unit size is set to 2cm×2cm, and the total number of grids is 625 (25×25 grids). By performing ray tracing and counting the number of occlusion points Nb, the occlusion ratio R = Nb / N can be calculated. According to the occlusion ratio, the occlusion level can be divided into five levels: complete occlusion (R≥90%), severe occlusion (70%≤R<90%), partial occlusion (30%≤R<70%), slight occlusion (10%≤R<30%) and basically no occlusion (R<10%).

[0060] Finally, the degree of occlusion is converted to confidence (visibility). For a situation with almost no occlusion, the confidence 1-R will be greater than or equal to 90%.

[0061] In one embodiment, the step S104 further includes: Based on the depth information and the area information, the cargo is subjected to an intra-layer consistency check; wherein the intra-layer consistency check step comprises: performing a depth change control of the central area of ​​each layer of the cargo area according to a preset control strategy, and performing edge processing on each layer of the cargo area according to the structure of the central area depth change control; the edge processing comprises depth mutation rationality verification and edge depth correction; Based on the depth information and the area information, the interlayer relationship detection is performed on the cargo; wherein the interlayer relationship detection step includes: calculating the perspective deviation of the cargo area in the adjacent layer, and performing depth correction according to the result of the perspective deviation calculation; The results of the intra-layer consistency detection and the results of the inter-layer relationship detection are summarized and used as the cargo consistency information.

[0062] When performing depth consistency verification in this embodiment, it is necessary to ensure the accuracy of intra-layer consistency and inter-layer relationship. For example, intra-layer consistency requires that the depth variation of the same layer does not exceed 3 cm, and the influence of edge effects needs to be excluded. For another example, the consideration of inter-layer relationships includes ensuring that the depth difference between adjacent layers is 30 cm ± 2 cm, and taking into account the influence of viewing angle. These standards together ensure the accuracy and reliability of depth data. It is understandable that at the edge of the cargo, due to the sudden change in depth, measurement errors are prone to occur, so some verification corrections are required. The layer heights involved in this embodiment are all set based on the height of a layer of cargo in the preset business scenario, and are not fixed values.

[0063] Specifically, when performing intra-layer consistency checks, the first thing to pay attention to is the depth change control in the central area. The core area is defined as the area more than 5 cm away from the edge. The depth change threshold should be less than 2 cm, and the depth of adjacent measurement points should show a gradual trend. Special treatment is required for the edge area. The edge is defined as the range of ±5 cm from the cargo boundary. The rationality of the depth mutation needs to be verified. The depth difference with the background should be between 25 and 35 cm (standard cargo thickness). At the same time, the width of the mutation band should consider the viewing angle and resolution and be controlled at 2 to 4 cm. The edge depth correction should be based on the local plane fitting results, and the edge depth value should be extrapolated by the plane equation, but the correction is only applied when the fitting confidence exceeds 85%.

[0064] When verifying the relationship between layers, first ensure that the depth difference between adjacent layers meets the predetermined standard, such as 30cm±2cm. Next, it is necessary to compensate for the effect of viewing angle, which mainly includes two aspects: the effect of camera tilt angle and the effect of horizontal viewing angle. For the effect of camera tilt angle, it is necessary to calculate the downward tilt angle θ of the camera (usually 15°) and the angle α between the line of sight and the normal vector of the cargo surface, and apply the depth correction factor cos(α). For the effect of horizontal viewing angle, it is necessary to transform the image plane to the world coordinate system, calculate the horizontal offset angle β, and then apply the lateral depth correction cos(β).

[0065] Depth value correction is divided into vertical and horizontal directions. In the vertical direction, the actual layer height can be obtained by measuring the layer height and dividing it by cos(α), but the α angle must be less than 60° to ensure the reliability of the measurement. In the horizontal direction, the actual spacing can be obtained by measuring the spacing and dividing it by cos(β), and the β angle must be less than 45° to ensure the measurement accuracy.

[0066] Next, the correction value verification is performed, including physical constraint check and continuity verification. The physical constraint check ensures that the corrected layer height is within the range of 30cm±1cm and the corrected spacing is within the range of 25cm±1cm. The continuity verification involves checking the depth gradient of adjacent measurement points and performing outlier detection (values ​​exceeding 3 standard deviations σ). Through these steps, the accuracy of the inter-layer relationship and the reliability of the measurement data can be ensured.

[0067] After that, the verification results are processed. First, the conditions for passing the verification need to be clarified. For intra-layer consistency, three sub-conditions need to be met: the depth change in the center area is less than 2 cm, the depth mutation in the edge area is reasonable, and the continuity meets the requirements. For inter-layer relationships, the layer height is required to meet the standard (for example, 30 cm ± 2 cm), and the data should be reasonable after viewing angle compensation.

[0068] If the verification fails, you need to take appropriate handling strategies. For the case of intra-layer consistency failure, for example, if the deviation is less than 4 cm, you can use the local plane fitting results to correct it, record the correction and reduce the confidence level (for example -0.2). If the deviation is greater than or equal to 4 cm, it is marked as an abnormal area and the re-inspection process is triggered. If the re-inspection still fails, the status can be downgraded to "manual confirmation required". For the case of inter-layer relationship failure, if the layer height is abnormal, first check whether there is a problem with cargo dumping or support status. If the physical state is normal but the depth is abnormal, re-execute the view compensation. If the compensation fails, it is marked as "manual confirmation required". If the data is still abnormal after the view compensation, you need to check whether the camera parameters are accurate and re-execute the calibration process. If the problem persists, enter the manual processing process.

[0069] In one embodiment, the step S105 includes: Calculating the confidence of the cargo based on the cargo obstruction information and the cargo consistency information to obtain a confidence score of the cargo; Compare the confidence score with a preset score to obtain a score comparison result, and construct a stacking matrix according to the confidence score; The score comparison result and the stack type matrix are combined to determine whether the goods actually exist, and the quantity of the goods is estimated accordingly.

[0070] When outputting the cargo quantity estimation result, this embodiment first summarizes the location status of the cargo to obtain a confidence score. Status collection includes classification of directly observed positions, inferred positions, and abnormal positions. The visibility of the directly observed position must be greater than 90%, the depth feature must be clear, and a weight coefficient of 1.0 must be assigned. The inferred position is inferred based on physical rules, and the confidence score is between 0.8 and 0.95. The abnormal position has unclear features, there are rule conflicts, and manual confirmation is required. Confidence calculation involves basic scores, such as setting the direct observation position to 1.0, partial occlusion to 0.9, and complete inference to 0.8. In terms of quantity statistics, basic statistics include counting the positions with goods, only counting the positions with confidence scores greater than 0.8, and counting by layer. Distribution analysis generates a stacking matrix and calculates the fill rate.

[0071] Furthermore, the output results are verified, including the verification of the stacking rules and the verification of the rationality of the quantity. The verification of the stacking rules requires comparison with the standard stacking type to check the abnormal distribution. The rationality of the quantity is ensured by comparing historical data and verifying the business rules. Furthermore, the output processing includes the generation of the result report, covering the total number of quantity statistics, the number of layers and the confidence distribution, as well as the abnormal marking, including the location number, the abnormal type and the processing suggestions. In terms of data storage, it is necessary to save the image data (original image, depth map, visualization results), processing parameters (algorithm configuration, threshold setting) and statistical results (JSON format, including timestamp).

[0072] In actual application scenarios, real-time quality monitoring and feedback are used to manage goods. Specifically, first of all, quality monitoring includes real-time indicator monitoring, where the depth quality requires that the proportion of effective depth points exceeds 95% and the noise level is controlled below 1 cm. In terms of recognition quality, the confidence distribution and abnormality ratio are mainly monitored to ensure that the latter is less than 1%. In terms of performance indicators, the processing delay must be less than 1 second, and the GPU utilization rate is kept below 80%. In order to adapt to the changing environment, automatic adjustment strategies can be implemented, including depth threshold adjustment based on ambient lighting and distance changes, and algorithm parameter adjustment for adaptive noise thresholds and dynamic confidence requirements.

[0073] In terms of feedback mechanisms, on the one hand, it is achieved through immediate feedback, which includes visual display of depth map overlay, status marking and exception prompts, and status updates after each processing, which contain confidence information. On the other hand, it is achieved through historical data, which includes logging of processing parameters, intermediate results and exceptions, as well as performance tracking, such as accuracy statistics, processing efficiency and resource usage.

[0074] In an actual application scenario, a non-full stack of 30 pieces (6 layers and 5 columns) is taken as an example to illustrate the cargo management method provided in this embodiment. First, in the data input process, a raw image in RGB format with a resolution of 2560×1440 is used. The image is taken by the camera from a distance of about 4 meters, and the camera lens is tilted down 15 degrees to capture the target scene. The focal length of the camera is set to 16mm, providing a wide field of view of 60 degrees horizontally and 40 degrees vertically. In order to ensure the accuracy of image processing, the intrinsic parameter matrix and distortion coefficient of the camera can be obtained from the software development kit (SDK). Then, depth estimation is performed, that is, image preprocessing is first performed, including scaling the image to 384×384 pixels, normalizing the brightness to the range of [0,1], and applying a 3×3 Gaussian filter to remove noise. Next, the processed image is input into the DepthAnything model to obtain a relative depth map of 384×384 pixels, where the depth ranges from 0 (nearest) to 1 (farthest). Then depth post-processing is performed to convert the relative depth into actual distance based on camera parameters and apply bilateral filtering to remove depth noise. At the same time, abnormal depth values ​​(greater than 6 meters or less than 2 meters) are detected and corrected to complete the depth post-processing step.

[0075] Then, in the process of regional analysis, the pallet is first located. Through detection, the position of the edge of the pallet can be determined to be (x=320, y=980), and the plane depth of the pallet is measured to be 3.8 meters. Based on this information, a local coordinate system with the center of the pallet as the origin can be established to facilitate subsequent analysis. Next, the pallet is analyzed in layers. In the vertical direction, 5 significant height intervals can be detected, with a layer spacing of approximately 30 cm, and the positions of 6 different levels including the pallet layer are identified. In the horizontal direction, 4 column intervals are detected, with a column spacing of approximately 25 cm, and the positions of the 5 columns are determined accordingly. When performing space occupancy analysis, a depth threshold of ±5cm is set to ensure the accuracy of the analysis. The point cloud density of each position is analyzed to accurately determine the presence or absence of goods. The following judgment results are obtained, where 1 indicates that there is goods and 0 indicates that there is no goods: Level 6: 1 1 1 0 0 / / The first 3 positions are in stock; Layer 5: 1 1 1 1 0 / / The first 4 positions are in stock; Layer 4: 1 1 1 1 1 / / All in stock; Layer 3: 1 1 1 1 1 / / All in stock; Layer 2: 1 1 1 1 1 / / All in stock; Layer 1: 1 1 1 1 1 / / All in stock (including pallets); When performing occlusion processing, occlusion detection is first performed. The test results show that the goods are directly visible at the last two positions of the 6th layer and the last position of the 5th layer, and it is confirmed that there are no goods at these positions. For other positions, the visibility of the goods is good. During the verification process, gravity support checks are performed to ensure that all goods have lower-level support. In addition, the rationality of the spacing is also checked to confirm that all spacing is within the standard range. Finally, the depth consistency is checked to ensure that the depth changes of each layer are continuous.

[0076] In this quantity count, 30 locations were observed in detail. The results show that there are 30 directly observed locations, 0 blocked locations, 27 locations with goods, and 3 empty locations. In terms of result verification, it was confirmed that the layout meets the 30-piece stacking standard, the quantity is within a reasonable range, and no physical rules are violated. The output results show that the total number is 23 pieces, with a confidence level of 98%, and no abnormal marks were found. In addition, the distribution characteristics show that there are missing parts in the top layer. In the feedback and recording link, the system can generate deep visualization images to clearly show the layout and status of the goods. The specific location of each item will be clearly marked to ensure the accuracy of the information. In addition, the system will also display the quantity statistics results to provide intuitive data support for inventory management and decision-making.

[0077] In another practical application scenario, the effect of the cargo management method provided by this embodiment is evaluated. First, in terms of depth estimation technology, this embodiment successfully applies the Depth Anything model to cargo depth estimation, realizes accurate acquisition of depth information under a single view, significantly reduces the complexity of the system, and does not require multi-camera calibration. Secondly, in terms of spatial analysis technology, this embodiment develops a cargo space positioning method based on depth information, which can accurately identify non-full stacking conditions and establish a complete occlusion processing mechanism. Finally, in terms of system integration innovation, this embodiment simplifies the camera deployment requirements, requires only a single camera configuration, optimizes the image acquisition process, improves acquisition efficiency, and maximizes the use of collaborative control of stacker motion.

[0078] Specifically, when conducting technical performance evaluation, depth estimation performance evaluation is one of the key links. First, the depth estimation accuracy evaluation requires the use of standard test objects, measurements within a range of 2-6 meters, recording absolute errors and relative errors, and statistically analyzing error distribution characteristics. Secondly, the effective depth point evaluation requires calculating the effective point ratio, analyzing spatial distribution characteristics, evaluating noise levels, and testing edge effects. Through these methods, the performance of depth estimation technology can be fully understood.

[0079] When evaluating recognition performance, we first focus on whole-pallet recognition. This includes testing under standard lighting conditions to ensure that the system performs in an ideal environment. In addition, different pallet types are tested to verify the system's adaptability to various stacking methods. Through these tests, the recognition accuracy rate can be statistically analyzed and the error types can be analyzed to further optimize the recognition algorithm.

[0080] For the evaluation of incomplete stack recognition, a variety of typical missing scenarios were designed to simulate the incomplete stacking situations that may occur in actual applications. By testing different missing ratios, the position accuracy of the system in identifying incomplete stacks was evaluated, and the difficulties in the recognition process were analyzed, so as to propose improvement measures.

[0081] To evaluate occlusion scenarios, we designed a variety of occlusion test scenarios to simulate occlusion situations that may occur in the real world. By testing different occlusion levels, we evaluate the accuracy of the system's inference under occlusion conditions, and analyze failure cases to identify and resolve potential problems. Through these comprehensive evaluations, we can fully understand the recognition performance of the system and provide a basis for subsequent improvement work.

[0082] When conducting adaptability assessment, the first thing to focus on is environmental adaptability assessment, which includes the test of light adaptability. You can test different light intensities, evaluate the impact of backlight on the system, and analyze shadow effects. In addition, you can also test the impact of light changes on system performance.

[0083] Secondly, business scenario adaptability assessment is also an indispensable part. In terms of stack type adaptability, we will test the performance of standard stack types, evaluate the system's adaptability to stack type variants, and analyze the effect of stack type switching. At the same time, we can also test the ability to handle abnormal stack types.

[0084] In terms of motion adaptability, you can test the system's performance at different speeds, evaluate the impact of acceleration on the system, and analyze image quality. In addition, you can test the system's synchronization performance to ensure stable operation in various motion scenarios. Through these comprehensive tests, you can fully evaluate the system's adaptability and ensure that it can meet the expected performance standards in various environments and business scenarios.

[0085] When conducting an economic evaluation, the costs must be carefully calculated first. Hardware costs include a list of required equipment, an estimate of installation costs, a forecast of maintenance costs, and an evaluation of future update costs. Operating costs involve the calculation of energy consumption, labor costs, maintenance costs, and training costs. Benefit evaluation is divided into two parts: direct benefits and indirect benefits. Direct benefits include improved inventory efficiency, labor savings, improved accuracy, and time savings. Indirect benefits are reflected in improved management efficiency, enhanced inventory accuracy, improved decision support, and improved customer satisfaction. By comprehensively evaluating these costs and benefits, a scientific and reasonable judgment can be made on the economic feasibility of the project.

[0086] Figure 3 A schematic block diagram of a cargo management device 300 based on a depth estimation algorithm provided by an embodiment of the present invention, the device 300 includes: An image acquisition unit 301 is used to acquire images of goods to be estimated; A depth estimation unit 302, configured to obtain depth information of the cargo image based on a depth estimation algorithm; A first analysis unit 303 is used to perform cargo area analysis on the cargo according to the depth information to obtain area information; A second analysis unit 304 is configured to perform an occlusion analysis on the goods in combination with the depth information and the area information to obtain goods occlusion information, and to perform a consistency analysis on the goods in combination with the depth information and the area information to obtain goods consistency information; The quantity estimation unit 305 is used to estimate the quantity of goods based on the goods shielding information and the goods consistency information, and output an estimation result.

[0087] In one embodiment, if Figure 4 As shown, the depth estimation unit 302 includes: A preprocessing unit 401, used for preprocessing the cargo image; The model processing unit 402 is used to input the preprocessed cargo image into the monocular depth estimation model, and the monocular depth estimation model outputs a corresponding depth image; A post-processing unit 403 is used to convert the relative depth into absolute distance on the depth image, and after the processing is completed, perform bilateral filtering denoising and outlier detection on the depth image; A point cloud generating unit 404 is used to generate a three-dimensional point cloud image from the depth image, and perform point cloud denoising processing on the three-dimensional point cloud image; The point cloud denoising unit 405 is used to extract the depth information of the goods based on the three-dimensional point cloud image after the point cloud denoising process.

[0088] In one embodiment, the first analysis unit 303 includes: A spatial alignment unit, used to perform spatial alignment processing on the goods in combination with the depth information, and to construct a shelf coordinate system after the spatial alignment processing is completed; A feature extraction unit, configured to locate the pallet for carrying the goods based on the shelf coordinate system using a deep edge detection algorithm, and extract pallet features according to the positioning result; A layered analysis unit, used for performing a layered analysis on the goods based on the shelf coordinate system; The information setting unit is used to analyze the space occupation of the goods in combination with the pallet characteristics and the result of the layered analysis to determine whether there are goods on each layer of the pallet, and set the determination result as the area information.

[0089] In one embodiment, the second analysis unit 304 includes: An occlusion detection unit, configured to perform occlusion detection on the goods by using a ray tracing technology based on line of sight projection, and determine the occlusion degree of the goods that are occluded; wherein the occlusion degree includes complete occlusion, partial folding and no occlusion; A state judgment unit, used to judge the state of the completely obscured goods by combining the depth information, the area information and the degree of obstruction, so as to determine whether the goods actually exist; The first summarizing unit is used to summarize the results of the occlusion degree and state judgment into the occlusion information.

[0090] In one embodiment, the second analysis unit 304 further includes: An intra-layer detection unit is used to perform intra-layer consistency detection on the cargo based on the depth information and the area information; wherein the intra-layer consistency detection step includes: performing a central area depth change control on each layer of the cargo area according to a preset control strategy, and performing edge processing on each layer of the cargo area according to the structure of the central area depth change control; the edge processing includes depth mutation rationality verification and edge depth correction; An interlayer detection unit is used to perform interlayer relationship detection on the cargo based on the depth information and the area information; wherein the interlayer relationship detection step includes: calculating the perspective deviation of the cargo area in the adjacent layer, and performing depth correction according to the result of the perspective deviation calculation; The second summarizing unit is used to summarize the results of the intra-layer consistency detection and the results of the inter-layer relationship detection as the cargo consistency information.

[0091] In one embodiment, the quantity estimation unit 305 includes: A confidence calculation unit, configured to perform confidence calculation on the cargo based on the cargo shielding information and the cargo consistency information to obtain a confidence score of the cargo; A matrix construction unit, used to compare the confidence score with a preset score to obtain a score comparison result, and to construct a stack matrix according to the confidence score; The cargo judging unit is used to judge whether the cargo actually exists by combining the score comparison result and the stack type matrix, so as to estimate the cargo quantity.

[0092] In one embodiment, the space alignment unit includes: a mutation band extraction unit, configured to obtain a depth mutation band in the depth image based on the depth information, and extract a target depth mutation band that meets a preset depth value from the depth mutation band; A linear structure extraction unit, used to extract a linear structure from the three-dimensional point cloud image, and determine a parallel line segment group according to the extracted linear structure; A candidate selection unit is used to select a first candidate coordinate system horizontal axis and a first candidate coordinate system vertical axis that are perpendicular to each other according to the target depth mutation band; and to select a second candidate coordinate system horizontal axis and a second candidate coordinate system vertical axis that are perpendicular to each other according to the parallel line segment group; The coordinate system construction unit is used to select the horizontal axis of the coordinate system by combining the first candidate coordinate system horizontal axis and the second candidate coordinate system horizontal axis, and to select the vertical axis of the coordinate system by combining the first candidate coordinate system vertical axis and the second candidate coordinate system vertical axis, so as to construct the shelf coordinate system.

[0093] Since the embodiments of the apparatus part correspond to the embodiments of the method part, please refer to the description of the embodiments of the method part for the embodiments of the apparatus part, which will not be repeated here.

[0094] The embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed, the steps provided in the above embodiment can be implemented. The storage medium may include: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program codes.

[0095] The embodiment of the present invention also provides a computer device, which may include a memory and a processor, wherein a computer program is stored in the memory, and when the processor calls the computer program in the memory, the steps provided in the above embodiment may be implemented. Of course, the computer device may also include various network interfaces, power supplies and other components.

[0096] The various embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same and similar parts between the various embodiments can be referred to each other. For the system disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of this application, several improvements and modifications can be made to the present application, and these improvements and modifications also fall within the scope of protection of the claims of this application.

[0097] It should also be noted that, in this specification, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprises", "comprising" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the statement "comprising a ..." does not exclude the presence of other identical elements in the process, method, article or device including the element.

Claims

1. A cargo management method based on a depth estimation algorithm, characterized in that: include: Collecting cargo images of cargo to be estimated; Acquiring depth information of the cargo image based on a depth estimation algorithm; Performing cargo area analysis on the cargo according to the depth information to obtain area information; performing an occlusion analysis on the goods in combination with the depth information and the area information to obtain goods occlusion information, and performing a consistency analysis on the goods in combination with the depth information and the area information to obtain goods consistency information; Cargo quantity estimation is performed based on the cargo shielding information and cargo consistency information, and an estimation result is output.

2. The cargo management method based on the depth estimation algorithm according to claim 1 is characterized in that: The acquiring the depth information of the cargo image based on the depth estimation algorithm includes: Preprocessing the cargo image; Inputting the preprocessed cargo image into a monocular depth estimation model, and the monocular depth estimation model outputs a corresponding depth image; Performing relative depth to absolute distance conversion processing on the depth image, and after the processing is completed, performing bilateral filtering denoising and outlier detection processing on the depth image; Generating a three-dimensional point cloud image from the depth image, and performing point cloud denoising processing on the three-dimensional point cloud image; Based on the three-dimensional point cloud image after point cloud noise reduction processing, the depth information of the goods is extracted.

3. The cargo management method based on the depth estimation algorithm according to claim 2 is characterized in that: The performing cargo area analysis on the cargo according to the depth information to obtain area information includes: Performing spatial alignment processing on the goods in combination with the depth information, and constructing a shelf coordinate system after the spatial alignment processing is completed; Based on the shelf coordinate system, a pallet for carrying the goods is positioned using a deep edge detection algorithm, and pallet features are extracted according to the positioning result; Performing a hierarchical analysis on the goods based on the shelf coordinate system; The space occupation of the goods is analyzed in combination with the pallet characteristics and the result of the layered analysis to determine whether there are goods on each layer of the pallet, and the determination result is set as the area information.

4. The cargo management method based on the depth estimation algorithm according to claim 1, characterized in that: The step of performing an occlusion analysis on the cargo in combination with the depth information and the area information to obtain cargo occlusion information includes: Using ray tracing technology based on line of sight projection to detect the obstruction of the goods, and determining the degree of obstruction for goods that are obstructed; wherein the degree of obstruction includes complete obstruction, partial folding and no obstruction; Combining the depth information, the area information and the degree of occlusion, determining the state of the completely occluded goods to determine whether the goods actually exist; The results of the occlusion degree and state judgment are summarized as the occlusion information.

5. The cargo management method based on the depth estimation algorithm according to claim 1, characterized in that: The cargo area includes multiple layers, and the cargo consistency analysis is performed on the cargo in combination with the depth information and the area information to obtain cargo consistency information, including: Based on the depth information and the area information, the cargo is subjected to an intra-layer consistency check; wherein the intra-layer consistency check step comprises: performing a depth change control of the central area of ​​each layer of the cargo area according to a preset control strategy, and performing edge processing on each layer of the cargo area according to the structure of the central area depth change control; the edge processing comprises depth mutation rationality verification and edge depth correction; Based on the depth information and the area information, the interlayer relationship detection is performed on the cargo; wherein the interlayer relationship detection step includes: calculating the perspective deviation of the cargo area in the adjacent layer, and performing depth correction according to the result of the perspective deviation calculation; The results of the intra-layer consistency detection and the results of the inter-layer relationship detection are summarized and used as the cargo consistency information.

6. The cargo management method based on the depth estimation algorithm according to claim 1 is characterized in that: The estimating the quantity of goods based on the goods shielding information and the goods consistency information, and outputting the estimation result, includes: Calculating the confidence of the cargo based on the cargo obstruction information and the cargo consistency information to obtain a confidence score of the cargo; Compare the confidence score with a preset score to obtain a score comparison result, and construct a stacking matrix according to the confidence score; The score comparison result and the stack type matrix are combined to determine whether the goods actually exist, and the quantity of the goods is estimated accordingly.

7. The cargo management method based on the depth estimation algorithm according to claim 3 is characterized in that: The spatial alignment processing of the goods is performed in combination with the depth information, and a shelf coordinate system is constructed after the spatial alignment processing is completed, including: Acquire a depth mutation band in the depth image based on the depth information, and extract a target depth mutation band that meets a preset depth value from the depth mutation band; Extracting a linear structure from the three-dimensional point cloud image, and determining a parallel line segment group according to the extracted linear structure; Selecting a first candidate coordinate system horizontal axis and a first candidate coordinate system vertical axis that are perpendicular to each other according to the target depth mutation band; and selecting a second candidate coordinate system horizontal axis and a second candidate coordinate system vertical axis that are perpendicular to each other according to the parallel line segment group; The shelf coordinate system is constructed by combining the first candidate coordinate system horizontal axis and the second candidate coordinate system horizontal axis to obtain the coordinate system horizontal axis, and combining the first candidate coordinate system vertical axis and the second candidate coordinate system vertical axis to obtain the coordinate system vertical axis.

8. A cargo management device based on a depth estimation algorithm, characterized in that: include: An image acquisition unit, used for acquiring images of goods to be estimated; A depth estimation unit, configured to obtain depth information of the cargo image based on a depth estimation algorithm; A first analysis unit, configured to perform cargo area analysis on the cargo according to the depth information to obtain area information; A second analysis unit is used to perform an occlusion analysis on the goods in combination with the depth information and the area information to obtain goods occlusion information, and to perform a consistency analysis on the goods in combination with the depth information and the area information to obtain goods consistency information; The quantity estimation unit is used to estimate the quantity of the goods based on the goods shielding information and the goods consistency information, and output an estimation result.

9. A computer device, characterized in that: The method comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method for managing goods based on the depth estimation algorithm as described in any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the cargo management method based on the depth estimation algorithm according to any one of claims 1 to 7 is implemented.

Images