An occluded object detection method and system based on multi-view fusion

Abstract

The application relates to the field of computer vision and robot perception, and discloses a multi-view fusion-based occluded target detection method and system, which comprises the following steps: synchronously collecting color images and depth images of an occluded target in a goods stacking scene from different views, and keeping a fixed relative position relationship between cameras of different views; performing target detection on the color images of each view, combining the depth images, and outputting initial detection results; calculating three-dimensional detection results of each view based on the initial detection results, wherein the three-dimensional detection results at least contain boundary box center coordinates and confidence; calculating the center distance of the three-dimensional detection results of each view, and identifying the multi-view detection results of the same goods when the distance is less than a preset threshold; and obtaining fusion detection results of the occluded target based on the boundary box center coordinates and the confidence of the multi-view detection results of the same goods. The application can significantly improve the detection performance of an occluded scene.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computer vision and robot perception, and in particular to a multi-view fusion-based occluded target detection method and system, mainly applied to intelligent logistics equipment, automated unloading, cargo sorting and other application fields that need to handle complex occlusion scenarios. BACKGROUND

[0002] In intelligent logistics and automated unloading applications, target detection is a core technical link. However, existing target detection techniques still have significant limitations when faced with complex occlusion scenarios.

[0003] In intelligent unloading robot applications, the existing technology has the following key problems:

[0004] Problem one: detection blind area in occlusion scenarios In the scenario of densely stacked goods, single-view observation often has a large blind area. When goods A is partially occluded by goods B or the end of the robotic arm occludes the box at a certain angle, traditional detection algorithms are likely to judge it as background or produce false positives.

[0005] Problem two: coordinate transformation cumulative error and its propagation effect The existing multi-view fusion method has a serious cumulative error problem when performing coordinate system conversion, mainly in the following aspects:

[0006] First, the compound propagation of calibration error: each camera in the multi-view system needs to be calibrated independently, and the calibration error will be propagated and accumulated through the transformation matrix. Let the point in the camera coordinate system be , and the cumulative error ϵ can be expressed as:

[0007]

[0008] where is the three-dimensional coordinates of the target in the camera coordinate system, T is the 4x4 transformation matrix from the camera coordinate system to the base coordinate system, is the true coordinates of the target in the base coordinate system, and ϵ is the cumulative error of coordinate transformation;

[0009] Second, the coupling of rotation and translation errors: traditional methods treat rotation and translation errors as a whole and cannot independently compensate for individual error components, resulting in error amplification. In long-distance target detection, the angle deviation of the rotation error will be significantly amplified by the distance factor:

[0010]

[0011] where is the angle deviation (radians) of the rotation error, is the distance from the target to the camera, For the amplified position error;

[0012] Furthermore, time-varying calibration drift: due to mechanical vibration, thermal expansion and contraction and other factors, the relative position between the camera and the robot arm will change slightly, resulting in the drift of the calibration parameters over time, but the existing method lacks dynamic monitoring and compensation mechanism.

[0013] Finally, regional error distribution is uneven: the calibration accuracy of different regions of the workspace is different, but the traditional method uses a global unified transformation matrix, which cannot optimize the calibration accuracy of local regions.

[0014] Traditional methods lack hierarchical modeling and intelligent compensation mechanism for cumulative errors, resulting in insufficient spatial alignment accuracy when fusing multi-view data, which significantly affects the success rate of grabbing and work efficiency in intelligent unloading tasks with high precision positioning requirements.

[0015] Problem three: most existing multi-view fusion methods use static fusion mechanisms such as weighted average or voting, which are difficult to dynamically adjust according to the confidence, depth quality and spatial geometric consistency of different view detection results. Due to the lack of intelligent weight allocation strategy, redundant information between different views cannot be fully utilized, and even low-quality detection results may be introduced, which may weaken the overall detection performance. At the same time, existing methods have not established a fusion quality evaluation and adaptive optimization mechanism, resulting in the failure of multi-view fusion to play its due role.

[0016] In summary, existing target detection techniques still have significant shortcomings in complex occlusion scenarios: on the one hand, single-view observation leads to blind spots in target recognition; on the other hand, coordinate transformation cumulative error and lack of intelligent multi-view fusion strategy further restrict the detection accuracy and robustness of the system. Therefore, it is urgent to propose a detection method based on multi-view information fusion, taking into account occlusion adaptability and high-precision spatial geometric modeling, to break through the technical bottlenecks of existing technologies. SUMMARY

[0017] The main purpose of the present application is to solve the technical problem of insufficient detection accuracy of single-view target detection in complex occlusion scenarios in the prior art. A multi-view fusion-based occluded target detection method, comprising the following steps:

[0018] Synchronously collecting color images and depth images of occluded targets in the goods stacking scene from different views, and maintaining a fixed relative position relationship between the cameras of different views;

[0019] Performing target detection on the color images of each view, combining the depth images, and outputting the initial detection results, which at least include two-dimensional bounding box coordinates, depth information and confidence; the confidence is used to judge the reliability of the bounding box;

[0020] Based on the initial detection result, a three-dimensional detection result of each view is calculated, and the three-dimensional detection result at least includes a bounding box center coordinate and a confidence level;

[0021] A center distance of the three-dimensional detection result of each view is calculated, and when the distance is less than a preset threshold, the multi-view detection results of the same goods are identified;

[0022] Based on the bounding box center coordinate and the confidence level of the multi-view detection results of the same goods, a fusion detection result of the occluded target is obtained.

[0023] The second aspect of the present application provides an occluded target detection system based on multi-view fusion, comprising:

[0024] A multi-view data acquisition unit is configured to synchronously acquire color images and depth images of occluded targets in a goods stacking scene from different views, and the cameras at different views maintain a fixed relative position relationship;

[0025] A target detection unit is configured to perform target detection on the color images of each view, combine the depth images, and output an initial detection result, wherein the initial detection result at least includes a two-dimensional bounding box coordinate, depth information and a confidence level;

[0026] A coordinate conversion unit is configured to calculate a three-dimensional detection result of each view based on the initial detection result, and the three-dimensional detection result at least includes a bounding box center coordinate and a confidence level;

[0027] A multi-view fusion unit is configured to calculate a center distance of the three-dimensional detection result of each view, and when the distance is less than a preset threshold, the multi-view detection results of the same goods are identified; and based on the bounding box center coordinate and the confidence level of the multi-view detection results of the same goods, a fusion detection result of the occluded target is obtained.

[0028] The third aspect of the present application provides an electronic device, comprising a memory and at least one processor, the memory stores instructions, and the memory and the at least one processor are interconnected by a circuit; the at least one processor calls the instructions in the memory, so that the electronic device executes the above-mentioned multi-view fusion-based occluded target detection method.

[0029] The fourth aspect of the present application provides a computer readable storage medium, the computer readable storage medium stores instructions, when it runs on a computer, makes the computer execute the above-mentioned multi-view fusion-based occluded target detection method.

[0030] The present application has the following beneficial effects:

[0031] 1.The present application effectively solves the target occlusion problem under single view through multi-view cooperative observation mechanism. Multiple RGB-D cameras are used to observe the target scene from different angles at the same time, realizing view complementation. When the goods are occluded in a certain view, other views can still provide effective observation, significantly improving the integrity of target detection in complex scenes.

[0032] 2.The present application innovatively establishes a unified base coordinate system fusion framework, and realizes high-precision spatial alignment of detection results from different views through an accurate geometric transformation model. This method not only ensures the spatial consistency of multi-view data, but also provides a reliable geometric basis for subsequent intelligent fusion.

[0033] 3.The present application proposes an intelligent association determination mechanism based on three-dimensional space Euclidean distance, which can accurately identify the correspondence of detection results from different views. Through Euclidean distance calculation and threshold determination, the problems of repeated detection and false association in traditional methods are effectively avoided.

[0034] 4.The present application adopts a multi-view fusion strategy based on confidence weighting, which fully utilizes the differences in detection quality of each view. The fusion weight dynamic adjustment mechanism can adapt to the detection quality of each view in different scenes. When temporary interference occurs in a certain view, the system can automatically reduce its weight to ensure the stability of the overall detection result.

[0035] 5.The present application realizes an advanced depth quality evaluation system, which uses random sampling combined with quartile distance filtering algorithm to effectively improve the accuracy and reliability of three-dimensional reconstruction, providing high-quality depth information for subsequent fusion algorithms. BRIEF DESCRIPTION OF DRAWINGS

[0036] Figure 1 The overall flowchart of the intelligent unloading occluded target detection method of the multi-view fusion of the present application. DETAILED DESCRIPTION

[0037] The terms "first", "second", "third", "fourth" and the like in the description and claims of the present application, and the above drawings (if any) are used to distinguish similar objects, and do not necessarily indicate a particular order or sequence. It should be understood that the data thus used can be interchanged, where appropriate, so that the embodiments described herein can be carried out in sequences other than those illustrated or described herein. In addition, the terms "comprise" or "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device that includes a series of steps or units does not necessarily limit to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0038] For ease of understanding, the specific process of the embodiments of the present application is described below. Please refer to Figure 1 The first embodiment of the occluded object detection method based on multi-view fusion in the embodiments of the present application comprises:

[0039] Color images and depth images of occluded objects in a goods stacking scene are synchronously collected from different views, and the cameras at different views maintain a fixed relative positional relationship;

[0040] Target detection is performed on the color images at each view, and the initial detection results are output in combination with the depth images, wherein the initial detection results at least include two-dimensional bounding box coordinates, depth information and confidence; the confidence is used to judge the reliability of the bounding box;

[0041] Based on the initial detection results, three-dimensional detection results at each view are calculated, wherein the three-dimensional detection results at least include bounding box center coordinates and confidence;

[0042] The center distance of the three-dimensional detection results at each view is calculated, and when the distance is less than a preset threshold, it is identified as multi-view detection results of the same goods;

[0043] Based on the bounding box center coordinates and confidence of the multi-view detection results of the same goods, the fusion detection results of the occluded objects are obtained.

[0044] As a preferred embodiment, specifically, the scheme of the present application is:

[0045] A plurality of RGB-D cameras are deployed on an intelligent unloading robot platform, and color images and depth images of a goods stacking scene in a truck or container are synchronously collected from different angles, and the cameras maintain a fixed relative positional relationship;

[0046] A deep learning detection model trained for goods occlusion scenes is used to perform independent target detection on the color images at each camera view, and the detection model can effectively detect partially occluded goods with an exposed area accounting for more than 1 / 2 of the overall area, and output initial detection results including bounding box coordinates, confidence scores and class labels;

[0047] Based on the intrinsic matrix and the hand-eye calibration matrix of each camera, the detection results at each view are converted from the pixel coordinate system to the robot base coordinate system, the accurate alignment of the detection results at different views in a unified three-dimensional space is realized, and three-dimensional detection results are obtained;

[0048] The three-dimensional Euclidean distances of the center points of the detection results at different views in the base coordinate system are calculated, and when the distance is less than a preset threshold, the multi-view detection results of the same goods are identified, and a cross-view target association relationship is established;

[0049] The multi-view detection results identified as the same goods are confidence-weighted fused, the detection frame with the highest confidence is selected as the reference frame, the center coordinates of the fused detection result are calculated by weighted average, the fusion detection result is obtained, and the goods positioning accuracy is improved and the detection blind area caused by shielding is eliminated.

[0050] Specifically:

[0051] Step 1: A multi-view detection system is constructed by using multiple RGB-D cameras, and the cameras are arranged on the fixed support of the intelligent unloading robot platform for data acquisition.

[0052] The specific content of this step is as follows:

[0053] In this embodiment, a multi-view detection system is constructed by using multiple RGB-D cameras, and the cameras are arranged on the fixed support of the intelligent unloading robot platform. The system uses a synchronous acquisition mechanism to ensure the time consistency of the images of each camera. The collected color images and depth images are preprocessed, including distortion removal, noise filtering and scale normalization.

[0054] Step 2: A deep learning model based on Transformer is used to detect targets in each camera view image, and an initial detection result is output, which at least includes two-dimensional bounding box coordinates, depth information and confidence. The detection model is specially trained on a shielding goods data set.

[0055] The specific content of this step is as follows:

[0056] A convolutional neural network is used as the backbone network to extract multi-scale features, and a special data enhancement strategy is used to improve the detection ability of the shielding target. The initial detection result includes bounding box coordinates, confidence score and class label. Each camera independently performs target detection and outputs an initial detection result format containing bounding box coordinates, confidence and depth information.

[0057] Step 3: An adaptive high-precision geometric calibration coordinate transformation model is used to convert each view detection result from the pixel coordinate system to the mechanical arm base coordinate system to obtain a three-dimensional detection result, which at least includes bounding box center coordinates and confidence. This step integrates hierarchical error modeling, dynamic calibration correction and multi-stage verification mechanism, which significantly improves the coordinate transformation accuracy.

[0058] The specific content of this step is as follows:

[0059] First, convert from pixel to camera coordinate system:

[0060] Convert the center point coordinates of the detection target bounding box from pixel coordinates to camera three-dimensional coordinates:

[0061]

[0062] where is the top-left pixel coordinate of the bounding box, is the bottom-right pixel coordinate of the bounding box, is the center pixel coordinate of the bounding box;

[0063] Convert the pixel coordinate to normalized coordinate using the camera intrinsic matrix, and combine with the depth value to get the 3D position in the camera coordinate system:

[0064]

[0065] where is the center pixel coordinate, is the focal length parameter of camera i, is the principal point coordinate of camera i, is the normalized coordinate;

[0066]

[0067] where is the conversion of depth value to meter unit, is the 3D position.

[0068] Then, transform from the camera coordinate system to the base coordinate system:

[0069] Use the transforms3d library to create a rotation matrix based on Euler angles, convert Euler angles from degrees to radians:

[0070]

[0071] where is the Euler angle of the camera relative to the base coordinate system of the robot arm (unit: degree), is the converted radian value, is the conversion factor from degree to radian;

[0072] Create a rotation matrix:

[0073]

[0074] where is the 3 × 3 rotation matrix, is the conversion function from Euler angle to rotation matrix, , , are the rotation angles (radians) of X, Y, Z axes respectively;

[0075] Construct the homogeneous transformation matrix:

[0076]

[0077] wherein is a 4x4 homogeneous transformation matrix for converting points in the camera coordinate system to the robot base coordinate system, is a 3x3 rotation matrix, is a 3x1 translation vector, is a 1x3 zero matrix;

[0078] Apply the transformation to get the coordinates in the base coordinate system and convert to millimeter units:

[0079]

[0080] wherein is the three-dimensional coordinates of the target in the robot base coordinate system (unit: millimeter), [x, y, z, 1]Tis the homogeneous coordinates of the target in the camera coordinate system (unit: meter), is the coordinate transformation matrix, and 1000 is the unit conversion coefficient from meters to millimeters;

[0081] Finally, depth quality assessment and noise filtering are performed:

[0082] To ensure the reliability of the depth information, depth quality assessment is performed on each detection box. A number of random coordinate points are generated within the bounding box:

[0083]

[0084]

[0085] wherein and are a set of pixel coordinate points generated randomly within the bounding box, is the boundary of the bounding box in the X-axis direction, is the boundary in the Y-axis direction, is the number of sampling points (typical value is 50-100), is a random integer generation function;

[0086] Get the corresponding depth value to form a sampling set:

[0087]

[0088] wherein is the set of depth values obtained by sampling, is the input depth image, is the pixel coordinates of the random sampling points, and the square brackets represent the index operation of the image;

[0089] ​Invalid points with depth value of 0 are filtered, and the quartile deviation method is used to remove outliers for the remaining valid depth values. The robust depth standard deviation is calculated to evaluate the depth quality. When the depth quality does not meet the threshold requirement, the detection box is filtered.

[0090] For the target bounding box detected by the camera, first calculate the center point pixel coordinates:

[0091]

[0092] wherein, is the pixel coordinate of the target center point in the image, is the boundary of the bounding box in the X-axis direction, is the boundary in the Y-axis direction;

[0093] Convert the pixel coordinates to three-dimensional coordinates in the camera coordinate system using the camera intrinsic parameters:

[0094]

[0095] wherein is the three-dimensional coordinate of the target in the camera coordinate system, is the pixel coordinate of the target center point in the image, is the principal point coordinate (pixel) of camera i, is the focal length parameter (pixel) of camera i, and Z is the depth value (m).

[0096] Convert to the base coordinate system through the hand-eye calibration matrix, and the transformation matrix is composed of a rotation matrix R and a translation vector t:

[0097]

[0098] wherein is the homogeneous coordinate of the target in the base coordinate system of the robot arm, is the homogeneous coordinate of the target in the camera coordinate system, is the 3x3 rotation matrix of the i-th camera, is the 3x1 translation vector of the i-th camera, is a 1x3 zero vector.

[0099] Step 4: Realize cross-view target association recognition by calculating the three-dimensional spatial Euclidean distance of different view detection results in the base coordinate system. When the spatial distance is less than a predetermined threshold, it is determined that the multi-view detection results are of the same target.

[0100] The specific content of this step is as follows:

[0101] For detection results from different cameras, calculate their three-dimensional spatial distance in the base coordinate system:

[0102] Let the target center coordinates detected by camera j be The spatial distance is:

[0103]

[0104] When , it is determined to be the same target. In this embodiment, a suitable spatial correlation threshold is set.

[0105] Step 5: Perform intelligent weighted fusion on the multi-view detection results identified as the same target, select the detection box with the highest confidence as the reference, calculate the fused center coordinates and bounding box information through weighted average, and obtain the fused detection result.

[0106] The specific content of this step is as follows:

[0107] For multiple detection results identified as the same target, intelligent fusion is performed:

[0108] Select the detection box with the highest confidence as the reference;

[0109] Calculate the dynamic weight based on the detection quality, which comprehensively considers the confidence, depth quality and spatial geometric consistency:

[0110]

[0111] wherein is the dynamic weight of the kth detection result, is the confidence score, is the depth quality factor, is the geometric consistency factor;

[0112] The depth quality factor α_depth(k) is calculated based on the robust depth standard deviation:

[0113]

[0114] wherein is the robust depth standard deviation of the kth detection result, is the depth quality threshold parameter.

[0115] The geometric consistency factor is based on the spatial geometric similarity of the detection result and the reference box:

[0116]

[0117] wherein is the center coordinates of the kth detection result, is the center coordinates of the reference box with the highest confidence, ​a geometric consistency threshold parameter, denotes the Euclidean distance.

[0118] Calculate the weighted average fusion coordinates:

[0119]

[0120] wherein is the center coordinate of the kth detection result, is the final center coordinate after weighted fusion;

[0121] Calculate the comprehensive confidence using weighted average and introducing a perspective diversity reward mechanism:

[0122]

[0123] wherein is a multi-perspective reward coefficient, is the number of perspectives participating in fusion;

[0124] Update the fused detection results, including base coordinate system coordinates, comprehensive confidence, fusion flag, source camera list and spatial uncertainty evaluation.

[0125] Step 6: Optionally, to ensure the reliability of the fusion results, perform depth quality assessment on each detection box, filter depth noise using random sampling and statistical filtering method, and ensure the accuracy of three-dimensional detection.

[0126] The specific content of this step is as follows:

[0127] In order to ensure the reliability of three-dimensional detection of the target after multi-perspective fusion, perform depth quality assessment on each fused detection box. Let the fused detection box be (in the base coordinate system), perform random sampling within the bounding box to obtain a set of depth values.

[0128] First, generate random coordinate points within the bounding box region, and obtain the corresponding depth values to form a sampling set:

[0129]

[0130] Filter out invalid points with depth value of 0 or abnormality, and the remaining valid depth values Use quantile statistical method to remove outliers:

[0131]

[0132]

[0133]

[0134] Filter the bounding box by analyzing the determination of the robust depth standard deviation and the area rationality test of the sampling set, and obtain the depth information.

[0135] The calculation formula of the robust depth standard deviation is:

[0136]

[0137] wherein, is a single depth value in the filtered depth value set, is the arithmetic mean of the filtered depth value, is the number of elements in the filtered depth value set, is the robust depth standard deviation;

[0138] When or , filter the bounding box corresponding to the sampling set, wherein is the depth standard deviation threshold, is the maximum depth threshold.

[0139] The determination rule of the area rationality test is:

[0140]

[0141] wherein , is the lower left corner coordinate of the fusion bounding box in the base coordinate system, is the upper right corner coordinate, is the projection area of the fusion bounding box in the YZ plane;

[0142] When , it is considered to be an abnormally large box and is filtered, wherein is the maximum area threshold.

[0143] Step 7: Establish a time sequence stabilization mechanism for the fusion detection result, eliminate detection jitter and maintain target tracking consistency.

[0144] The specific content of this step is as follows:

[0145] In order to eliminate the jitter phenomenon of the fusion detection result between consecutive frames, a time window stabilization mechanism is established. Let the current frame fusion detection result be , and the historical frame cache result be . For each current frame target , calculate the IoU overlap between it and the historical frame target :

[0146]

[0147] The relative area ratio is calculated at the same time to determine the consistency of the target scale:

[0148]

[0149] wherein represents the intersection area of two targets, represents the union area of two targets, , represents the area of the target ;

[0150] When and , is the IoU overlap threshold, is the relative area ratio threshold; the continuous observations identified as the same target are executed target association and attribute fusion.

[0151] The position coordinates, confidence and geometric attributes of the target are updated by weighted average, and the weight is designed as:

[0152]

[0153] wherein α is the time decay factor, T is the current time, and t is the historical time. A sliding window mechanism is established to maintain a fixed length of historical sequence, ensuring the real-time response capability of the system in dynamic scenes.

[0154] Step 8: Build a three-dimensional space constraint relationship graph of multi-view fusion target to generate an intelligent unloading optimal sequence.

[0155] The specific content of this step is as follows:

[0156] Based on the stable target set after multi-view fusion , a three-dimensional space constraint relationship graph is built to guide the intelligent unloading operation sequence. For any target pair , the condition for determining the spatial blocking relationship is:

[0157] Depth constraint condition:

[0158]

[0159] wherein represents the X-axis coordinate (depth direction) of the target in the base coordinate system of the robot arm, represents the X-axis coordinate of the target in the base coordinate system, is the depth difference threshold, used to determine whether two targets are in the same depth layer, and the typical value range is 50-200 millimeters.

[0160] Horizontal overlap condition: the projection of the target on the YZ plane has significant overlap, and the overlap area accounts for more than a set threshold.

[0161] When the target is in front and meets the above constraints, a blocking relationship is established. After traversing all target pairs, the constraint degree of each target is calculated:

[0162]

[0163] where, represents the constraint degree of the target , i.e. the number of other targets that block it; is an indicator function that returns 1 when the condition in the parentheses is true, and 0 otherwise; is a blocking condition judgment function that returns true when target blocks target , and the judgment conditions include: (1) X coordinate of is less than the X coordinate of (i.e. is in front), (2) both targets meet the depth constraint condition, and (3) the projections of the two targets on the YZ plane have overlap. The targets with a constraint degree of 0 are screened out to form an unblocked set :

[0164]

[0165] where, represents the unblocked target set, which contains all targets with a constraint degree of 0, i.e. the target set that is not blocked by any other target and can be directly unloaded.

[0166] The unblocked set is sorted according to multiple sorting criteria to generate a priority sequence: first by Z coordinate (height) in descending order, then by X coordinate (depth) in ascending order, and finally by Y coordinate. The sorting function is defined as:

[0167]

[0168] After each target is unloaded, the system updates the constraint relationship graph in real time, removes the processed target and recalculates the constraint degree of the remaining targets to ensure dynamic optimization of the unloading sequence.

[0169] The above describes the multi-view fusion-based occluded target detection method in the embodiments of the present application, and the following describes the multi-view fusion-based occluded target detection device in the embodiments of the present application:

[0170] The multi-view data acquisition unit is used for synchronously acquiring color images and depth images of the occlusion target in the goods stacking scene from different views, and the cameras at different views maintain a fixed relative position relationship;

[0171] The target detection unit is used for target detection on the color images at each view, and outputs an initial detection result in combination with the depth images, the initial detection result at least including two-dimensional bounding box coordinates, depth information and confidence;

[0172] The coordinate conversion unit is used for calculating three-dimensional detection results at each view based on the initial detection result, the three-dimensional detection results at least including bounding box center coordinates and confidence;

[0173] The multi-view fusion unit is used for calculating center distances of the three-dimensional detection results at each view, and when the distance is less than a preset threshold, the multi-view detection results of the same goods are identified; and the fusion detection result of the occlusion target is obtained based on the bounding box center coordinates and confidence of the multi-view detection results of the same goods.

[0174] The electronic device can be different in configuration or performance, and can include one or more processors (central processing units, CPUs) (for example, one or more processors) and a memory, and one or more storage media (for example, one or more mass storage devices) for storing application programs or data. The memory and the storage medium can be temporary storage or persistent storage. The programs stored in the storage medium can include one or more modules, and each module can include a series of instruction operations in the electronic device. Further, the processor can be configured to communicate with the storage medium and execute the series of instruction operations in the storage medium on the electronic device.

[0175] The electronic device can further include one or more power supplies, one or more wired or wireless network interfaces, one or more input / output interfaces, and / or one or more operating systems, such as Windows Serve, Mac OS X, Unix, Linux, FreeBSD, etc. Those skilled in the art can understand that the structure of the electronic device in the embodiment does not constitute a limitation based on the electronic device, and can include more or fewer components, or combine certain components, or different component arrangements.

[0176] The electronic device provided by the embodiments of the present application can have a large difference due to different configurations or performances, and can include one or more processors (central processing units, CPU) (for example, one or more processors) and a memory, one or more storage media (for example, one or more mass storage devices) for storing application programs or data. The memory and the storage media can be temporary storage or persistent storage. The programs stored in the storage media can include one or more modules, and each module can include a series of instruction operations in the electronic device. Furthermore, the processor can be configured to communicate with the storage media and execute the series of instruction operations in the storage media on the electronic device.

[0177] The electronic device can further include one or more power supplies, one or more wired or wireless network interfaces, one or more input / output interfaces, and / or one or more operating systems, such as Windows Serve, Mac OS X, Unix, Linux, FreeBSD, and the like. Those skilled in the art can understand that the structure of the electronic device does not constitute a limitation based on the electronic device, and can include more or fewer components than the foregoing, or combine certain components, or different component arrangements.

[0178] The present application also provides a computer readable storage medium, which can be a non-volatile computer readable storage medium or a volatile computer readable storage medium, and the computer readable storage medium stores instructions, and when the instructions are executed on a computer, the computer executes the steps of the foregoing method.

[0179] Those skilled in the art can clearly understand that, for the convenience and brevity of the description, the specific working process of the system or device, unit described above can refer to the corresponding process in the foregoing method embodiments, which will not be described here.

[0180] The integrated unit, if implemented in the form of a software function unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application or the entire or part of the technical solutions that essentially contribute to the prior art can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various media that can store program codes.

[0181] The above description and the above embodiments are only used to illustrate the technical solutions of the present application, but not to limit them. Although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacements for some technical features. These modifications or replacements do not make the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for detecting an occluded object based on multi-view fusion, characterized in that, The method comprises the following steps: synchronously collecting color images and depth images of occluded targets in a cargo stacking scene from different perspectives, the cameras at different perspectives maintaining a fixed relative positional relationship; performing target detection on the color images at each perspective, combining the depth images, and outputting initial detection results, the initial detection results at least including two-dimensional bounding box coordinates, depth information, and confidence; based on the initial detection results, calculating three-dimensional detection results at each perspective, the three-dimensional detection results at least including bounding box center coordinates and confidence; The center distance of the three-dimensional detection result of each perspective is calculated, and when the distance is less than a preset threshold, multi-perspective detection results of the same goods are identified, and the center distance is The calculation formula is: Wherein, the target center coordinate detected by the camera i is , and the target center coordinate detected by the camera j is ; The bounding box center coordinates and the confidence of the multi-view detection result of the same goods are used to obtain a fusion detection result of the occluded target, and the fusion detection result at least includes a final center coordinate after weighted fusion ; wherein, is a center coordinate of the detection result, is a dynamic weight, is a confidence score, is a depth quality factor, is a geometric consistency factor; is a robust depth standard deviation of the detection result, is a depth quality threshold parameter; is a center coordinate of the reference box with the highest confidence, is a geometric consistency threshold parameter, denotes the Euclidean distance. 2.The occluded object detection method based on multi-view fusion according to claim 1, wherein, the depth information of the fused detection results is obtained through the following steps: calculating center point coordinates for each detection bounding box and obtaining depth values within the bounding box through random sampling; generating a plurality of random coordinate points within the bounding box through random sampling, obtaining corresponding depth values to form a sampling set; filtering out invalid points with a depth value of 0, and removing outliers from the remaining valid depth values through quantile statistical methods; filtering the bounding box through determination of the robust depth standard deviation of the sampling set and area rationality test to obtain the depth information. 3.The method of claim 2, wherein, The calculation formula of the robust depth standard deviation is: wherein is a single depth value in the filtered depth value set, is an arithmetic mean of the filtered depth values, is a number of elements of the filtered depth value set, is a robust depth standard deviation; When or the filtered sample set corresponds to a bounding box, where is a threshold for the depth standard deviation, is a threshold for the maximum depth.

4. The occluded object detection method based on multi-view fusion according to claim 2, characterized in that, The determination rule of the area rationality test is: wherein , is the left lower corner coordinate of the fusion bounding box in the base coordinate system, is the right upper corner coordinate, is the projection area of the fusion bounding box in the YZ plane; When an unusually large box is considered abnormal and filtered, where is the maximum area threshold.

5. The occluded object detection method based on multi-view fusion according to claim 1, wherein, The fusion detection result further includes a comprehensive confidence : wherein, is a multi-view reward coefficient, is the number of views participating in the fusion.

6. The occluded object detection method based on multi-view fusion according to claim 1, wherein, The method further comprises a timing stabilization mechanism: Let the current frame fusion detection result be , and the history frame buffer result be ; For each current frame target , calculate its IoU overlap with the historical frame target : Simultaneously calculating the relative area ratio To determine the target scale consistency: wherein denotes the intersection area of two objects, denotes the union area of two objects, , denotes the area of an object ; When and , is an IoU overlap threshold value, is a relative area ratio threshold value; consecutive observations identified as the same target are associated and attribute fusion is performed.

7. The occluded object detection method based on multi-view fusion according to claim 6, characterized in that, The historical frame buffer result is maintained through the following method: updating the position coordinates, confidence, and geometric properties of the target through weighted averaging, establishing a sliding window mechanism to maintain a fixed-length history sequence, and the weight is designed as: wherein α is a time decay factor, T is the current time, and t is the historical time.

8. A system for detecting an occluded object based on multi-view fusion, the system comprising: The system comprises: a multi-perspective data acquisition unit configured to synchronously collect color images and depth images of occluded targets in a cargo stacking scene from different perspectives, the cameras at different perspectives maintaining a fixed relative positional relationship; a target detection unit configured to perform target detection on the color images at each perspective, combine the depth images, and output initial detection results, the initial detection results at least including two-dimensional bounding box coordinates, depth information, and confidence; a coordinate conversion unit configured to calculate three-dimensional detection results at each perspective based on the initial detection results, the three-dimensional detection results at least including bounding box center coordinates and confidence; A multi-view fusion unit is used to calculate the center distance of the 3D detection results from each viewpoint. When the distance is less than a preset threshold, it is identified as a multi-view detection result of the same goods; the center distance... The calculation formula is: wherein the target center coordinate detected by the camera i is , and the target center coordinate detected by the camera j is ; The bounding box center coordinates and the confidence of the multi-view detection result of the same goods are used to obtain a fusion detection result of the occluded target, and the fusion detection result at least includes a final center coordinate after weighted fusion ; wherein, is a center coordinate of the detection result, is a dynamic weight, is a confidence score, is a depth quality factor, is a geometric consistency factor; is a robust depth standard deviation of the detection result, is a depth quality threshold parameter; is a center coordinate of the reference box with the highest confidence, is a geometric consistency threshold parameter, denotes the Euclidean distance. 9.An electronic device comprising a memory and at least one processor, the memory having instructions stored therein; the at least one processor invoking the instructions in the memory to cause the electronic device to perform the steps of the multi-perspective fusion-based occluded target detection method according to any one of claims 1-7.

10. A computer-readable storage medium having stored thereon instructions, the computer-readable storage medium comprising: the instructions, when executed by the processor, implement the steps of the multi-perspective fusion-based occluded target detection method according to any one of claims 1-7.

Images