Air cargo compartment loading target depth attribution method based on subject area screening and binocular vision main occupancy layer determination

Abstract

The application provides an aviation cargo compartment loading target depth attribution method based on subject area screening and binocular vision main occupancy layer determination, and belongs to the technical field of aviation cargo compartment loading visual perception and three-dimensional positioning, and comprises the following steps: obtaining a target category, a target detection frame and a local response distribution by using YOLOv10 in a left view of the aviation cargo compartment determined by a double target, mapping to generate a subject confidence map, and screening a candidate subject area in combination with a boundary suppression zone; then, establishing pixel correspondence according to a polar line constraint in the candidate subject area, obtaining occupancy evidence in combination with parallax consistency and left-right correspondence stability, determining a main occupancy layer according to a subject confidence map overlap degree and a target category scale constraint, and obtaining a subject in-point set and a subject representative depth; finally, outputting a subject depth attribution result and an operation position parameter by means of three-dimensional reverse projection space target instances. The application can effectively suppress non-subject area interference, and improve the accuracy, stability and operation positioning consistency of aviation cargo compartment loading target depth attribution.

Description

Technical Field

[0001] This invention relates to the field of visual perception and three-dimensional positioning technology for cargo hold loading, and in particular to a method for assigning the depth of cargo hold loading targets based on subject area screening and binocular vision main occupancy layer determination. Background Technology

[0002] Aircraft cargo hold loading operations are characterized by confined spaces, complex lighting, and mutual obstruction between cargo, containers, and surrounding structures, with boundaries tightly pressed against the hold walls. Before performing actions such as grasping, pushing, and positioning verification, robots or automated equipment need to obtain reliable 3D positions and operating depth ranges of targets within the hold. Binocular vision establishes epipolar constraints through left and right camera calibration, calculates depth from pixel disparity, and combines it with 2D target detection results to achieve target localization. However, in cargo hold scenarios, target appearance textures are uniform, edges are similar to gaps / loading gaps in the hold walls, and adjacent target contours easily fall within the same field of view, making depth estimation susceptible to interference from non-target areas. This affects the consistency of subsequent 3D instance construction and operational parameter output.

[0003] In existing technologies, common solutions include: using deep learning-based 2D object detection networks (such as the YOLO series) to output object categories and detection boxes on monocular images, and then performing binocular stereo matching within the detection box region or calculating depth from dense disparity maps / point clouds; or using instance segmentation networks to first obtain object masks, and then converging depth within the masks to generate 3D object information; there are also solutions that use convolutional networks to perform disparity regression to obtain a more continuous depth distribution within the target region, and using the target center point or statistical depth as a reference.

[0004] The above-mentioned solutions typically have the following problems in aviation cargo hold loading scenarios: gaps in the bulkhead, loading gaps, and edges of adjacent targets are easily mixed into the detection frame or coarse mask, resulting in multiple layers of aliasing in the parallax distribution; binocular correspondence is unstable at weak textures and occluded boundaries, and depth statistics are easily influenced by a small number of incorrect matches; there is a lack of a mechanism to determine which depth layer the target subject should belong to, making the output representative depth inconsistent with the actual workable subject space range, thus affecting the reliability of 3D instance and work position parameters. Summary of the Invention

[0005] In view of this, the present invention provides a method for attributing the loading target depth in an aircraft cargo hold based on subject region screening and binocular vision main occupancy layer determination, in order to solve the problems existing in the prior art.

[0006] To address the aforementioned problems, this invention provides a method for attributing the depth of cargo hold loading targets in aviation cargo compartments based on subject region filtering and binocular vision main occupancy layer determination, comprising:

[0007] S1: Obtain the left and right views of the aircraft cargo hold after bi-target calibration. Run YOLOv10 in the left view to obtain target semantic boxes containing target categories, target detection boxes, and local response distributions.

[0008] S2: Generate a boundary suppression band within the target detection box based on the boundary distance according to the target semantic box, and map the local response distribution to the subject confidence map; under the constraint of the target category, perform continuous region filtering on the subject confidence map, and determine the set of high confidence entity pixels located outside the boundary suppression band, far away from the cargo hold wall gap, loading gap and adjacent target contour, and satisfying the connectivity area condition as candidate subject regions, and determine the candidate subject regions as the binocular matching range;

[0009] S3: Establish pixel correspondence between left and right views within the candidate subject area according to epipolar constraints, calculate disparity consistency and left-right correspondence stability of corresponding pixels, and perform weighted aggregation and connectivity merging of similar disparity pixels according to the subject confidence map to form layer occupancy evidence; filter layer occupancy evidence according to left-right consistency threshold and connectivity area threshold, and determine the layer occupancy evidence that meets the binocular correspondence stability and continuous occupancy conditions as valid layer occupancy evidence.

[0010] S4: Calculate the overlap of each depth layer based on the valid layer occupancy evidence and the subject confidence map, and perform attribution determination on multiple depth layers that meet the threshold under the target category scale constraint to determine the main occupancy layer and output the subject interior point set and the subject representative depth;

[0011] S5: Perform 3D back projection based on the set of points inside the subject and the representative depth of the subject to form a spatial target instance corresponding to the target semantic box;

[0012] S6: Generate the main depth attribution result of the air cargo hold loading target based on the space target instance, and output the operation position parameters.

[0013] Preferably, S1 specifically includes:

[0014] The left and right views of the aircraft cargo hold, determined by the bi-target, are input into the shared weight backbone network of YOLOv10. The Stem layer and C2f layer are used to extract convolutional features from the left and right views of the aircraft cargo hold to create multi-scale feature maps of the left and right views.

[0015] The left-view multi-scale feature map and the right-view multi-scale feature map are input into the FPN-PAN feature fusion layer. The left-view multi-scale feature map and the right-view multi-scale feature map are aligned and fused to form the left-view fused feature map and the right-view fused feature map.

[0016] Input the left view fused feature map into the classification regression head, perform target category prediction and target detection box regression on the left view fused feature map, and obtain the target category and target detection box;

[0017] The left view fusion feature map is input into the local response distribution branch, and region alignment sampling is performed on the left view fusion feature map according to the target detection box to obtain the local response distribution;

[0018] The target category, target detection box, and local response distribution are associated to form a target semantic box.

[0019] Further optimization, S2 specifically includes:

[0020] Receive the target detection box and local response distribution in the target semantic box, perform region alignment sampling on the left view fusion feature map according to the target detection box, and resample the local response distribution to the sampling grid consistent with the target detection box to form a response alignment map;

[0021] The response alignment map is input to the response mapping layer, which is composed of sequentially connected two-dimensional convolutional neurons, and the response alignment map is processed by neighborhood aggregation, channel compression, and confidence mapping to obtain the main confidence map. The confidence value of each pixel in the main confidence map corresponds to the judgment value of the pixel at the same position in the target detection box belonging to the set of high-confidence entity pixels.

[0022] The boundary distance map is calculated based on the boundary distance from each pixel within the target detection box to the boundary of the target detection box. The boundary distance threshold and boundary bandwidth threshold are determined in combination with the target category constraint. The boundary distance map is then thresholded to obtain the boundary suppression band.

[0023] The main confidence map is aligned with the boundary suppression band. High confidence pixels in the main confidence map are selected under the constraint of the main confidence threshold. High confidence pixels that fall into the boundary suppression band are marked as edge-fitting pixels to form the main candidate map.

[0024] Continuous region filtering is performed on the candidate graph of the main body. The connected area, boundary overlap ratio, boundary continuity length and internal break length of each connected region are calculated. Connected regions that meet the connected area condition, the boundary overlap ratio is lower than the overlap threshold and the internal break length is lower than the break threshold are identified as solid connected regions.

[0025] Based on the target category constraints, the contour of the entity connected region is determined. The extension direction and strip width of the entity connected region along the boundary of the target detection box are calculated. The entity connected region whose boundary continuous length reaches the edge length threshold and whose strip width is lower than the strip width threshold is determined as the cargo hold wall gap, loading gap or adjacent target contour corresponding region, and is screened out from the entity connected region to obtain the candidate subject region.

[0026] Candidate subject regions are mapped to corresponding regions in the left-view fusion feature map and the right-view fusion feature map, and the pixel index set of the candidate subject regions is used as the binocular matching range of the binocular main occupancy layer extraction network.

[0027] Further optimization, S3 specifically includes:

[0028] The candidate subject region is mapped to the left view fusion feature map and the right view fusion feature map. Under the constraint of the target semantic box, region alignment sampling is performed on the left view fusion feature map and the right view fusion feature map to obtain the left region feature and the right region feature.

[0029] The left-side region features, right-side region features, subject confidence map, and candidate subject regions are fused along the channel direction to form fused features, and the candidate disparity search interval is determined on the fused features based on the binocular matching range.

[0030] Based on the candidate disparity search interval and epipolar constraints, disparity matching cost calculation is performed on the fused features to form a disparity cost body;

[0031] The disparity cost volume is input into an occupancy aggregation unit composed of sequentially connected 3D convolutional neurons. Occupancy aggregation is performed on the disparity cost volume along the disparity direction and the spatial direction to obtain the layer occupancy response. The pixel correspondence between the features of the left region and the features of the right region is established based on the layer occupancy response.

[0032] Based on pixel correspondence, disparity consistency and left-right correspondence stability are calculated, and left-right consistency score is calculated based on disparity consistency and left-right correspondence stability to obtain effective pixel correspondence;

[0033] The pixel confidence values ​​in the subject confidence map are mapped to the disparity layers corresponding to the effective pixels. Weighted aggregation and connectivity merging are performed on the effective pixels that are adjacent in disparity and spatially connected to form layer occupancy evidence. The connected area value and occupancy continuity length corresponding to the layer occupancy evidence are calculated.

[0034] The layer occupancy evidence, left-right consistency score, connected area value, and occupancy continuity length inputs are sequentially connected by fully connected neurons to form a layer decision head. The layer occupancy evidence that meets the binocular correspondence stability and continuous occupancy conditions is selected as valid layer occupancy evidence based on the left-right consistency threshold and connected area threshold.

[0035] Further optimization, S4 specifically includes:

[0036] Receive valid layer occupancy evidence, subject confidence map, candidate subject region and target semantic box. Expand the valid layer occupancy evidence into the corresponding position of the target detection box in the sampling grid of the candidate subject region according to the depth layer number to form multiple depth layer pixel regions. Calculate the number of pixel supports, pixel support density, edge distribution ratio and spatial connectivity distribution of each depth layer pixel region to obtain the depth layer support value.

[0037] Based on the subject confidence map, position overlap calculation is performed on the pixel regions of each depth layer. High confidence pixels that reach the subject confidence threshold in the subject confidence map are extracted. The coverage ratio, subject confidence cumulative value, center coverage ratio and boundary overlap ratio of high confidence pixels in the pixel regions of each depth layer are statistically analyzed. The above indicators are then fused to obtain the overlap degree of each depth layer.

[0038] Based on the target category in the target semantic box and the target detection box, a target category scale constraint is constructed. Combined with the dual target calibration parameters, the pixel area of ​​each depth layer is converted into the corresponding entity size. The matching degree between the corresponding entity size and the length relationship, width relationship and height range in the target category scale constraint is calculated to obtain the scale consistency score. The contact ratio between the pixel area of ​​each depth layer and the boundary of the target detection box is calculated to obtain the boundary contact value.

[0039] The layer determination head, composed of fully connected neurons, is used to input the overlap, scale consistency score, boundary contact value, depth layer support value, left-right consistency score, connected area value, and continuous length of the effective layer occupancy evidence for each depth layer. This head is then used to perform a classification determination on each depth layer, resulting in a classification determination score for each depth layer.

[0040] Depth layers whose attribution scores reach the attribution threshold, whose overlap of each depth layer reaches the overlap threshold, whose scale consistency scores reach the scale threshold, and whose boundary contact values ​​are lower than the contact threshold are identified as candidate attribution layers. Candidate attribution layers whose number of pixel supports is lower than the number threshold or whose spatial connectivity distribution does not meet the connectivity conditions are filtered out based on the depth layer support values.

[0041] Perform inter-layer comparison on the candidate layers of attribution, determine the candidate layer with the highest attribution score as the main occupancy layer, and filter the pixels in the main occupancy layer that reach the subject confidence threshold, meet the left-right consistency threshold, are located in the candidate subject area and are in the connected region to obtain the set of subject in-points.

[0042] The depth of the subject is calculated based on the layer position of the subject's interior point set in the main occupied layer, the depth distribution center value corresponding to the subject's interior point set, and the bi-objective positioning parameters.

[0043] Further optimization, S5 specifically includes:

[0044] The receiver layer's decision head outputs the set of in-body points, the representative depth of the subject, and the target semantic box. It extracts the pixel positions within the target detection box for each pixel in the set of in-body points and combines the pixel positions with the representative depth of the subject according to the pixel correspondence to construct a back-projected pixel set.

[0045] Based on the camera imaging parameters obtained through binocular calibration, the positions of each pixel in the back-projection pixel set are mapped to back-projection rays, and the positions of three-dimensional points are determined on the back-projection rays according to the representative depth of the subject, thus obtaining the three-dimensional point set of the subject.

[0046] Spatial aggregation is performed on the main body's three-dimensional point set to calculate the spatial center position, spatial range, and depth distribution boundary of the main body's three-dimensional point set, thus forming the main body's spatial parameters;

[0047] The spatial parameters of the subject are associated with the target category, target detection box, and subject representation depth in the target semantic box to construct a spatial target instance corresponding to the target semantic box.

[0048] Further optimization, S6 specifically includes:

[0049] Receive spatial target instances and extract target category, target detection box, subject representative depth, and subject spatial parameters from the spatial target instances;

[0050] The projection deviation is calculated based on the projection relationship between the target detection box and the spatial center position. When the projection deviation meets the projection threshold, the target category, target detection box, subject representative depth and spatial center position are bound together to form the subject depth attribution result.

[0051] The coordinates of the work location are determined based on the spatial center position in the main spatial parameters, and the work depth range and work coverage range are determined based on the spatial range and depth distribution boundary, thus obtaining the work location parameters;

[0052] The main body depth attribution results are correlated with the operation location parameters to output the main body depth attribution results and specific parameters of the loading target in the air cargo hold.

[0053] The present invention provides a method for attributing the depth of loading targets in air cargo holds based on subject region filtering and binocular vision-based primary occupancy layer determination. This method can suppress non-subject interference such as gaps, gaps, and adjacent contours within a two-dimensional detection frame. Based on stable binocular correspondence, it determines the primary occupancy layer of the target subject from multiple depth layers, thereby providing the representative depth of the subject and three-dimensional positional parameters suitable for loading operations. Specifically:

[0054] 1. This invention constructs a subject confidence map based on the local response distribution mapping output by the YOLOv10 detection head, and combines boundary suppression bands and connected morphology constraints to screen candidate subject regions. It no longer simply uses the entire detection box or a simple segmentation mask as the depth statistical range, but can model the target subject within the detection box as quantifiable and transferable confidence evidence. The boundary suppression bands effectively eliminate edge interference caused by bulkhead gaps, loading gaps, and adjacent target contours, greatly reducing the participation of non-target pixels in subsequent binocular matching and depth calculation.

[0055] 2. This invention performs pixel matching only within the selected candidate subject area based on epipolar constraints. At the same time, it introduces a dual verification mechanism of disparity consistency and left-right correspondence stability to filter effective pixel correspondences. This is different from the traditional method of dense matching of the entire image or detection box. It significantly reduces the matching search space and reduces the interference of incorrect matching points at weak textures and occluded boundaries on the depth statistics results.

[0056] 3. To address the pain point of multiple layers of parallax aliasing within the detection frame and the inability to distinguish the true depth of the target, this invention first aggregates and generates layer occupancy evidence for each depth layer and filters the effective layers. Then, it combines the subject confidence map to calculate the depth layer overlap. At the same time, it introduces target category scale constraints and boundary contact penalties to complete the determination of the main occupancy layer, establishes a complete multi-layer depth attribution discrimination logic, and solves the problem of unstable subject representative depth caused by traditional solutions that rely solely on the mean, median, and single connected component depth statistics. This ensures that the depth attribution results are consistent with the true spatial location of the cargo entity.

[0057] 4. This invention completes three-dimensional inverse projection based on the set of internal points of the main body corresponding to the main occupying layer and the representative depth of the main body, generating standardized spatial target instances, simultaneously calculating the spatial center, spatial range and depth boundary, and adding a two-dimensional-three-dimensional projection deviation verification step before outputting the operation position parameters. Unlike existing solutions that only output single-point depth or coarse point cloud center, this invention achieves consistent binding between two-dimensional detection results and three-dimensional entity spatial parameters, improving the consistency and reliability of operation position parameter output when automated loading equipment grabs, pushes, and verifies positioning. It can adapt to complex operation scenarios such as narrow and obstructed air cargo holds with cargo close to the bulkhead, and stably outputs accurate three-dimensional positioning information that can be directly used for loading operations. Attached Figure Description

[0058] Figure 1 A flowchart of the method for attributing the depth of loading targets in an aviation cargo hold based on subject region filtering and binocular vision main occupancy layer determination provided by the present invention;

[0059] Figure 2 Left view of an aircraft cargo hold;

[0060] Figure 3 This is a schematic diagram showing the results of extracting the main body region based on the detection frame;

[0061] Figure 4 A schematic diagram illustrating the reconstruction of the three-dimensional spatial target and the output of operational parameters;

[0062] Figure 5 A diagram showing the comparison of the depth of the primary occupied layer. Figure 5 (a) is a depth attribution diagram without using the present invention. Figure 5 (b) is a schematic diagram of the main occupancy layer depth assignment using the present invention. Detailed Implementation

[0063] The present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0064] like Figure 1 As shown, this invention provides a method for attributing the depth of loading targets in an aircraft cargo hold based on subject region filtering and binocular vision main occupancy layer determination, including the following steps:

[0065] S1: The left and right views of the aircraft cargo hold, determined by dual-target mapping, are input into the shared weight backbone network of YOLOv10. Convolutional feature extraction is performed on the left and right views using Stem and C2f layers to create multi-scale feature maps for the left and right views. These feature maps are then input into the FPN-PAN feature fusion layer for alignment and fusion, resulting in a fused feature map for the left and right views. The fused feature map for the left view is input into the classification regression head for target category prediction and target detection box regression, yielding the target category and target detection box. The fused feature map for the left view is then input into the local response distribution branch, and region alignment sampling is performed on the fused feature map based on the target detection box to obtain the local response distribution. Finally, the target category, target detection box, and local response distribution are correlated to form the target semantic box.

[0066] Specifically, S1 includes the following steps:

[0067] S11: Will Defined as a left view of an aircraft cargo hold after bi-targeting, Defined as a right-hand view of an aircraft cargo hold after bi-targeting, Defined as a YOLOv10 main occupancy layer fusion network. This represents the network parameters of the shared weight backbone network, the FPN-PAN feature fusion layer, the classification and regression head, and the local response distribution branch;

[0068] S12: with To perform one forward inference operation on the input, the shared weight backbone network first... and Convolutional feature extraction is performed separately. The Stem layer completes the initial downsampling and channel unpacking. The fourth-level C2f layer extracts the target texture, occlusion edge and box contour information in order from shallow to deep. Since the left view and the right view share the same set of backbone network parameters, the left view features and the right view features adopt the same expression rule in channel semantics. When establishing pixel correspondence on the left view fusion feature map and the right view fusion feature map in step S3, the same source feature benchmark can be directly used.

[0069] S13: Will Defined as a multi-scale feature map of the left view, Defined as a multi-scale feature map of the right view; each and All features are generated by two-dimensional convolutional neurons connected in the order of convolution, normalization, and activation. After obtaining the multi-scale feature maps of the left and right views, these maps are input into the FPN-PAN feature fusion layer. The FPN-PAN feature fusion layer first upsamples the high-level semantic features and aligns them laterally with adjacent layers, then downsamples and backpropagates the fusion result, thus simultaneously preserving target category discrimination information and target edge localization information. Defined as the left view fusion feature map, Defined as the right view fusion feature map, the left view fusion feature map is used for target category prediction, target detection box regression and local response distribution extraction, and the right view fusion feature map is used for region alignment sampling in the binocular matching link;

[0070] S14: After obtaining the left view fusion feature map, input the left view fusion feature map into the classification regression head. The classification regression head outputs the class score and bounding box parameters of the candidate positions at three scales of the left view fusion feature map, and filters the valid candidate records according to the class threshold. Defined as the set of valid candidate records, where, Indicates the first Target categories, Indicates the first One target detection bounding box, The number of valid candidate records is represented by the target detection box expressed in the left view coordinate system. The boundary of the target detection box directly corresponds to the spatial coverage of the target in the left view. The target category and the target detection box participate in the target category constraint and boundary distance calculation in S2, respectively. Therefore, the target detection box output in S1 is no longer used as a simple detection result, but as a spatial index benchmark for constructing the link in the main confidence map.

[0071] S15: After obtaining the target category and target detection box, input the left view fused feature map into the local response distribution branch, and... Defined as the response map output by the local response distribution branch, which consists of two 3x3 convolutional layers and one 1x1 convolutional layer connected sequentially. The local response distribution branch extracts fine-grained location responses from the high-resolution layer in the fusion feature map of the left view. Maintain spatial correspondence with the coordinates of the left view, and then based on each target detection box. In response graph Perform region alignment sampling on the upper part, , Defined as a fixed sampling grid index, Defined as the first The local response distribution corresponding to each target detection box is obtained by region-aligned sampling based on the target detection box boundaries in the response map. A fixed grid is established on the top and the grid position is mapped to the target detection box coverage area, so that the local response distribution and the target detection box use the same spatial reference. Each sample value in the local response distribution corresponds to the response intensity of a local position inside the target detection box to the target category. The local response distribution is directly mapped to the main confidence map in step S2. S1 generates the local response distribution inside the YOLOv10 detection head instead of constructing an independent image mask after the target detection box is generated.

[0072] S16: After completing the generation of target category, target detection box, and local response distribution, Defined as the first Each target semantic box will Defined as a set of target semantic boxes, the target detection boxes within the target semantic boxes provide the target boundary range, the target category provides the target category constraint, and the local response distribution provides the local semantic response field inside the target detection boxes; after the target semantic boxes are formed, the local response distribution can directly enter the subject confidence map mapping process, the target detection boxes can directly enter the boundary distance calculation process, and the left view fusion feature map and the right view fusion feature map can directly enter the region alignment sampling process of the binocular main occupancy layer extraction network, thereby completing the data connection between step S1 and the candidate subject region determination link, the layer occupancy evidence formation link, and the main occupancy layer attribution determination link;

[0073] S2: Receive the target detection box and local response distribution in the target semantic box. Perform region alignment sampling on the fusion feature map of the left view according to the target detection box, and resample the local response distribution to the sampling grid consistent with the target detection box to form a response alignment map. Input the response alignment map into the response mapping layer set at the output of the YOLOv10 detection head and composed of sequentially connected two-dimensional convolutional neurons. Perform neighborhood aggregation, channel compression and confidence mapping on the response alignment map to obtain the main confidence map. The confidence value of each pixel in the main confidence map corresponds to the judgment value of the pixel at the same position in the target detection box belonging to the set of high-confidence entity pixels.

[0074] A boundary distance map is calculated based on the boundary distance from each pixel within the target detection box to the boundary of the target detection box. The boundary distance threshold and boundary bandwidth threshold are determined in conjunction with the target category constraint. The boundary distance map is then thresholded to obtain the boundary suppression band. The subject confidence map is aligned with the boundary suppression band. Under the subject confidence threshold constraint, high-confidence pixels in the subject confidence map are selected, and high-confidence pixels that fall within the boundary suppression band are marked as edge-fitting pixels to form the subject candidate map.

[0075] Continuous region filtering is performed on the candidate images of the main body. The connected area, boundary overlap ratio, boundary continuity length, and internal break length of each connected region are calculated. Connected regions that meet the connected area condition, boundary overlap ratio below the overlap threshold, and internal break length below the break threshold are identified as solid connected regions. Contour determination is performed on the solid connected regions according to the target category constraints. The extension direction and strip width of the solid connected regions along the boundary of the target detection box are calculated. Solid connected regions whose boundary continuity length reaches the edge length threshold and strip width is below the strip width threshold are identified as cargo hold wall gaps, loading gaps, or regions corresponding to adjacent target contours, and are filtered out from the solid connected regions to obtain candidate main body regions. The candidate main body regions are mapped to the corresponding regions of the left view fusion feature map and the right view fusion feature map, and the pixel index set of the candidate main body regions is used as the binocular matching range of the binocular main occupancy layer extraction network.

[0076] Specifically, S2 includes the following steps:

[0077] S21: For the target semantic box set Each target semantic box Processed sequentially, the high-resolution layer of the fused feature map of the left view is denoted as... In the target detection box Establish a fixed sampling grid within Based on the target detection box exist The process involves performing region-aligned sampling, using the grid generated by the region-aligned sampling as a unified index axis within the target detection bounding box, and then distributing the local response. Resampling to sampling grid Forming a response alignment map Response Alignment Map Each sampling point is associated with the target detection box. Internal fixed position correspondence, response alignment diagram This serves as a common spatial benchmark for constructing the subject confidence map, calculating boundary distances, and selecting candidate subject regions.

[0078] S22: Will Defined as a response mapping layer set at the output of the YOLOv10 sensor head. This represents the parameters of the response mapping layer, which consists of sequentially connected two-dimensional convolutional neurons. Each two-dimensional convolutional neuron processes the response alignment map in the order of convolution, normalization, and activation. The response mapping layer first aligns the response map. The system performs neighborhood aggregation, integrating semantic responses distributed across adjacent sampling points into a single location for judgment. Then, it performs channel compression and confidence mapping to obtain the main confidence map. Main confidence graph With the target detection box Shared sampling grid Main confidence graph Each pixel confidence value represents the determination value that the corresponding position inside the target detection box belongs to the set of high-confidence entity pixels. (Main confidence map) In step S3, the preceding input object, which serves as evidence of layer occupancy, is continuously passed;

[0079] S23: The supervision labels of the response mapping layer are generated by the annotation of the main body region within the target detection box. The loading target in the left view is manually annotated, or the instance segmentation result that has been manually verified is used as the main body region annotation. Then, based on the target detection box and the region alignment sampling grid, the main body region annotation is resampled to the corresponding position of the response alignment map to form a main body region supervision map consistent with the target detection box. The position of the target main body in the main body region supervision map is marked as a high confidence supervision position, and the position located in the wall gap, loading gap, adjacent target outline and background is marked as a non-subject supervision position. The response mapping layer is trained with the response alignment map as input and the main body region supervision map as the supervision target, so that the main body confidence map can stably represent the determination result of each position in the target detection box belonging to the set of high confidence entity pixels.

[0080] S24: Based on target category Query target category constraint parameter vector , Indicates the subject confidence threshold. Indicates the boundary distance threshold. Indicates the boundary bandwidth threshold. This represents the area threshold corresponding to the connectivity area condition. Indicates the overlap threshold. Indicates the fracture threshold. This indicates the threshold for the edge length. This indicates the strip width threshold, which is then applied to the sampling grid. Top of target detection box The shortest distance from each sampling point to the four boundaries is calculated to form a boundary distance map. When the boundary distance is less than When the boundary distance falls within the range of the core edge sampling point, the corresponding sampling point is marked as the core edge sampling point; when the boundary distance falls within the range of the core edge sampling point, the corresponding sampling point is marked as the core edge sampling point. and When the boundary bandwidth is limited, the corresponding sampling points are marked as in-band sampling points; the core edge sampling points and in-band sampling points together form the boundary suppression band. Boundary suppression zone With subject confidence plot Shared sampling grid No coordinate transformation needs to be performed again during the continuous region filtering stage:

[0081] S25: In the subject confidence diagram and boundary suppression zone After completing the positional alignment, perform a confidence plot on the main body. Perform subject confidence threshold filtering to obtain a set of high-confidence entity pixels. It will fall into the boundary suppression zone The set of high-confidence entity pixels within the boundary is marked as edge-fitting pixels, and will be located in the boundary suppression band. The set of high-confidence entity pixels outside the main body is labeled as candidate entity pixels, thereby generating the main body candidate image. Main candidate image Discrete states are used to record the category of each sampling point. Background sampling points are recorded as background states, candidate entity pixels as entity states, and edge-fitting pixels as edge-fitting states. Then, the main candidate image is processed. Connectivity is marked using the eight-neighbor connectivity rule to obtain a set of connected regions. For each connected region The connected area, boundary overlap ratio, boundary continuity length, and internal break length are calculated. The connected area is calculated based on the number of non-background sampling points; the boundary overlap ratio is calculated based on the proportion of pixels touching the edge in the connected region to the connected area; and the boundary continuity length is calculated based on the target detection bounding box. After projecting the outline of the connected region onto each of the four edges, the length of the longest continuous projection segment is counted. The length of the internal fracture is then scanned line by line according to the main extension direction of the connected region in the candidate image. Then, the longest gap segment is counted, and the boundary suppression zone is determined. The set of sampling points with a value of one is denoted as . Connecting regions The internal fracture length, boundary continuity length, and strip width are denoted as follows: , and Candidate subject area The following unified screening rules are used to determine the criteria:

[0082] ;

[0083] In the formula, Indicates the first Candidate subject regions corresponding to each target semantic box; This indicates that a union is performed on the preserved connected regions that meet the conditions; Represents the set of connected components The first in One connected region; This indicates that sampling points corresponding to the boundary suppression band are removed from the preserved connected regions; Indicates the boundary suppression zone The set of sampling points with a value of one; Indicates the candidate graph of the main body The set of connected regions obtained according to the eight-neighbor connectivity rule; Represents the sampling grid The sampling point index in the data; Represents the object detection box Internal fixed sampling grid; Represents the subject confidence graph At the sampling point Confidence value at; Indicates the subject confidence threshold; This indicates that the connected region is entirely composed of high-confidence sampling points that reach the subject confidence threshold; Indicates the number of sampling points; Represents connected regions The connected area; Indicates the area threshold; Indicates intersection; Represents connected regions The proportion of boundary overlap; Indicates the overlap threshold; Represents connected regions The internal fracture length; Indicates the fracture threshold; This indicates that the condition within the parentheses is not true. Represents connected regions The continuous length of the boundary; Indicates the edge length threshold; This indicates that both conditions are met simultaneously; Represents connected regions The width of the stripe; Indicates the strip width threshold; Indicates the target semantic box index; This represents the index of the connected region. According to this unified filtering rule, connected regions that meet the conditions of connected area, boundary overlap ratio, and internal break length, but do not meet the edge-attached strip contour condition, are retained. After retention, sampling points corresponding to the boundary suppression zone are removed to obtain the candidate main region. ;

[0084] S26: Select candidate subject regions In the sampling grid Convert to pixel index set According to the target detection box Feature map fusion to left view The coordinate mapping relationship will Map to the corresponding region of the fused feature map in the left view; according to the row correspondence determined by the dual-map, The row index is synchronously mapped to a high-resolution layer of the right view fused feature map. Corresponding regions: left view fused feature map corresponding region, right view fused feature map corresponding region, subject confidence map. and pixel index set The pixel index set is fed into the binocular main occupancy layer extraction network. As the binocular matching range, candidate subject region As the space constraint object in step S3;

[0085] S3: Map the candidate subject region to the left view fusion feature map and the right view fusion feature map. Under the constraint of the target semantic box, perform region alignment sampling on the left view fusion feature map and the right view fusion feature map to obtain the left region feature and the right region feature. Fuse the left region feature, the right region feature, the subject confidence map and the candidate subject region along the channel direction to form the fusion feature. Determine the candidate disparity search interval on the fusion feature according to the binocular matching range.

[0086] Based on the candidate disparity search interval and epipolar constraints, disparity matching cost calculation is performed on the fused features to form a disparity cost volume. The disparity cost volume is then input into an occupancy aggregation unit composed of sequentially connected 3D convolutional neurons. Occupancy aggregation is performed on the disparity cost volume along the disparity direction and spatial direction to obtain the layer occupancy response. Pixel correspondences between left and right region features are established based on the layer occupancy response. Disparity consistency and left-right correspondence stability are calculated based on the pixel correspondences, and left-right consistency scores are calculated based on the disparity consistency and left-right correspondence stability to obtain the effective pixel correspondences.

[0087] The pixel confidence values ​​in the subject confidence map are mapped to the disparity layers corresponding to the effective pixels. Weighted aggregation and connectivity merging are performed on the effective pixel correspondences that are disparate and spatially connected to form layer occupancy evidence. The connected area value and occupancy continuity length corresponding to the layer occupancy evidence are calculated. The layer occupancy evidence, left-right consistency score, connected area value and occupancy continuity length are input into the layer decision head composed of fully connected neurons connected sequentially. The layer occupancy evidence is filtered according to the left-right consistency threshold and the connected area threshold. The layer occupancy evidence that meets the binocular correspondence stability and continuity occupancy conditions is determined as valid layer occupancy evidence.

[0088] Specifically, S3 includes the following steps:

[0089] S31: For each target semantic box Process sequentially, and obtain candidate main body regions using S2. Pixel index set Subject confidence diagram Left view fusion feature map Feature map fused with right view As input, in the target detection box Under constraints, respectively for and Perform region alignment sampling to obtain the features of the left region. and right-side region features Candidate subject area Convert to sampling grid Aligned candidate subject region label map ,when At that time, candidate subject region labeling map Write a valid marker at the corresponding position; when At that time, candidate subject region labeling map Write an invalid flag at the corresponding position, and then... , , and The pieces are joined together along the channel direction to form a fusion feature. ;

[0090] S32: Based on the epipolar constraints obtained through bi-target calibration, perform a matching search only within the right region that shares the same row index as the left sampling point, and based on the target detection box... Candidate disparity search intervals are determined by the lateral boundary differences of the corresponding areas in the left and right views. ,in, Indicates the lower bound of the candidate disparity search interval. This represents the upper bound of the candidate disparity search interval, for each and each Perform disparity-by-disparity matching cost calculation, and arrange the right-side region features on the same row by disparity layer. Perform a lateral displacement, then compare the current positions of the right-side region features with those of the left-side region features after the displacement. Combine the left-side region feature values, right-side region feature values, corresponding position feature differences, subject confidence map values, and candidate subject region labels into a matching cost description, forming the disparity cost body. Parallax cost The input occupancy aggregation unit, which consists of sequentially connected 3D convolutional neurons, performs occupancy aggregation synchronously along the disparity and spatial directions to obtain the layer occupancy response. For each The disparity layer with the largest response is selected as the candidate corresponding layer, denoted as . And based on this, establish a set of pixel correspondences. ;

[0091] S33: After obtaining the set of corresponding pixels Then, for each left sampling point Create a local window ,in, Represents the window radius, and denots disparity consistency as... , According to local window The candidate corresponding layer deviation of adjacent pixels within the same layer is determined, and the stability of the left and right correspondence is denoted as... , The reverse correspondence verification is performed from the right sampling point to the left sampling point to determine the result, and then... and Merge into left-right consistency scores And the left-right consistency threshold is denoted as , retain satisfaction The pixel correspondence is used to obtain the set of valid pixel correspondences. Set the effective pixels to Write the marker map corresponding to the valid pixels ,when The corresponding pixel belongs to the set of valid pixel correspondences. hour, Record it as 1 if it is not 1, otherwise record it as 0. For each Constructing a spatial neighborhood index set Spatial neighborhood index set From sampling points The surrounding first-order eight-neighbor sampling points are composed of each , will the The cumulative subject confidence value of each disparity layer is denoted as: And calculate according to the following rules:

[0092] ;

[0093] In the formula, Indicates the first A target semantic bounding box in the disparity layer The cumulative confidence value of the subject;

[0094] Indicates the sampling grid All sampling points within the range are accumulated;

[0095] Indicates the index of the current sampling point;

[0096] This represents a fixed sampling grid obtained by region-aligned sampling;

[0097] Presentation layer occupies response In the parallax layer and sampling points The response value at the location;

[0098] Represents the subject confidence graph At the sampling point Pixel confidence value at;

[0099] Indicates sampling point Parallax consistency;

[0100] Indicates sampling point The left and right sides correspond to stability;

[0101] This indicates that the valid pixels correspond to the marker map at the sampling points. The marker value at the location;

[0102] Indicates sampling point The candidate corresponding layer;

[0103] Indicates sampling point Spatial neighborhood index set;

[0104] Represents spatial neighborhood index set The number of sampling points within;

[0105] Represents spatial neighborhood index set The neighborhood sampling point index;

[0106] This represents the tag value corresponding to the valid pixel at the neighboring sampling point;

[0107] A value of 1 is used if the candidate corresponding layer of a neighboring sampling point is consistent with the current disparity layer; otherwise, a value of 0 is used.

[0108] Indicates the candidate disparity search interval The current disparity layer index in the data.

[0109] The neighborhood term in parentheses incorporates the spatial connectivity support of the same disparity layer into the calculation process of the main confidence cumulative value.

[0110] S34: In obtaining After that, it will satisfy And in the spatial neighborhood index set Effective pixels that maintain connectivity are grouped into subsets. This forms a layer that occupies the evidence set. Each layer occupies evidence Record the effective pixel set and subject confidence accumulation value of this disparity layer. Connectivity area value and occupying a continuous length Among them, the connected area value Based on the number of connected supporting pixels after merging, the length occupied is... Based on the longest continuous projection length of the connected supporting pixels in the main extension direction, the layer determination head is recorded as follows. , The layer decision header parameters represent the cumulative confidence value of the subject of the evidence occupied by each layer. The corresponding mean score for left-right consistency and the connected area value. and occupying a continuous length Concatenate into layer input vectors Then input the layer judgment header The execution layer filtering will satisfy the left-right consistency threshold. And the connected area value is not less than The layer occupancy evidence was determined to be valid layer occupancy evidence. The effective layer occupies evidence As the input object for calculating the overlap of each depth layer and determining the ownership of the main occupied layer in step S4;

[0111] S4: Receive valid layer occupancy evidence, subject confidence map, candidate subject region and target semantic box, expand the valid layer occupancy evidence into the corresponding position of the target detection box in the sampling grid of the candidate subject region according to the depth layer number, form multiple depth layer pixel regions, and count the number of pixel supports, pixel support density, edge distribution ratio and spatial connectivity distribution of each depth layer pixel region to obtain the depth layer support value.

[0112] Based on the subject confidence map, positional overlap calculations are performed on pixel regions of each depth layer. High-confidence pixels that reach the subject confidence threshold in the subject confidence map are extracted. The coverage ratio, cumulative subject confidence value, center coverage ratio, and boundary overlap ratio of high-confidence pixels in pixel regions of each depth layer are statistically analyzed. The above indicators are then fused to obtain the overlap degree of each depth layer. Target category scale constraints are constructed based on the target category and target detection box in the target semantic box. Combined with the bi-target calibration parameters, pixel regions of each depth layer are converted into corresponding entity sizes. The matching degree between the corresponding entity size and the length, width, and height range in the target category scale constraints is calculated to obtain the scale consistency score. Finally, the contact ratio between pixel regions of each depth layer and the boundary of the target detection box is calculated to obtain the boundary contact value.

[0113] The layer decision head, composed of fully connected neurons, is used to input the overlap of each depth layer, scale consistency score, boundary contact value, depth layer support value, left-right consistency score, connected area value, and continuous length of occupied area into the effective layer occupancy evidence. This head is used to perform a classification decision on each depth layer, resulting in a classification decision score for each depth layer. Depth layers whose classification decision scores reach the classification threshold, whose overlap of each depth layer reaches the overlap threshold, whose scale consistency score reaches the scale threshold, and whose boundary contact value is lower than the contact threshold are identified as candidate layers for classification. Candidate layers for classification are then filtered out based on the depth layer support value if the number of pixel supports is lower than the number threshold or if the spatial connectivity distribution does not meet the connectivity conditions.

[0114] Inter-layer comparison is performed on the candidate layers of attribution, and the candidate layer with the highest attribution score is determined as the main occupancy layer. Within the main occupancy layer, pixels that meet the subject confidence threshold, satisfy the left-right consistency threshold, are located in the candidate subject region and are in the connected region are filtered to obtain the subject intrapoint set. The subject depth is calculated based on the layer position of the subject intrapoint set in the main occupancy layer, the depth distribution center value corresponding to the subject intrapoint set, and the bi-target positioning parameters.

[0115] Specifically, S4 includes the following steps:

[0116] S41: For each target semantic box Process sequentially, using the valid layer occupation evidence output in step S3. Subject confidence diagram and candidate subject area As input, each valid layer occupying evidence in the valid layer occupying evidence The corresponding depth layer number is denoted as Number the depth layers In the sampling grid Expand the corresponding pixel set within the target detection box. The corresponding positions form a set of deep layer pixel regions. For each depth layer pixel region Statistical pixel support count Pixel support density Marginal distribution ratio and spatial connectivity distribution Number of pixels supported Pixel support density based on the number of effective sampling points The edge distribution ratio is calculated based on the ratio of the number of valid sampling points to the area of ​​the smallest enclosed region. Based on the percentage of valid sampling points located within the edge band of the target detection box, spatial connectivity distribution The maximum connected component percentage and the number of connected components after the eight-neighbor connectivity label are jointly determined. This is denoted as the depth layer support value.

[0117] S42: Based on the subject confidence diagram Perform position overlap calculation and then perform the subject confidence map. The subject confidence threshold is met. The set of sampling points is denoted as , target detection box The distance from the inside to the boundary is no greater than the boundary bandwidth threshold. The set of sampling points is denoted as the boundary region. , target detection box Inner boundary area The set of sampling points outside the central region is denoted as the central region. When fusing the coverage ratio of high-confidence pixels, the cumulative confidence value of the main body, the coverage ratio of the center, and the overlap ratio of the boundaries, the overlap ratio of each depth layer is calculated according to the following rules. :

[0118] ;

[0119] In the formula, Indicates the first The first target semantic box The degree of overlap between depth layers in each depth layer pixel region; Represents the subject confidence graph The subject confidence threshold is reached. The set of high-confidence pixels; Indicates the first Each depth layer pixel region; Represents the object detection box The central area within; Represents the object detection box The boundary area within; Represents the subject confidence graph At the sampling point Pixel confidence value at; Represents the sampling grid The index of sampling points within; Indicates the number of sampling points within the set; Indicates intersection; This indicates that the summation is performed on the sample points that meet the conditions; This indicates taking the larger value within the parentheses; Indicates the normalized lower bound; Indicates the target semantic box index; The depth layer number is indicated. According to the calculation rule, the first item corresponds to the coverage ratio of high-confidence pixels, the second item corresponds to the normalized result of the cumulative confidence value of the subject, the third item corresponds to the coverage ratio of the center, and the fourth item corresponds to the inverse constraint of the boundary overlap ratio.

[0120] S43: Obtain the overlap of each depth layer Then, based on the target semantic box Target categories and object detection bounding box Construct target category scale constraints Target category scale constraint It consists of the length relationship, width relationship, height range, and allowable deviation range corresponding to the current target category, and combined with the dual-target positioning parameters, it divides each depth layer pixel area The horizontal and vertical spans are converted into corresponding entity dimensions, and the corresponding entity dimensions are then compared with the target category scale constraints. Item-by-item comparison yields a scale consistency score. At the same time, for each depth layer pixel region Statistical and object detection boxes The number of sampling points where the four edges meet accounts for a certain percentage of the pixel support count. The proportion is used to obtain the boundary contact value. ;

[0121] S44: The decision vector corresponding to each depth layer is then denoted as:

[0122] ;

[0123] in, Indicates the evidence of effective layer occupancy The left-right consistency score recorded in the middle. Indicates the evidence of effective layer occupancy The connected area value recorded in the middle, Indicates evidence of effective layer occupancy The records in the middle occupy a continuous length, and the decision vector will be determined. The input is a layer decision head consisting of sequentially connected fully connected neurons, and the output is the assignment score for each layer at each depth. , will satisfy Not lower than the attribution threshold , Not lower than the overlap threshold , Not lower than the scale threshold , Below the contact threshold The depth layer is determined as the candidate layer for belonging, and based on the depth layer support value... Perform secondary filtering; when the number of pixels supports Below the quantity threshold When the corresponding candidate layer is selected, it will be eliminated; when the spatial connectivity is distributed... When the proportion of the largest connected component is lower than the connectivity threshold or the number of connected components is higher than the component threshold, the corresponding candidate layer will be filtered out.

[0124] S45: Perform inter-layer comparison on the retained candidate layers and assign the classification score. The largest candidate layer is determined as the primary occupied layer, and the corresponding depth layer pixel region is denoted as... When the attribution determination scores of multiple attribution candidate layers When the values ​​are the same, priority is given to selecting the overlap between different depth layers. Larger candidate layers; overlap between depth layers If the scores are still the same, continue to select the scale consistency score. After determining the primary occupied layer and the larger candidate layer, perform eight-neighbor connectivity labeling within the primary occupied layer to extract the largest connected region, denoted as . , and then from The middle selection simultaneously satisfies the subject confidence graph Reaching the subject confidence threshold The left-right consistency score reaches the left-right consistency threshold. Located in the candidate main area By sampling points within the body, a set of points within the body is obtained. Finally, based on the set of interior points of the main body Read the corresponding depth layer number at the layer position in the main occupied layer, and set the in-body points according to the dual-targeting parameters. The corresponding layer depth values ​​are mapped to a depth sequence. After sorting the depth sequence numerically, the value corresponding to the middle position is taken as the center value of the depth distribution. Then, combined with the depth layer number of the main occupying layer and the dual-target positioning parameters, the representative depth of the main body is calculated. Set of interior points of the main body and the depth of the main body As the direct input object for the three-dimensional back projection in step S5;

[0125] S5: The receiver layer's decision head outputs the set of in-body points, the representative depth of the subject, and the target semantic box. It extracts the pixel positions within the target detection box for each pixel in the set of in-body points and combines the pixel positions with the representative depth of the subject according to the pixel correspondence to construct a back-projection pixel set. Based on the camera imaging parameters obtained through dual-target calibration, it maps each pixel position in the back-projection pixel set to a back-projection ray and determines the three-dimensional point positions on the back-projection ray based on the representative depth of the subject to obtain the three-dimensional point set of the subject.

[0126] Spatial aggregation is performed on the three-dimensional point set of the subject, and the spatial center position, spatial range and depth distribution boundary of the three-dimensional point set of the subject are calculated to form the subject spatial parameters; the subject spatial parameters are associated with the target category, target detection box and subject representative depth in the target semantic box to construct a spatial target instance corresponding to the target semantic box;

[0127] Specifically, S5 includes the following steps:

[0128] S51: Set of body points of the main body output by the layer decision head Main body representative depth and target semantic box As input, the target detection box The reverse mapping from the corresponding sampling grid to the pixel coordinates of the left view is denoted as... Set of interior points of the main body Input point by point This yields the pixel position sequence inside the target detection bounding box. ,in Indicates the first The pixel position of the left view corresponding to each in-body point. This indicates the number of points within the main body, followed by the position of each pixel. Depth of representation of the main body Construct a back-projected pixel set by combining pixels according to a one-to-one correspondence. ,in Indicates the first A back-projected pixel record, a back-projected pixel set Preserve the set of interior points of the main body The results of filtering the spatial distribution of the target ontology, and the representative depth of the subject determined by the main occupying layer. Directly write it into the subsequent 3D backprojection process;

[0129] S52: The imaging parameters of the left camera obtained through binocular targeting are denoted as... The left camera's optical center is recorded as Left camera imaging parameters Includes focal length parameters, principal point parameters, and pixel scale parameters, and is correlated with the target detection bounding box. The coordinate system of the left view remains consistent, and the back-projected pixel set is consistent. Each back-projected pixel is recorded According to pixel position With left camera imaging parameters Determine the light center passing through the left camera. The back projection ray, denoted as Subsequently, in the back projection ray Above, according to the depth of the main representative By extracting 3D points at the corresponding depth positions, the main body's 3D point set is obtained. Due to the main three-dimensional point set Only the set of interior points of the subject The corresponding pixel positions are generated, forming a 3D point set for the main body. Maintain the same subject affiliation with the main occupying layer, and separate it from the background area outside the target detection box;

[0130] S53: For the main body's three-dimensional point set Perform spatial aggregation, specifically, extract the main body's three-dimensional point sets separately. The horizontal, vertical, and depth coordinates of all 3D points are calculated and sorted along these three axes. The minimum and maximum values ​​along these three axes form the 3D bounding boundary. The spatial center is then determined based on the midpoint of this 3D bounding boundary. The span in the three coordinate directions is combined to form the spatial range. The minimum and maximum values ​​in the depth coordinate direction form the depth distribution boundary. Central position of space Spatial range and depth distribution boundary Together they constitute the main spatial parameters When the main body's three-dimensional point set When multiple 3D points have the same depth coordinates, the depth distribution boundary It consists of the same front and back boundary values; when the main three-dimensional point set When there is a distribution width in the depth coordinate direction, the depth distribution boundary It directly reflects the position of the front and rear boundaries;

[0131] S54: Complete the main spatial parameters After generation, the main spatial parameters will be... With the target semantic box Target categories Target detection box and the depth of the main body By target index By associating them, spatial target instances can be formed. Space target instance With the target semantic box Maintaining a one-to-one correspondence, and serving as the third type of output of the YOLOv10 main occupancy layer fusion network, it is used in step S6 based on spatial target instances. This results in the attribution of the main depth of the target cargo hold and the parameters of its operational location.

[0132] S6: Receive spatial target instances and extract target category, target detection box, subject representative depth, and subject spatial parameters from the spatial target instances; calculate the projection deviation based on the projection relationship between the target detection box and the spatial center position; when the projection deviation meets the projection threshold, bind the target category, target detection box, subject representative depth, and spatial center position to form the subject depth attribution result;

[0133] The coordinates of the operation position are determined based on the spatial center position in the main spatial parameters, and the operation depth range and operation coverage range are determined based on the spatial range and depth distribution boundary to obtain the operation position parameters; the main depth attribution result is correlated with the operation position parameters to output the main depth attribution result and specific parameters of the loading target in the air cargo hold;

[0134] Specifically, S6 includes the following steps:

[0135] S61: An instance of a space target formed by S5 For input, a space target instance This is the third type output of the YOLOv10 main occupancy layer fusion network, and along the target index. Consistent with the target semantic box, for each spatial target instance Extract target category Target detection box Main body representative depth and main spatial parameters ,in, Indicates the center position of space. Indicates spatial range, Representing the depth distribution boundary, due to spatial object instances From the set of interior points of the main body Depth of representation of the main body Back projection formation, step S6 at the same target index The consistency of the two-dimensional target detection bounding box and the three-dimensional subject space parameters is then determined.

[0136] S62: Calculate the projection deviation and move the target detection box. The geometric center is denoted as The center position of space Based on the imaging parameters of the left camera obtained through binocular calibration Projecting the pixel coordinates onto the left view yields the projection point at the center of the space. When the projection point at the center of space Falling into the target detection box When inside, extract the projection points at the center of the space. With respect to the geometric center of the target detection box The absolute values ​​of the lateral and longitudinal deviations are calculated, and the larger of the two values ​​is recorded as the projection deviation. When the projection point is at the center of space Falling into the target detection box When the target is external, the current spatial target instance is directly determined to not meet the projection threshold. Spatial target instances will not be included in subsequent bindings, for those that meet the following conditions. Spatial target instances, target categories Target detection box Main body representative depth and the center of space Indexed by the same target Binding is performed to form a subject-specific deep attribution result. ;

[0137] S63: Obtaining subject depth attribution results Then, based on the main spatial parameters Construct the operation position parameters and determine the spatial center position. As work location coordinates , to define the depth distribution boundary The front and back boundary values ​​constitute the operating depth range. , spatial range The horizontal and vertical ranges constitute the work coverage area. When the spatial range When the depth direction range is included, the depth direction range is not included in the operation coverage area. And by the range of operating depth Individual characterization, thereby forming the operation location parameters Operation location parameters Subject depth attribution results Shared target index and maintain contact with space target instances One-to-one correspondence;

[0138] S64: Assign subject depth results With operation position parameters Perform correlation to form the output result of the current target. For all that meet the projection threshold After performing the same processing on the spatial target instances, the output result set is obtained. Output result set Each output includes the target category, target detection box, main body representative depth, spatial center position, operation position coordinates, operation depth range, and operation coverage range, and serves as the main body depth attribution result and operation position parameter for the target loaded in the air cargo hold;

[0139] Left view of an aircraft cargo hold scene as follows Figure 2 As shown, the solid rectangle represents the cargo container, and the dashed rectangle represents the target detection bounding box obtained by YOLOv10. The target detection bounding box only reflects the coverage area of ​​the target in the image plane. In addition to the target body, the inside may also include non-subject areas such as cargo hold walls, loading gaps, and the outlines of adjacent targets. Figure 2 The initial input form used to illustrate the present invention is to use the target detection box as a spatial index for subsequent candidate subject region extraction, main occupied layer determination and subject representative depth determination, instead of using the target detection box directly as the basis for the target's true outline or depth attribution, thereby providing a basis for subsequent subject depth attribution discrimination.

[0140] The result of extracting the main body region based on the detection bounding box is as follows: Figure 3 As shown, the solid rectangle represents the box outline, the dashed rectangle represents the detection box, and the area filled with diagonal lines is the candidate subject area obtained by this invention based on the subject confidence map and boundary suppression band; the narrow strip area located between the box and the adjacent box is marked as a gap and is not filled. Figure 3 This invention demonstrates that by performing connected component analysis and stripe morphology determination on the local response distribution within the detection frame, it can automatically eliminate cargo hold wall gaps, loading gaps, and adjacent contours, retaining only the pixel set that highly overlaps with the target entity. This provides a cleaner and more accurate foreground region for subsequent binocular matching and main occupancy layer extraction, significantly reducing background interference compared to directly using the detection frame.

[0141] After obtaining the main region and the depth representing the main region, the present invention reconstructs the three-dimensional spatial target and outputs the results of the operation parameters as follows: Figure 4 As shown in the figure, the spatial locations of multiple goods are represented by wireframe cubes. The center point and its depth distribution boundary in the longitudinal direction are marked inside the central target. The rectangle filled with diagonal lines below the cube represents the operation coverage area given by this invention. This figure illustrates that this invention uses the set of internal points of the main body extracted by the binocular master occupancy layer, combined with the camera calibration parameters, to perform three-dimensional back projection, which can restore the true position, size, and front and rear boundaries of the target in the cargo hold coordinate system, and further derive the operation position coordinates and coverage area of ​​the loading and unloading equipment. Thus, it realizes the direct driving of three-dimensional loading and operation planning by two-dimensional detection results, which is more suitable for engineering applications than the traditional method that only outputs 2D boxes.

[0142] Depth attribution without using the method of this invention is as follows: Figure 5As shown in (a), the dashed detection box of the target on the right covers the main body of the container, the gap between the container and the cargo hold wall, and part of the wall. The entire detection box area is filled with diagonal lines as the area of ​​incorrect depth attribution. This means that the existing method usually calculates or averages the depth directly within the detection box, which causes the gap and the wall to be mistakenly identified as the main body of the target. This makes the estimated depth biased towards the cargo hold wall or the background position, resulting in the phenomenon that the target is attracted to the wall. This figure is used to illustrate that if the main body and non-main body areas are not distinguished, the depth estimation is difficult to reflect the true front and back position of the cargo. In scenarios where the cargo is close to the wall, obstructed, or stacked, it will seriously affect subsequent loading decisions and the safety of mechanical operations.

[0143] The depth attribution effect after adopting the present invention is as follows: Figure 5 As shown in (b), in the main image on the left, only the internal area of ​​the container is filled with diagonal lines to represent the depth of the main occupancy layer. Gaps and walls remain blank, indicating that their depth is no longer attributed to the target. The side view illustration in the upper right corner further demonstrates the true front-back relationship between the container and the cargo hold wall: the container is in front, and the gaps and walls are in deeper areas. The main occupancy depth layer accurately falls on the main body of the container. This figure proves that the present invention, through main body region extraction, binocular matching consistency constraint, and multi-index layer determination, only assigns the depth to the real target entity. This effectively eliminates the interference of gaps and background, significantly improves the depth accuracy and stability in complex scenarios such as near walls and stacking, and provides a reliable foundation for accurately generating loading poses and working areas.

[0144] Those skilled in the art will understand that the embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Moreover, this application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The solutions in the embodiments of this application can be implemented in various computer languages, such as the object-oriented programming language Java and the interpreted scripting language JavaScript.

[0145] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0146] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0147] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0148] Although preferred embodiments of this application have been described, those skilled in the art, once they learn the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0149] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if these modifications and variations of this application fall within the scope of the claims of this application and their equivalents, this application also intends to include these modifications and variations.

Claims

1. A method for attributing the depth of loading targets in an aircraft cargo hold based on subject region filtering and binocular vision main occupancy layer determination, characterized in that, include: S1: Obtain the left and right views of the aircraft cargo hold after bi-target calibration. Run YOLOv10 in the left view to obtain target semantic boxes containing target categories, target detection boxes, and local response distributions. S2: Generate a boundary suppression band within the target detection box based on the boundary distance according to the target semantic box, and map the local response distribution to the subject confidence map; under the constraint of the target category, perform continuous region filtering on the subject confidence map, and determine the set of high confidence entity pixels located outside the boundary suppression band, far away from the cargo hold wall gap, loading gap and adjacent target contour, and satisfying the connectivity area condition as candidate subject regions, and determine the candidate subject regions as the binocular matching range; S3: Establish pixel correspondence between left and right views within the candidate subject area according to epipolar constraints, calculate disparity consistency and left-right correspondence stability of corresponding pixels, and perform weighted aggregation and connectivity merging of similar disparity pixels according to the subject confidence map to form layer occupancy evidence; filter layer occupancy evidence according to left-right consistency threshold and connectivity area threshold, and determine the layer occupancy evidence that meets the binocular correspondence stability and continuous occupancy conditions as valid layer occupancy evidence. S4: Calculate the overlap of each depth layer based on the valid layer occupancy evidence and the subject confidence map, and perform attribution determination on multiple depth layers that meet the threshold under the target category scale constraint to determine the main occupancy layer and output the subject interior point set and the subject representative depth; S5: Perform 3D back projection based on the set of points inside the subject and the representative depth of the subject to form a spatial target instance corresponding to the target semantic box; S6: Generate the main depth attribution result of the air cargo hold loading target based on the space target instance, and output the operation position parameters.

2. The method for attributing the depth of loading targets in an aircraft cargo hold based on subject region filtering and binocular vision main occupancy layer determination according to claim 1, characterized in that, S1 specifically includes: The left and right views of the aircraft cargo hold, determined by the bi-target, are input into the shared weight backbone network of YOLOv10. The Stem layer and C2f layer are used to extract convolutional features from the left and right views of the aircraft cargo hold to create multi-scale feature maps of the left and right views. The left-view multi-scale feature map and the right-view multi-scale feature map are input into the FPN-PAN feature fusion layer. The left-view multi-scale feature map and the right-view multi-scale feature map are aligned and fused to form the left-view fused feature map and the right-view fused feature map. Input the left view fused feature map into the classification regression head, perform target category prediction and target detection box regression on the left view fused feature map, and obtain the target category and target detection box; The left view fusion feature map is input into the local response distribution branch, and region alignment sampling is performed on the left view fusion feature map according to the target detection box to obtain the local response distribution; The target category, target detection box, and local response distribution are associated to form a target semantic box.

3. The method for attributing the depth of loading targets in an aircraft cargo hold based on subject region filtering and binocular vision main occupancy layer determination according to claim 1, characterized in that, S2 specifically includes: Receive the target detection box and local response distribution in the target semantic box, perform region alignment sampling on the left view fusion feature map according to the target detection box, and resample the local response distribution to the sampling grid consistent with the target detection box to form a response alignment map; The response alignment map is input to the response mapping layer, which is composed of sequentially connected two-dimensional convolutional neurons, and the response alignment map is processed by neighborhood aggregation, channel compression, and confidence mapping to obtain the main confidence map. The confidence value of each pixel in the main confidence map corresponds to the judgment value of the pixel at the same position in the target detection box belonging to the set of high-confidence entity pixels. The boundary distance map is calculated based on the boundary distance from each pixel within the target detection box to the boundary of the target detection box. The boundary distance threshold and boundary bandwidth threshold are determined in combination with the target category constraint. The boundary distance map is then thresholded to obtain the boundary suppression band. The main confidence map is aligned with the boundary suppression band. High confidence pixels in the main confidence map are selected under the constraint of the main confidence threshold. High confidence pixels that fall into the boundary suppression band are marked as edge-fitting pixels to form the main candidate map. Continuous region filtering is performed on the candidate graph of the main body. The connected area, boundary overlap ratio, boundary continuity length and internal break length of each connected region are calculated. Connected regions that meet the connected area condition, the boundary overlap ratio is lower than the overlap threshold and the internal break length is lower than the break threshold are identified as solid connected regions. Based on the target category constraints, the contour of the entity connected region is determined. The extension direction and strip width of the entity connected region along the boundary of the target detection box are calculated. The entity connected region whose boundary continuous length reaches the edge length threshold and whose strip width is lower than the strip width threshold is determined as the cargo hold wall gap, loading gap or adjacent target contour corresponding region, and is screened out from the entity connected region to obtain the candidate subject region. Candidate subject regions are mapped to corresponding regions in the left-view fusion feature map and the right-view fusion feature map, and the pixel index set of the candidate subject regions is used as the binocular matching range of the binocular main occupancy layer extraction network.

4. The method for attributing the depth of loading targets in an aircraft cargo hold based on subject region filtering and binocular vision main occupancy layer determination according to claim 1, characterized in that, S3 specifically includes: The candidate subject region is mapped to the left view fusion feature map and the right view fusion feature map. Under the constraint of the target semantic box, region alignment sampling is performed on the left view fusion feature map and the right view fusion feature map to obtain the left region feature and the right region feature. The left-side region features, right-side region features, subject confidence map, and candidate subject regions are fused along the channel direction to form fused features, and the candidate disparity search interval is determined on the fused features based on the binocular matching range. Based on the candidate disparity search interval and epipolar constraints, disparity matching cost calculation is performed on the fused features to form a disparity cost body; The disparity cost volume is input into an occupancy aggregation unit composed of sequentially connected 3D convolutional neurons. Occupancy aggregation is performed on the disparity cost volume along the disparity direction and the spatial direction to obtain the layer occupancy response. The pixel correspondence between the features of the left region and the features of the right region is established based on the layer occupancy response. Based on pixel correspondence, disparity consistency and left-right correspondence stability are calculated, and left-right consistency score is calculated based on disparity consistency and left-right correspondence stability to obtain effective pixel correspondence; The pixel confidence values ​​in the subject confidence map are mapped to the disparity layers corresponding to the effective pixels. Weighted aggregation and connectivity merging are performed on the effective pixels that are adjacent in disparity and spatially connected to form layer occupancy evidence. The connected area value and occupancy continuity length corresponding to the layer occupancy evidence are calculated. The layer occupancy evidence, left-right consistency score, connected area value, and occupancy continuity length inputs are sequentially connected by fully connected neurons to form a layer decision head. The layer occupancy evidence that meets the binocular correspondence stability and continuous occupancy conditions is selected as valid layer occupancy evidence based on the left-right consistency threshold and connected area threshold.

5. The method for attributing the depth of loading targets in an aircraft cargo hold based on subject region filtering and binocular vision main occupancy layer determination according to claim 1, characterized in that, S4 specifically includes: Receive valid layer occupancy evidence, subject confidence map, candidate subject region and target semantic box. Expand the valid layer occupancy evidence into the corresponding position of the target detection box in the sampling grid of the candidate subject region according to the depth layer number to form multiple depth layer pixel regions. Calculate the number of pixel supports, pixel support density, edge distribution ratio and spatial connectivity distribution of each depth layer pixel region to obtain the depth layer support value. Based on the subject confidence map, position overlap calculation is performed on the pixel regions of each depth layer. High confidence pixels that reach the subject confidence threshold in the subject confidence map are extracted. The coverage ratio, subject confidence cumulative value, center coverage ratio and boundary overlap ratio of high confidence pixels in the pixel regions of each depth layer are statistically analyzed. The above indicators are then fused to obtain the overlap degree of each depth layer. Based on the target category in the target semantic box and the target detection box, a target category scale constraint is constructed. Combined with the dual target calibration parameters, the pixel area of ​​each depth layer is converted into the corresponding entity size. The matching degree between the corresponding entity size and the length relationship, width relationship and height range in the target category scale constraint is calculated to obtain the scale consistency score. The contact ratio between the pixel area of ​​each depth layer and the boundary of the target detection box is calculated to obtain the boundary contact value. The layer determination head, composed of fully connected neurons, is used to input the overlap, scale consistency score, boundary contact value, depth layer support value, left-right consistency score, connected area value, and continuous length of the effective layer occupancy evidence for each depth layer. This head is then used to perform a classification determination on each depth layer, resulting in a classification determination score for each depth layer. Depth layers whose attribution scores reach the attribution threshold, whose overlap of each depth layer reaches the overlap threshold, whose scale consistency scores reach the scale threshold, and whose boundary contact values ​​are lower than the contact threshold are identified as candidate attribution layers. Candidate attribution layers whose number of pixel supports is lower than the number threshold or whose spatial connectivity distribution does not meet the connectivity conditions are filtered out based on the depth layer support values. Perform inter-layer comparison on the candidate layers of attribution, determine the candidate layer with the highest attribution score as the main occupancy layer, and filter the pixels in the main occupancy layer that reach the subject confidence threshold, meet the left-right consistency threshold, are located in the candidate subject area and are in the connected region to obtain the set of subject in-points. The depth of the subject is calculated based on the layer position of the subject's interior point set in the main occupied layer, the depth distribution center value corresponding to the subject's interior point set, and the bi-objective positioning parameters.

6. The method for attributing the depth of loading targets in an aircraft cargo hold based on subject region filtering and binocular vision main occupancy layer determination according to claim 1, characterized in that, S5 specifically includes: The receiver layer's decision head outputs the set of in-body points, the representative depth of the subject, and the target semantic box. It extracts the pixel positions within the target detection box for each pixel in the set of in-body points and combines the pixel positions with the representative depth of the subject according to the pixel correspondence to construct a back-projected pixel set. Based on the camera imaging parameters obtained through binocular calibration, the positions of each pixel in the back-projection pixel set are mapped to back-projection rays, and the positions of three-dimensional points are determined on the back-projection rays according to the representative depth of the subject, thus obtaining the three-dimensional point set of the subject. Spatial aggregation is performed on the main body's three-dimensional point set to calculate the spatial center position, spatial range, and depth distribution boundary of the main body's three-dimensional point set, thus forming the main body's spatial parameters; The spatial parameters of the subject are associated with the target category, target detection box, and subject representation depth in the target semantic box to construct a spatial target instance corresponding to the target semantic box.

7. The method for attributing the depth of loading targets in an aircraft cargo hold based on subject region filtering and binocular vision main occupancy layer determination according to claim 1, characterized in that, S6 specifically includes: Receive spatial target instances and extract target category, target detection box, subject representative depth, and subject spatial parameters from the spatial target instances; The projection deviation is calculated based on the projection relationship between the target detection box and the spatial center position. When the projection deviation meets the projection threshold, the target category, target detection box, subject representative depth and spatial center position are bound together to form the subject depth attribution result. The coordinates of the work location are determined based on the spatial center position in the main spatial parameters, and the work depth range and work coverage range are determined based on the spatial range and depth distribution boundary, thus obtaining the work location parameters; The main body depth attribution results are correlated with the operation location parameters to output the main body depth attribution results and specific parameters of the loading target in the air cargo hold.

Images