A multi-camera based single piece separation package positioning method, apparatus, and medium

Abstract

The application discloses a kind of based on multi-camera single piece separation package positioning method, equipment and medium, including arrangement multiple cameras, according to the image recognition package target detection frame of camera;According to package target detection frame and depth data, obtain package initial mask area;Again determine whether package is irregular package, if it is irregular package, then to package initial mask area is regularized processing, output package mask area;Combining camera internal parameter, convert package mask area and depth data into package point cloud, and projection and fitting obtain package projection frame;The projection frame of camera is mapped to other camera coordinate system to form mapping frame, matching projection frame and mapping frame, carry out fusion and output the global fusion frame of package;Global fusion frame is handled, and the vertex sequence of corrected frame is generated, and package number is distributed.The application can realize large field of view, high precision, high real-time package single piece separation and pose positioning under dynamic belt scene.

Description

Technical Field

[0001] This invention relates to the field of logistics parcel sorting technology, and in particular to a method, device and medium for locating single-piece separated parcels based on multiple cameras. Background Technology

[0002] With the rapid development of e-commerce and logistics industries, automation and intelligence in parcel sorting have become core industry requirements. Separating and locating dynamic parcels is the core foundation of automated sorting systems. Only by accurately acquiring the spatial position and posture information of each parcel can sorting equipment be guided to complete efficient and accurate sorting operations, avoiding missorting and omissions.

[0003] In existing technologies, the methods for single-item separation and pose localization of packages in dynamic scenarios mainly have the following technical solutions and shortcomings: Two-dimensional vision positioning solutions rely on RGB cameras to acquire planar images and use target detection algorithms to identify package locations. However, this solution can only obtain two-dimensional coordinates and lacks depth information, making it unable to accurately determine the spatial grounding position of the package. Furthermore, this solution is prone to misjudgment and missed detection when packages are stuck together, overlapping, or moving at high speeds, and it cannot output package posture information, resulting in insufficient sorting accuracy and reliability.

[0004] Single RGBD camera positioning solutions introduce a depth camera to acquire 3D information and output the depth coordinates of the package. However, a single camera has a natural blind spot, failing to cover the full width of the dynamic conveyor belt or long transport paths. Furthermore, depth information from a single viewpoint is susceptible to interference from package surface reflections, texture loss, and belt vibration, resulting in significant pose positioning errors that cannot meet the accuracy requirements of high-speed sorting scenarios. While some multi-camera collaborative positioning solutions use multiple cameras to expand the field of view, existing multi-camera fusion algorithms often rely on simple image stitching or bounding box overlays, without optimization for multi-camera fusion techniques specific to dynamic conveyor belt scenarios.

[0005] Therefore, existing technologies suffer from defects such as the contradiction between field of view coverage and positioning accuracy, poor adaptability to dynamic scenes, low efficiency of multi-camera collaboration, and insufficient robustness. Summary of the Invention

[0006] Purpose of the invention: The purpose of this invention is to provide a method, device and medium for single-piece separation and positioning of packages based on multiple cameras, which can achieve large field of view, high precision and high real-time performance in dynamic belt scenarios for single-piece separation and pose positioning of packages.

[0007] Technical Solution: To achieve the above objectives, the present invention provides a single-item separation and positioning method based on multiple cameras, comprising the following steps: S1. Multiple cameras are arrayed above the single-piece separation area, with a reference camera, a first camera, and an optional second camera in each row; images and depth data of all cameras are acquired synchronously, and the target detection box is identified based on the image of each camera; S2. Based on the package target detection box and depth data, determine the package type. If it is a small or large package, perform binarization segmentation on the depth data to obtain the initial mask region of the package. If it is a package with missing depth or a flat package, perform semantic segmentation on the package target detection box to obtain the initial mask region of the package. Then determine whether the package is an irregular package. If it is an irregular package, perform regularization processing on the initial mask region of the package and output the package mask region. S3. Combining camera intrinsic parameters, the wrapping mask area and depth data are converted into wrapping point clouds, projected onto the belt plane, and then fitted to obtain the wrapping projection box. S4. In each row, the projection frame of the first camera is mapped to the coordinate system of the reference camera to obtain the mapping frame of the first camera. The mapping frame of the first camera and the projection frame of the reference camera are matched. The successfully matched contour points are merged into the first-level fused contour point set. The first-level fused projection frame is output based on the first-level fused contour point set and the unmatched single-frame contour point set. If the cameras are arranged in a single row and there are no second cameras, the output of the first-level fused projection box is the wrapping global fused box; If the cameras are arranged in a single row and there is a second camera, the projection frame of the second camera is mapped to the coordinate system of the reference camera to obtain the mapping frame of the second camera; the mapping frame of the second camera is matched with the projection frame after the first level fusion, and the contour points of the successfully matched contours are merged into the second level fusion contour point set. Based on the second level fusion contour point set and the single frame contour point set of the unmatched ones, the projection frame after the second level fusion is output as the global fusion frame. If the cameras are arranged in a matrix and there is no second camera, map the first-level fused projection box of the second row to the coordinate system of the reference camera in the first row to obtain the mapping box of the second row; match the mapping box of the second row with the first-level fused projection box of the first row, and merge the successfully matched contour points into a row-level fused contour point set. Based on the row-level fused contour point set and the unmatched single-frame contour point set, output the row-level fused projection box as the global fused box. S5. Perform deduplication and global ROI boundary constraint processing on the global fused bounding box, generate the corrected bounding box vertex sequence, assign wrapper numbers, and output the wrapper numbers and the corrected global fused bounding box.

[0008] Optionally, in step S2, the package type is determined based on the target detection frame and depth data of the package. Specifically, this includes the following steps: when the length or width of the target detection frame area is less than 200 pixels, it is determined to be a small package; when both the length and width of the target detection frame area are greater than 200 pixels, it is determined to be a large package; when the area of ​​the target detection frame area with a depth data of 0 is greater than 30% of the target detection frame area, it is determined to be a package with missing depth; when all the depth data of the package is less than 20mm, it is determined to be a flat package.

[0009] Optionally, in step S2, if the package is small or large, the depth data is binarized and segmented to obtain the initial mask area of ​​the package, specifically including the following steps: For small or large packages, extract the depth data corresponding to each pixel within the bounding box to generate a depth map within the bounding box, and statistically analyze the depth values ​​of all pixels within the bounding box to generate a depth histogram. Based on the depth histogram, calculate the depth segmentation threshold between the upper surface and non-upper surface regions of the package, and use the depth segmentation threshold to perform binarization segmentation of the depth map within the bounding box to obtain the mask region of the package. The method for obtaining the depth segmentation threshold of small packages is as follows: First, a 1×5 smooth convolution kernel is constructed to filter the depth histogram. Then, the main peak depth value of the depth histogram is coarsely located and retrieved. Then, the interval is expanded to the left and right by 30 pixels from the center of the main peak depth value, and the depth weighted average is performed to obtain the depth segmentation threshold. The method for determining the depth segmentation threshold of a large package is as follows: starting from the maximum depth of the camera, traverse each depth value one by one along the direction of decreasing depth and accumulate the corresponding number of pixels; when the accumulated number of pixels first reaches 80% of the total number of pixels in the depth histogram, the depth value traversed is determined as the depth segmentation threshold.

[0010] Optionally, determining whether the package is an irregular package in step S2 specifically includes the following steps: Based on the initial mask area of ​​the package, the mask regularity is calculated. When the mask regularity is less than 0.85, it is judged as an irregular package; when the mask regularity is not less than 0.85, it is judged as a regular package. Mask regularity is defined as the ratio of the actual area of ​​the mask to the area of ​​the mask convex hull. The actual area of ​​the mask is obtained by counting the total number of all non-zero pixels in the mask region. The mask convex hull is the smallest convex polygon that can completely enclose all the contour pixels of the mask. First, the outer contour point set of the mask is extracted, and then the smallest convex polygon is solved based on the outer contour point set to obtain the mask convex hull. The area of ​​the mask convex hull is the total number of pixels in the region covered by the smallest convex polygon.

[0011] Optionally, step S2 involves regularizing the area of ​​the initial mask, specifically including the following steps: Calculate the principal direction of the initial mask region of the package, and align the principal direction of the initial mask region of the package with the X-axis of the Cartesian coordinate system through affine transformation to obtain the rotated mask region; perform binarization processing on the rotated mask region with 127 as the threshold, where pixel values ​​> 127 are determined as foreground and pixel values ​​≤ 127 are determined as background; traverse the foreground and solve its minimum bounding rectangle to determine the effective foreground region of the package. Set the grid step size to The effective foreground region of the package is meshed; the half-grid offset is calculated. Starting from the top-left vertex of the smallest enclosing rectangle, with As the initial offset, according to the grid step size Traverse the grid row by row and column by column, collecting the coordinates of the center point of each cell; using the center point of each cell as the center, extract a side with a length of... Determine the square neighborhood; count the pixel values ​​within this square neighborhood. The number of effective foreground pixels, if the number If the number is 0, the grid is marked as a valid grid; if the number is 0, the grid is marked as an invalid grid. Record the coordinates of all valid grids and count the number of consecutive valid grids arranged horizontally at each position; record the grid positions to locate the valid area inside the package, and count the number of consecutive valid grids horizontally to characterize the area occupied by the package laterally. A two-dimensional grid is constructed with valid grid cells as 1 and invalid grid cells as 0. The algorithm of column-by-column traversal and monotonic stack is used to find the largest rectangular region in the two-dimensional grid that is composed entirely of valid grid cells. The boundary parameters of the largest rectangular region are calculated and the top, bottom, left and right boundaries of the largest rectangular region are recorded. Based on the boundary parameters of the largest rectangular region, combined with the grid step size Given the starting coordinates of the minimum bounding rectangle, calculate the four rectangle vertices of the minimum bounding rectangle in the rotated image; by calculating the inverse matrix of the affine transformation, map the rectangle vertices of the rotated image back to the original image coordinate system through the inverse affine transformation to obtain the regularized bounding mask region.

[0012] Optionally, in step S4, mapping the projection frame to the reference camera coordinate system to obtain the mapped frame specifically includes the following steps: Let the pixel coordinates of the projection box in the image plane be... The homogeneous coordinates of the pixels are obtained as follows: homography matrix for: , Homogeneous coordinates of pixels With homography matrix Multiply to obtain the transformed homogeneous coordinates. After normalization, the final pixel coordinates in the reference camera coordinate system are obtained as follows: and mapping box; , , , , , , , in These are the final pixel coordinates in the unnormalized reference camera coordinate system.

[0013] Optionally, in step S4, matching the projection box and the mapping box specifically includes the following steps: Iterate through all pairwise combinations of the projected bounding box set and the mapped bounding box set, calculate the distance feature value dis and the overlap rate feature value OR for each pair of boxes, and set the effective matching pair filtering conditions to obtain all effective matching pairs; sort all effective matching pairs from best to worst, and select the best combination as the successful matching pair; after a successful match, mark the matched box, and mark the unmatched projected bounding box and mapped bounding box as unmatched, and set the unmatched projected bounding box and mapped bounding box as a single box; The distance feature value dis for each pair of boxes is calculated. The distance feature value dis is the sum of the absolute distances of the four vertices and one center point of the projected box and the mapped box, i.e., dis = d1 + d2 + d3 + d4 + d5, or the distance feature value dis is the sum of the absolute distances of the four vertices of the projected box and the mapped box, i.e., dis = d1 + d2 + d3 + d4; d1 to d5 are the Euclidean distances between two points; the overlap rate feature value OR is defined as the ratio of the intersection area of ​​the projected box and the mapped box to the smaller area of ​​the two boxes. Set the filtering conditions for valid matching pairs: dis∈(0.1, 300), or OR>0.49, or dis∈(301, 400) and OR≥0.2. Projected boxes and mapped boxes that meet any of the filtering conditions are valid matching pairs. Sort all valid matching pairs from best to worst. Specifically, first compare the overlap rate feature value OR, and rank the valid matching pairs with the larger overlap rate feature value OR first. If the overlap rate feature values ​​OR are equal, then compare the distance feature value dis, and rank the valid matching pairs with the smaller distance feature value dis first. Finally, the order from best to worst is formed.

[0014] Optionally, step S5 involves deduplication of the global fusion box and global ROI boundary constraint processing for vertex coordinates, specifically including the following steps: Traverse all global merge boxes, calculate the IOU between each pair of global merge boxes and the area of ​​a single box. If the IOU is greater than 0.8 and the area of ​​a single box is less than 500, it is determined to be a duplicate redundant box and removed. Obtain the coordinates of the four vertices of the global bounding box to be processed, and set the boundary range of the ROI. The ROI boundary is limited by the coordinates of the top left and bottom right vertices. Detect each of the four edges of the global fusion box, count the number of vertices on each edge that exceed the ROI boundary, and record the position of all out-of-bounds vertices and their respective boundary directions, which are up, down, left, and right. Based on the total number of out-of-bounds vertices and their distribution on the four edges, perform the corresponding correction operation, specifically: If there is only a single vertex that exceeds the boundary of the ROI, then the vertex that exceeds the boundary is vertically projected onto the corresponding ROI boundary line along its direction of exceeding the boundary, while keeping the other coordinate of the vertex unchanged. The projected vertex is used as the corrected vertex to form the corrected global fusion box. If two vertices in the same frame both exceed the same boundary of the ROI, then calculate the two intersection points of the outline segment of the global fusion box with the boundary of the ROI, replace the two original out-of-bounds vertices with the two intersection points, and leave the other vertices unchanged to form the corrected global fusion box. If two out-of-bounds vertices are located on two different boundaries of the ROI, then each out-of-bounds vertex is vertically projected onto the corresponding ROI boundary line along its respective out-of-bounds direction to obtain two corrected vertices, while the remaining vertices remain unchanged; this forms the corrected global fusion box.

[0015] Based on the same inventive concept, the present invention provides an electronic device including a processor and a storage medium; The storage medium is used to store instructions; The processor is configured to operate according to the instructions to perform the steps of the method described above.

[0016] Based on the same inventive concept, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described above.

[0017] Beneficial effects: Compared with the prior art, the present invention has the following significant advantages: (1) Based on the fusion of visual and depth data from multiple RGBD cameras, this invention achieves comprehensive coverage, high-precision positioning, robust recognition and efficient collaborative processing of packages in dynamic belt sorting scenarios by dynamically adapting to multi-camera deployment modes, lightweight target detection, cross-camera coordinate mapping and fusion, and finally outputs package positioning information that meets the requirements of automated sorting equipment. (2) For the detection box of a single small or large package, the present invention prioritizes generating a package mask based on the depth map; for flat parts or packages lacking depth, semantic segmentation is introduced to obtain RGB masks to ensure the robustness of mask generation and obtain more accurate position information. (3) The present invention performs regularization processing on the initial mask area of ​​the package, which improves the accuracy of the package projection frame and avoids the package depth information in the extracted detection frame being interfered with by the nearby packages when there are many packages in the field of view. (4) In order to solve the problem of inconsistent package frame positioning caused by the difference of multiple camera perspectives, this application converts the package position information under different camera perspectives to a unified coordinate system and achieves accurate fusion, and finally outputs globally consistent and accurate package position information, which is suitable for logistics sorting scenarios with multiple camera deployments. (5) The dual feature matching of the present invention ensures high-quality matching priority and one-to-one unique matching, and finally outputs the globally optimal matching result, while retaining the information of unmatched bounding boxes, providing an accurate association basis for subsequent multi-camera bounding box fusion; (6) The present invention uses global ROI boundary constraint processing of vertex coordinates to ensure that the vertices of all detection boxes eventually fall within the effective area of ​​the image, thereby avoiding processing errors caused by boundary anomalies and improving the accuracy of positioning. Attached Figure Description

[0018] Figure 1 This is a schematic diagram of the process of the present invention; Figure 2 This is a schematic diagram showing the location of packages in the sorting scenario of this invention; Figure 3 This is a schematic diagram of point cloud denoising processing in this invention; Figure 4 This is a schematic diagram illustrating the calculation of the distance feature value dis in this invention; Figure 5 This is a schematic diagram showing the fusion of the projection frames and mapping frames of the two cameras in this invention; Figure 6 This is a schematic diagram of the global ROI boundary constraint processing for vertex coordinates in this invention; Figure 7 This is a schematic diagram of the layout of the four cameras in this invention. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0020] like Figure 1As shown, to address the technical problems of insufficient field of view coverage, poor coordination consistency, weak robustness, insufficient real-time performance, and inaccurate pose output in existing dynamic belt sorting scenarios for single-item separation and pose localization, this invention proposes a single-item separation and package localization method based on multiple cameras, comprising the following steps: S1. Multiple cameras are arrayed above the single-piece separation area, with each row containing a reference camera, a first camera, and an optional second camera. The reference camera, the first camera, and the second camera are all RGBD cameras, which synchronously acquire the original RGB images and depth data of the area to be separated below the multiple cameras. The homography matrix between two adjacent cameras is calculated to obtain the first homography matrix of the reference camera and the first camera, the second homography matrix of the first camera and the second camera, and the third homography matrix of the reference camera and the second camera. If the reference camera, the first camera, and the second camera are arranged in a line, then according to the mapping relationship, the third homography matrix of the first camera and the third camera is the product of the first homography matrix and the second homography matrix.

[0021] like Figure 2 As shown, the ROI region within the field of view of each camera is selected. The ROI region is the area within the yellow box, which defines the effective detection range. The effective detection range must at least cover the entire separation area where the package will appear. The purple-red box is the target detection box of the package, and the green box is the global fusion box of the package to be located.

[0022] The lightweight PicoDet object detection model inputs RGB images from each camera and outputs bounding boxes containing objects within the images. The model quickly and accurately locates 2D bounding boxes within images from RGB images captured by the cameras, making it suitable for multi-camera single-object separation applications. The overall architecture of the lightweight PicoDet model uses LCNet as the backbone feature extraction network, outputting feature maps at different scales from layers 3 to 5. After processing by the LCPAN feature fusion network, it outputs 96-channel features, simplifying redundant dimensions in the feature fusion branches and reducing computational cost for feature transmission while preserving multi-scale semantic information.

[0023] The detection head adopts an improved PicoHeadV2 structure, with a built-in PicoFeat feature convolution module. It features two-stage convolution stacking, fixed 96-dimensional feature input and output channels, and employs a weight-sharing design for classification and regression branches. An SE channel attention mechanism is introduced to adaptively strengthen effective feature weights and suppress interference from invalid background features. Simultaneously, a Non-Maximum Suppression (NMS) algorithm is introduced in the post-processing stage to filter and suppress the target detection boxes output by the model. Redundant detection boxes with overlap exceeding a set threshold are removed, and detection boxes with high confidence are selected as the final output. This effectively solves the problems of overlapping boxes and false detections, improving the accuracy and reliability of the target detection results. Here, the confidence level is set to 0.4, and the NMS threshold is set to 0.5. The prediction results are the vertex coordinates and center coordinates of the target candidate boxes.

[0024] During the training phase, multi-dimensional strategy optimizations are implemented. At the data augmentation level, random cropping, random horizontal flipping, random color gamut distortion, and multi-scale adaptive scaling transformations are integrated to improve the model's robustness to changes in target scale, viewpoint, and illumination. The learning rate employs a linear warm-up combined with cosine decay scheduling, coupled with a Momentum optimizer and L2 regularization constraints, to suppress overfitting and accelerate convergence. Simultaneously, unused parameter retrieval and EMA (Exponential Moving Average) strategies are enabled to smooth model training weights and improve generalization performance during inference. Ultimately, this achieves a comprehensive optimization effect of compressing model parameters, improving inference speed, and maintaining detection accuracy. Combined with NMS (Non-Maximum Suppression) bounding box suppression strategies, the accuracy of detection results is further guaranteed.

[0025] S2. Based on the package target detection box and depth data, determine the package type. If it is a small or large package, perform binarization segmentation on the depth data to obtain the initial mask region of the package. If it is a package with missing depth or a flat package, perform semantic segmentation on the package target detection box to obtain the initial mask region of the package. Then determine whether the package is an irregular package. If it is an irregular package, perform regularization processing on the initial mask region of the package and output the package mask region.

[0026] Specifically, since the RGB image and depth data are aligned, the depth data corresponding to each pixel in the target detection box of the package is extracted to generate a depth map within the box, and the depth values ​​of all pixels in the box are statistically analyzed to generate a depth histogram. Based on the depth histogram, the depth segmentation threshold between the upper surface and non-upper surface regions of the package is calculated, and the depth map within the box is binarized using the depth segmentation threshold to obtain the initial mask region of the package.

[0027] When the length or width of the target detection box region is less than 200 pixels, it is determined to be a small package, and the depth segmentation threshold calculation steps for small packages are performed: first, a 1×5 smooth convolution kernel is constructed to filter the depth histogram to eliminate peak noise, then the main peak depth value of the depth histogram is coarsely located and retrieved, and then the interval is expanded to the left and right by 30 pixels from the center of the main peak depth value, and the depth weighted average is performed to obtain the depth segmentation threshold. When the length and width of the target detection box region of the package are both greater than 200 pixels, it is determined to be a large package, and the depth segmentation threshold calculation steps for large packages are performed: starting from the maximum depth of the camera, each depth value is traversed one by one along the direction of decreasing depth, and the corresponding number of pixels is accumulated; when the accumulated number of pixels first reaches 80% of the total number of pixels in the depth histogram, the depth value traversed is determined as the depth segmentation threshold.

[0028] When the area of ​​the target detection box region with a depth of 0 is greater than 30% of the target detection box area, it is determined that there is a missing depth component; when the depth data of the package is less than 20mm, it is determined that it is a flat component; for the missing depth component and the flat component, the target detection box of the package is semantically segmented by the RGB image semantic segmentation model to obtain the initial mask area of ​​the package, so as to ensure the effectiveness and accuracy of the mask generation.

[0029] The overall architecture of the RGB image semantic segmentation model is based on PPLiteSeg, with the STDC1 network as the backbone for feature extraction. The STDC1 network, through its efficient spatial and channel feature encoding structure, achieves efficient capture of multi-scale semantic information in images while maintaining low computational cost, making it suitable for deployment on mobile devices or in scenarios with limited computing resources. The backbone network employs pre-trained weights to enhance feature extraction capabilities and accelerate training convergence.

[0030] Furthermore, the RGB image semantic segmentation model incorporates an attention feature enhancement module to adjust and fuse the multi-level features output by the backbone network. The number of output channels is 32, 64, and 128 respectively, balancing feature representation capability and computational efficiency to provide sufficient multi-scale contextual information for the subsequent segmentation head. The segmentation head employs a lightweight convolutional structure with intermediate channels of 32, 64, and 64 respectively, used to efficiently integrate features from different levels and generate the final pixel-level semantic results.

[0031] During training, the RGB image semantic segmentation model employs the cross-entropy loss function and can utilize online hard sample mining strategies to enhance its ability to identify difficult-to-segment regions. Data processing and augmentation strategies include uniform size scaling, random cropping, horizontal flipping, and perturbations of brightness, contrast, and saturation, effectively improving the model's generalization performance. The optimizer uses stochastic gradient descent (SGD) with dynamic momentum, combined with a multinomial decay learning rate strategy and a warm-up mechanism to ensure the stability and convergence speed of the training process.

[0032] After obtaining the initial mask region of the wrapper, the mask regularity is calculated. The mask regularity is defined as the ratio of the actual area of ​​the wrapper mask to the area of ​​the mask convex hull. The actual area of ​​the wrapper mask is obtained by counting the total number of all non-zero pixels in the mask region. The mask convex hull refers to the smallest convex polygon that can completely enclose all the contour pixels of the mask. First, the outer contour point set of the wrapper mask is extracted, and then the smallest convex polygon is solved based on the outer contour point set to obtain the mask convex hull. The area of ​​the mask convex hull is the total number of pixels in the area covered by the smallest convex polygon.

[0033] When the mask regularity is less than 0.85, it indicates that the package contains a large number of noise interference areas, and the package is judged to be an irregular package. Therefore, the initial mask area of ​​the package is regularized as follows: Calculate the principal direction of the mask image; align the principal direction of the mask image with the X-axis of the Cartesian coordinate system through affine transformation to obtain the rotated mask image; perform binarization processing on the rotated mask image with a threshold of 127, where pixel values ​​> 127 are determined as foreground and pixel values ​​≤ 127 are determined as background; traverse the foreground region, solve its minimum bounding rectangle, and determine the effective foreground range.

[0034] Set the grid step size to The effective foreground region is divided into grids; the half-grid offset is calculated. Starting from the top-left vertex of the smallest enclosing rectangle, with As the initial offset, according to the grid step size Traverse the grid row by row and column by column, collecting the coordinates of the center point of each cell; using the center point of each cell as the center, extract a side with a length of... The grid is defined as a square neighborhood. The number of valid foreground pixels with a value greater than 0 within this square neighborhood is counted. If the count is greater than 0, the grid is marked as valid; if the count is 0, the grid is marked as invalid. Specifically, if the neighborhood is not entirely foreground, all pixels in the neighborhood are traversed and the number of foreground pixels is counted. If any foreground pixel exists within the neighborhood, the grid is considered valid.

[0035] Record the coordinates of all valid grids and count the number of consecutive valid grids arranged horizontally at each position; record the grid positions to locate the valid area inside the package, and count the number of consecutive valid grids horizontally to characterize the area occupied by the package laterally.

[0036] A two-dimensional grid is constructed with valid grid cells as 1 and invalid grid cells as 0. The algorithm of column-by-column traversal and monotonic stack is used to find the largest rectangular region in the two-dimensional grid that is composed entirely of valid grid cells. The boundary parameters of the largest rectangular region are calculated and the top, bottom, left and right boundaries of the largest rectangular region are recorded.

[0037] Based on the boundary parameters of the largest rectangular region, combined with the grid step size Given the starting coordinates of the minimum bounding rectangle, calculate the four rectangle vertices of the minimum bounding rectangle in the rotated image; by calculating the inverse matrix of the affine transformation, map the rectangle vertices of the rotated image back to the original image coordinate system through the inverse affine transformation to obtain the regularized bounding mask region, thus achieving de-adhesion of the bounding.

[0038] S3. Combining camera intrinsic parameters, the wrapping mask area and depth data are converted into wrapping point clouds, projected onto the belt plane, and then fitted to obtain the wrapping projection frame.

[0039] By combining camera intrinsic parameters, the masked area and depth information are converted into 3D point cloud data, specifically including the following steps: Let the camera intrinsic parameter matrix be set. for: , in, Let be the equivalent focal length of the camera in the horizontal direction. This represents the camera's equivalent focal length in the vertical direction, expressed in pixels. The principal point of the camera, which is the pixel coordinate of the intersection of the optical axis and the imaging plane; the pixel position in the depth map. The corresponding depth value Then the 3D point cloud coordinates in the camera coordinate system obtained by back-projection of this pixel The calculation formula is as follows: , To reduce noise interference in point cloud data, statistical analysis filtering was applied to the extracted point cloud for denoising. A search radius of 3 was set to remove outliers around the point cloud. Simultaneously, a minimum distance threshold of 1.5 between two points was set; points in the point cloud with a distance less than this threshold were identified as duplicates to remove redundant points. The denoising effect was as follows: Figure 3 As shown.

[0040] The filtered point cloud of the package is projected onto the belt plane, and the cv::minAreaRect function in OpenCV is called to fit the minimum area rotation rectangle, thus obtaining the package projection box. The feature information of the projection box is recorded, including the left bottom, left top, right bottom, right top, center point, all contour points, package height, etc.

[0041] S4. In each row, map the projection frame of the first camera to the coordinate system of the reference camera to obtain the mapping frame of the first camera. Match the mapping frame of the first camera with the projection frame of the reference camera. Merge the successfully matched contour points into the first-level fused contour point set. Output the first-level fused projection frame based on the first-level fused contour point set and the unmatched single-frame contour point set.

[0042] If the cameras are arranged in a single row and there are no second cameras, the output of the first-level blended projection box is the enclosing global blended box.

[0043] If the cameras are arranged in a single row and there is a second camera, the projection frame of the second camera is mapped to the coordinate system of the reference camera to obtain the mapping frame of the second camera; the mapping frame of the second camera is matched with the projection frame after the first level fusion, and the contour points of the successfully matched contours are merged into the second level fusion contour point set. Based on the second level fusion contour point set and the single frame contour point set of the unmatched ones, the projection frame after the second level fusion is output as the global fusion frame.

[0044] If the cameras are arranged in a matrix and there is no second camera, map the first-level fused projection box of the second row to the coordinate system of the reference camera in the first row to obtain the mapping box of the second row; match the mapping box of the second row with the first-level fused projection box of the first row, and merge the successfully matched contour points into a row-level fused contour point set. Based on the row-level fused contour point set and the unmatched single-frame contour point set, output the row-level fused projection box as the global fused box.

[0045] Specifically, if the cameras are arranged in a single row and there is a second camera, meaning the reference camera, the first camera, and the second camera are arranged in a line, for cascaded three-camera fusion, a cascaded strategy of pre-fusion followed by final fusion is adopted to achieve globally consistent fusion. The specific implementation process is as follows: like Figure 5 As shown, with the reference camera as the reference coordinate system, the 3×3 first homography matrix between the first camera and the reference camera can establish a one-to-one mapping relationship between pixels on the image planes of the two cameras; for the projection frame under the first camera, by performing homography transformation and normalization on its four vertices point by point, the entire projection frame can be mapped to the image coordinate system of the reference camera.

[0046] Assume the pixel coordinates in the image plane of the first camera are The homogeneous coordinates of the pixels are obtained as follows: The first homography matrix from the first camera to the reference camera for: , Set the pixel homogeneous coordinates of the first camera 1 With the first homography matrix Multiplying them yields the homogeneous coordinates of the corresponding points on the transformed reference camera image plane. After normalization, the final pixel coordinates in the reference camera coordinate system are obtained as follows: The mapping frame of the first camera in the reference camera coordinate system; , , , , , , ,

[0047] After obtaining the set of mapping frames of the first camera in the coordinate system of the reference camera, the projection frame of the reference camera and the mapping frame of the first camera in the coordinate system of the reference camera are matched. The matching process includes the following steps: Iterate through all pairwise combinations of the projection box set of the reference camera and the mapping box set of the first camera, and calculate the distance feature value dis and the overlap rate feature value OR for each pair of boxes. The calculation involves determining the distance feature value *dis* for each pair of bounding boxes. The distance feature value *dis* is the sum of the absolute distances between the four vertices and one center point of the projected and mapped bounding boxes, i.e., *dis* = d1 + d2 + d3 + d4 + d5, or simply the sum of the absolute distances between the four vertices of the projected and mapped bounding boxes, i.e., *dis* = d1 + d2 + d3 + d4; d1 to d5 are the Euclidean distances between the two points. The overlap ratio feature value *OR* is defined as the ratio of the intersection area of ​​the projected and mapped bounding boxes to the smaller area of ​​the two bounding boxes. For example... Figure 4 As shown, the distance feature value dis between the yellow box and the blue box is S1=d1+d2+d3+d4, and the distance feature value dis between the yellow box and the red box is S2=d1+d2+d3+d4.

[0048] Set the valid matching pair filtering conditions: dis∈(0.1, 300), or OR>0.49, or dis∈(301, 400) and OR≥0.2. Projected boxes and mapped boxes that satisfy any of the filtering conditions are valid matching pairs; where dis∈(0.1, 300) is for small and medium packages, indicating that the two packages are relatively close; OR>0.49 is for small and medium packages, indicating that the two packages have some overlap; dis∈(301, 400) and OR≥0.2 is for large packages, indicating that the two packages have some overlap. All valid matching pairs are sorted. The sorting priority is to first compare the overlap rate feature value OR, and then rank the valid matching pairs with the larger overlap rate feature value OR first. If the overlap rate feature values ​​OR are equal, then compare the distance feature value dis, and rank the valid matching pairs with the smaller distance feature value dis first. Finally, the pair is sorted in descending order of matching priority, and the first pair is selected as the successful matching pair.

[0049] For example, given four matching data points: OR = 0.9 and dis = 10, OR = 0.8 and dis = 5, OR = 0.9 and dis = 3, and OR = 0.7 and dis = 1, the data is first sorted by OR from highest to lowest. The two data points with OR = 0.9 are placed at the front, the one with OR = 0.8 is in the middle, and the one with OR = 0.7 is at the back. The two data points with the same OR = 0.9 are then sorted by dis from smallest to largest, with the one with dis = 3 placed before the one with dis = 10. This results in a sorted order based on matching priority from best to worst. The first pair is selected as the successful match. After a successful match, the matched box is marked to avoid duplicate associations, achieving a one-to-one unique match, and the successful match is recorded.

[0050] For unmatched reference camera projection frames and first camera mapping frames, mark them as unmatched to avoid losing the bounding box, and set both unmatched projection frames and mapping frames as single frames.

[0051] For the projected frame of the successfully paired reference camera and the mapped frame of the first camera, the original contour points of the reference camera are first merged with the contour points of the first camera after projection mapping, resulting in a fused contour point set of the reference camera and the first camera. For isolated frames, the contour points of the isolated frame are directly retained. It saves the vertex information of the projection frame, the fused contour point set, and the single frame contour point set to form the first-level fused contour point set of the reference camera and the first camera, and outputs the projection frame after the first-level fusion. The first-stage fused projection frame is then finally fused with the second camera's projection frame to output a globally fused frame that is consistent across all three cameras. This process includes the following steps: Using the reference camera as the global reference coordinate system, the third homography matrix of the reference camera and the second camera is called to map all the projection frames of the second camera to the reference camera coordinate system, generating the corresponding set of mapping frames.

[0052] Iterate through all pairwise combinations of the first-level fused projection box set of the reference camera and the mapping box set of the second camera, calculate the distance feature value dis and the overlap rate feature value OR for each pair of boxes; filter all valid matching pairs according to the effective matching pair filtering criteria and sort them to obtain the successfully matched pairs, and record the successfully matched pairs and the unsuccessfully matched individual boxes. For the first-level fused projection frame and the second camera mapping frame that have been successfully paired, the first-level fused contour point set of the reference camera is first merged with the contour points after projection mapping of the second camera to obtain the merged second-level fused contour point set of the reference camera and the second camera.

[0053] The cv::minAreaRect function in OpenCV is used to fit the final second-level fused contour point set and the single-frame contour point set, and outputs the enclosed global fused bounding box.

[0054] The three-camera fusion method can be directly reused in dual / quad-camera deployment scenarios, offering strong scalability. This layout can cover the entire area of ​​the logistics sorting conveyor belt. Using the reference camera as the reference coordinate system, it maximizes the reuse of core technologies such as cross-camera coordinate mapping and box matching fusion through a cascading strategy of fusing the reference camera and the first camera, and then fusing the second camera. This is the optimal implementation plan for three-camera scenarios.

[0055] In dual-camera scenarios: only the first-level fusion step of the three cameras is executed, skipping the final fusion step, and the projection box after the first-level fusion is directly output as the global fusion box; core coordinate mapping, matching fusion and other technologies can reuse the three-camera algorithm.

[0056] Four-camera scenario: Adapts to 2×2 arrangement layout, core technology is fully reused, only requiring extension of the three-camera cascading strategy to four-camera fusion. First, horizontal cascading within the same row is performed, then vertical global stitching between columns is performed, and finally global final fusion is performed. Only the pairwise camera homography matrix calibration and global fusion steps need to be added to quickly adapt to four-camera deployment. For example, Figure 7 As shown, the four cameras include camera 0, camera 1, camera 2 and camera 3. Camera 0 and camera 1 obtain the first-level fusion 01, and camera 2 and camera 3 obtain the second-level fusion 23. The first-level fusion 01 and the second-level fusion 23 are fused to obtain the final fused global image.

[0057] S5. Filter the global merge boxes, remove redundant boxes, and output the coordinate information of each merged box. Specifically, first, traverse all global merge boxes, calculate the IOU between each pair of global merge boxes and the area of ​​each box. If the IOU is greater than 0.8 and the area of ​​a box is less than 500, it is determined to be a duplicate redundant box and removed. After deduplication, perform global ROI boundary constraint processing on the remaining global merge boxes, specifically: Obtain the coordinates of the four vertices of the global bounding box to be processed, and set the boundary range of the ROI. The ROI boundary is limited by the coordinates of the top left and bottom right vertices. Detect each of the four edges of the global fusion box, count the number of vertices on each edge that exceed the ROI boundary, and record the position of all out-of-bounds vertices and their respective boundary directions, which are up, down, left, and right. like Figure 6 As shown, based on the total number of out-of-bounds vertices and their distribution on the four edges, corresponding correction operations are performed, specifically: If there is only a single vertex that exceeds the boundary of the ROI, then the vertex that exceeds the boundary is vertically projected onto the corresponding ROI boundary line along its direction of exceeding the boundary, while keeping the other coordinate of the vertex unchanged. The projected vertex is used as the corrected vertex to form the corrected global fusion box. If two vertices in the same frame both exceed the same boundary of the ROI, then calculate the two intersection points of the outline segment of the global fusion box with the boundary of the ROI, replace the two original out-of-bounds vertices with the two intersection points, and leave the other vertices unchanged to form the corrected global fusion box. If two out-of-bounds vertices are located on two different boundaries of the ROI, then each out-of-bounds vertex is vertically projected onto the corresponding ROI boundary line along its respective out-of-bounds direction to obtain two corrected vertices, while the remaining vertices remain unchanged; thus forming the corrected global fusion box. All the corrected vertices are combined in their original order to generate and output a corrected bounding box vertex sequence that conforms to the ROI boundary constraints; then, they are sorted in descending order according to the maximum Y coordinate of the package vertices, and a unique number is assigned to each package in turn. Finally, the package number and the corrected global fusion box are output to the sorting equipment.

[0058] This invention aims to address the shortcomings and deficiencies of existing parcel separation and pose localization technologies in dynamic belt conveyor sorting scenarios, and provides a real-time, efficient, accurate, and reliable multi-camera-based method for parcel separation and localization. Specifically, the technical problems to be solved are as follows: First, it addresses the limitations of existing single-camera field-of-view coverage and the lack of a unified coordinate mapping and accurate fusion mechanism for multi-camera collaboration, resulting in poor consistency and large errors in parcel localization across camera fields of view, and the inability to achieve accurate coverage of the entire belt conveyor. Second, it addresses the shortcomings of existing technologies that rely heavily on single visual or depth modal information, making them less adaptable to complex sorting scenarios such as parcel adhesion, missing depth data, and surface reflection, leading to false detections, missed detections, and insufficient robustness. Third, it addresses the shortcomings of existing multi-camera fusion algorithms that are cumbersome, computationally complex, lack scenario-based lightweight optimization, and whose single-frame processing time cannot keep pace with the high-speed dynamic belt conveyor sorting rhythm, resulting in insufficient real-time performance. Fourth, it addresses the shortcomings of existing technologies that struggle to accurately output parcel pose information, making it difficult to meet the precision operation requirements of automated sorting equipment. This invention enables full-width field-of-view coverage of packages in dynamic conveyor belt scenarios, high-precision pose positioning, robust recognition in complex scenarios, and efficient collaborative processing, providing stable and reliable technical support for automated logistics sorting systems.

Claims

1. A method for locating a single-item package based on multiple cameras, characterized in that, Includes the following steps: S1. Multiple cameras are arrayed above the single-piece separation area, with a reference camera, a first camera, and an optional second camera in each row; images and depth data of all cameras are acquired synchronously, and the target detection box is identified based on the image of each camera; S2. Based on the package target detection box and depth data, determine the package type. If it is a small or large package, perform binarization segmentation on the depth data to obtain the initial mask region of the package. If it is a package with missing depth or a flat package, perform semantic segmentation on the package target detection box to obtain the initial mask region of the package. Then determine whether the package is an irregular package. If it is an irregular package, perform regularization processing on the initial mask region of the package and output the package mask region. S3. Combining camera intrinsic parameters, the wrapping mask area and depth data are converted into wrapping point clouds, projected onto the belt plane, and then fitted to obtain the wrapping projection box. S4. In each row, the projection frame of the first camera is mapped to the coordinate system of the reference camera to obtain the mapping frame of the first camera. The mapping frame of the first camera and the projection frame of the reference camera are matched. The successfully matched contour points are merged into the first-level fused contour point set. The first-level fused projection frame is output based on the first-level fused contour point set and the unmatched single-frame contour point set. If the cameras are arranged in a single row and there are no second cameras, the output of the first-level fused projection box is the wrapping global fused box; If the cameras are arranged in a single row and there is a second camera, the projection frame of the second camera is mapped to the coordinate system of the reference camera to obtain the mapping frame of the second camera; the mapping frame of the second camera is matched with the projection frame after the first level fusion, and the contour points of the successfully matched contours are merged into the second level fusion contour point set. Based on the second level fusion contour point set and the single frame contour point set of the unmatched ones, the projection frame after the second level fusion is output as the global fusion frame. If the cameras are arranged in a matrix and there is no second camera, map the first-level fused projection box of the second row to the coordinate system of the reference camera in the first row to obtain the mapping box of the second row; match the mapping box of the second row with the first-level fused projection box of the first row, and merge the successfully matched contour points into a row-level fused contour point set. Based on the row-level fused contour point set and the unmatched single-frame contour point set, output the row-level fused projection box as the global fused box. S5. Perform deduplication and global ROI boundary constraint processing on the global fused bounding box, generate the corrected bounding box vertex sequence, assign wrapper numbers, and output the wrapper numbers and the corrected global fused bounding box.

2. The single-item separation and positioning method based on multiple cameras according to claim 1, characterized in that, In step S2, the type of package is determined based on the target detection box and depth data of the package. Specifically, the steps include: when the length or width of the target detection box area of ​​the package is less than 200 pixels, it is determined to be a small package. When the length and width of the target detection box area of ​​the package are both greater than 200 pixels, it is judged as a large package; when the area of ​​the target detection box area with a depth data of 0 is greater than 30% of the target detection box area, it is judged as a package with missing depth; when the depth data of the package is less than 20mm, it is judged as a flat package.

3. The single-item separation package positioning method based on multiple cameras according to claim 1, characterized in that, In step S2, if the package is small or large, the depth data is binarized to obtain the initial mask area of ​​the package. This includes the following steps: For small or large packages, extract the depth data corresponding to each pixel within the box to generate a depth map within the box, and statistically analyze the depth values ​​of all pixels within the box to generate a depth histogram. Based on the depth histogram, the depth segmentation threshold between the upper surface and non-upper surface regions of the package is obtained. The depth segmentation threshold is then used to perform binarization segmentation of the depth map within the box to obtain the mask region of the package. The method for obtaining the depth segmentation threshold of small packages is as follows: First, a 1×5 smooth convolution kernel is constructed to filter the depth histogram. Then, the main peak depth value of the depth histogram is coarsely located and retrieved. Then, the interval is expanded to the left and right by 30 pixels from the center of the main peak depth value, and the depth weighted average is performed to obtain the depth segmentation threshold. The method for determining the depth segmentation threshold of a large package is as follows: starting from the maximum depth of the camera, traverse each depth value one by one along the direction of decreasing depth and accumulate the corresponding number of pixels; when the accumulated number of pixels first reaches 80% of the total number of pixels in the depth histogram, the depth value traversed is determined as the depth segmentation threshold.

4. The single-item separation package positioning method based on multiple cameras according to claim 1, characterized in that, The step S2, determining whether the package is an irregular package, specifically includes the following steps: Based on the initial mask area of ​​the package, the mask regularity is calculated. When the mask regularity is less than 0.85, it is judged as an irregular package; when the mask regularity is not less than 0.85, it is judged as a regular package. Mask regularity is defined as the ratio of the actual area of ​​the mask to the area of ​​the mask convex hull. The actual area of ​​the mask is obtained by counting the total number of all non-zero pixels in the mask region. The mask convex hull is the smallest convex polygon that can completely enclose all the contour pixels of the mask. First, the outer contour point set of the mask is extracted, and then the smallest convex polygon is solved based on the outer contour point set to obtain the mask convex hull. The area of ​​the mask convex hull is the total number of pixels in the region covered by the smallest convex polygon.

5. The single-item separation package positioning method based on multiple cameras according to claim 1, characterized in that, Step S2 involves regularizing the initial mask area, specifically including the following steps: Calculate the principal direction of the initial mask region of the package, and align the principal direction of the initial mask region of the package with the X-axis of the Cartesian coordinate system through affine transformation to obtain the rotated mask region; perform binarization processing on the rotated mask region with 127 as the threshold, where pixel values ​​> 127 are determined as foreground and pixel values ​​≤ 127 are determined as background; traverse the foreground and solve its minimum bounding rectangle to determine the effective foreground region of the package. Set the grid step size to The effective foreground area of ​​the package is divided into grids; Calculate half-grid offset Starting from the top-left vertex of the smallest enclosing rectangle, with As the initial offset, according to the grid step size Traverse the grid row by row and column by column, collecting the coordinates of the center point of each cell; using the center point of each cell as the center, extract a side with a length of... Determine the square neighborhood; count the pixel values ​​within this square neighborhood. The number of effective foreground pixels, if the number If the number is 0, the grid is marked as a valid grid; if the number is 0, the grid is marked as an invalid grid. Record the coordinates of all valid grids and count the number of consecutive valid grids arranged horizontally at each position; record the grid positions to locate the valid area inside the package, and count the number of consecutive valid grids horizontally to characterize the area occupied by the package laterally. A two-dimensional grid is constructed with valid grid cells as 1 and invalid grid cells as 0. The algorithm of column-by-column traversal and monotonic stack is used to find the largest rectangular region in the two-dimensional grid that is composed entirely of valid grid cells. The boundary parameters of the largest rectangular region are calculated and the top, bottom, left and right boundaries of the largest rectangular region are recorded. Based on the boundary parameters of the largest rectangular region, combined with the grid step size Given the starting coordinates of the minimum bounding rectangle, calculate the four rectangle vertices of the minimum bounding rectangle in the rotated image; by calculating the inverse matrix of the affine transformation, map the rectangle vertices of the rotated image back to the original image coordinate system through the inverse affine transformation to obtain the regularized bounding mask region.

6. The single-item separation package positioning method based on multiple cameras according to claim 1, characterized in that, In step S4, the projection frame is mapped onto the reference camera coordinate system to obtain the mapped frame, which specifically includes the following steps: Let the pixel coordinates of the projection box in the image plane be... The homogeneous coordinates of the pixels are obtained as follows: homography matrix for: ， Homogeneous coordinates of pixels With homography matrix Multiply to obtain the transformed homogeneous coordinates. After normalization, the final pixel coordinates in the reference camera coordinate system are obtained as follows: and mapping box; ， ， ， ， ， ， ， in These are the final pixel coordinates in the unnormalized reference camera coordinate system.

7. The single-item separation package positioning method based on multiple cameras according to claim 1, characterized in that, In step S4, the projection box and the mapping box are matched, which specifically includes the following steps: Iterate through all pairwise combinations of the projected bounding box set and the mapped bounding box set, calculate the distance feature value dis and the overlap rate feature value OR for each pair of boxes, and set the effective matching pair filtering conditions to obtain all effective matching pairs; sort all effective matching pairs from best to worst, and select the best combination as the successful matching pair; after a successful match, mark the matched box, and mark the unmatched projected bounding box and mapped bounding box as unmatched, and set the unmatched projected bounding box and mapped bounding box as a single box; The distance feature value dis for each pair of boxes is calculated. The distance feature value dis is the sum of the absolute distances of the four vertices and one center point of the projected box and the mapped box, i.e., dis = d1 + d2 + d3 + d4 + d5, or the distance feature value dis is the sum of the absolute distances of the four vertices of the projected box and the mapped box, i.e., dis = d1 + d2 + d3 + d4; d1 to d5 are the Euclidean distances between two points; the overlap rate feature value OR is defined as the ratio of the intersection area of ​​the projected box and the mapped box to the smaller area of ​​the two boxes. Set the filtering conditions for valid matching pairs: dis∈(0.1, 300), or OR>0.49, or dis∈(301, 400) and OR≥0.

2. Projected boxes and mapped boxes that meet any of the filtering conditions are valid matching pairs. Sort all valid matching pairs from best to worst. Specifically, first compare the overlap rate feature value OR, and rank the valid matching pairs with the larger overlap rate feature value OR first. If the overlap rate feature values ​​OR are equal, then compare the distance feature value dis, and rank the valid matching pairs with the smaller distance feature value dis first. Finally, the order from best to worst is formed.

8. The single-item separation package positioning method based on multiple cameras according to claim 1, characterized in that, Step S5 involves deduplication of the global fusion box and global ROI boundary constraint processing for vertex coordinates, specifically including the following steps: Traverse all global merge boxes, calculate the IOU between each pair of global merge boxes and the area of ​​a single box. If the IOU is greater than 0.8 and the area of ​​a single box is less than 500, it is determined to be a duplicate redundant box and removed. Obtain the coordinates of the four vertices of the global bounding box to be processed, and set the boundary range of the ROI. The ROI boundary is limited by the coordinates of the top left and bottom right vertices. Detect each of the four edges of the global fusion box, count the number of vertices on each edge that exceed the ROI boundary, and record the position of all out-of-bounds vertices and their respective boundary directions, which are up, down, left, and right. Based on the total number of out-of-bounds vertices and their distribution on the four edges, perform the corresponding correction operation, specifically: If there is only a single vertex that exceeds the boundary of the ROI, then the vertex that exceeds the boundary is vertically projected onto the corresponding ROI boundary line along its direction of exceeding the boundary, while keeping the other coordinate of the vertex unchanged. The projected vertex is used as the corrected vertex to form the corrected global fusion box. If two vertices in the same frame both exceed the same boundary of the ROI, then calculate the two intersection points of the outline segment of the global fusion box with the boundary of the ROI, replace the two original out-of-bounds vertices with the two intersection points, and leave the other vertices unchanged to form the corrected global fusion box. If two out-of-bounds vertices are located on two different boundaries of the ROI, then each out-of-bounds vertex is vertically projected onto the corresponding ROI boundary line along its respective out-of-bounds direction to obtain two corrected vertices, while the remaining vertices remain unchanged; this forms the corrected global fusion box.

9. An electronic device, characterized in that, Including processor and storage media; The storage medium is used to store instructions; The processor is configured to operate according to the instructions to perform the steps of the method according to any one of claims 1 to 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program implements the steps of the method according to any one of claims 1 to 8.

Images