Multi-target recognition and positioning method and system for robot

Abstract

The application relates to the technical field of identification and positioning, and particularly discloses a multi-target identification and positioning method and system for a robot, which first utilizes a camera and a radar device to convert discrete three-dimensional point cloud data into a dense depth map in real time, which is pixel-level aligned with a two-dimensional video frame. Based on this, the image is dynamically segmented into multiple depth regions, and a customized slicing and scale transformation strategy is generated for each region. Then, far and near targets with a large size difference in the original image are respectively converted into scale-appropriate image slices for processing. Finally, through unified detection and result inversion on these scale-normalized slices and combined with the robot pose, three-dimensional tracking is performed to realize robust identification and accurate positioning of multi-scale targets, and the visual scale variation caused by the distance variation of the targets can be actively eliminated, so that robust identification and accurate positioning of multi-scale targets are realized.

Description

Technical Field

[0001] This application relates to the field of identification and positioning technology, and more specifically, to a method and system for multi-target identification and positioning of a robot. Background Technology

[0002] With the rapid development of artificial intelligence and automation technologies, intelligent robots are increasingly penetrating key areas such as autonomous driving, smart logistics, and security patrol. A core prerequisite for robots to operate autonomously in complex, dynamic, and unstructured environments is their ability to accurately and comprehensively perceive their surroundings and identify and locate various static and dynamic targets in real time. This requires robots to possess efficient and reliable multi-target recognition and localization capabilities, thereby providing crucial decision-making support for subsequent advanced tasks such as path planning, autonomous obstacle avoidance, and human-robot interaction, ensuring the safety and efficiency of their operation.

[0003] In existing technologies, a multi-sensor fusion strategy is typically employed, combining rich texture and color information from cameras with precise 3D spatial information from devices such as LiDAR, and then processing the fused data using deep learning algorithms. However, in practical applications, these solutions generally face the problem of significant scale differences caused by changes in target distance. Specifically, when a robot performs environmental perception, distant targets occupy only a small pixel area on the imaging plane, with weak and blurry visual features, making it difficult for mainstream detection models to effectively capture them, resulting in missed detections. At the same time, large targets at close range may occupy too much space in the imaging image, even exceeding the field of view of a single frame, causing truncation and preventing detection algorithms from acquiring their complete global features, leading to incomplete or failed recognition. This lack of adaptability to extreme scale changes significantly reduces the robustness and comprehensiveness of traditional detection frameworks when facing complex scenes with both near and far targets, severely restricting the robot's environmental understanding and operational safety in the real world.

[0004] Therefore, there is a need to provide an optimized method and system for multi-target recognition and localization of robots to solve the above-mentioned technical problems. Summary of the Invention

[0005] This application is made in order to solve the above-mentioned technical problems.

[0006] According to one aspect of this application, a method for multi-target recognition and localization of a robot is provided, comprising:

[0007] Acquire raw video frames and raw point clouds collected by cameras and radar devices;

[0008] A dense depth map is generated from the original point cloud based on camera extrinsic and intrinsic parameters.

[0009] Dynamic region segmentation and slicing strategies are used to generate a list of image slices from dense depth maps.

[0010] Based on the image slice processing list, the original video frames are adaptively sliced ​​to obtain an image slice set and a slice metadata set;

[0011] Based on the slice metadata set, a unified detection and coordinate inversion are performed on the image slice set to obtain the frame detection result;

[0012] Based on the robot's pose and the original point cloud, the frame detection results are used for 3D localization and tracking to obtain the final target list.

[0013] According to another aspect of this application, a multi-target recognition and localization system for a robot is provided, comprising:

[0014] The raw data acquisition module is used to acquire raw video frames and raw point clouds collected by cameras and radar devices;

[0015] The depth map generation module is used to generate a dense depth map from the original point cloud based on camera extrinsic and intrinsic parameters.

[0016] The depth map dynamic segmentation module is used to perform dynamic region segmentation and slicing strategy generation on dense depth maps to obtain an image slice processing list.

[0017] The adaptive slicing module is used to adaptively slice the original video frames based on the image slice processing list to obtain a set of image slices and a set of slice metadata.

[0018] The unified detection and coordinate inversion module is used to perform unified detection and coordinate inversion on the image slice set based on the slice metadata set to obtain the frame detection result;

[0019] The 3D localization and tracking module is used to perform 3D localization and tracking on the frame detection results based on the robot's pose and the original point cloud to obtain the final target list.

[0020] This application has at least the following technical effects:

[0021] Compared with existing technologies, the multi-target recognition and localization method and system for robots provided in this application first utilizes camera and radar equipment to convert discrete 3D point cloud data into dense depth maps aligned with the pixel level of 2D video frames in real time. Based on this, the image is dynamically segmented into multiple depth regions, and customized slicing and scale transformation strategies are generated for each region. Then, distant and near targets with significantly different sizes in the original image are transformed into image slices of appropriate scale for processing. Finally, by uniformly detecting and inverting the results of these scale-normalized slices, and combining this with 3D tracking based on the robot's pose, robust recognition and accurate localization of multi-scale targets are achieved. This proactively eliminates drastic changes in visual scale caused by variations in target distance, thus enabling robust recognition and accurate localization of multi-scale targets. Attached Figure Description

[0022] The above and other objects, features, and advantages of this application will become more apparent from the more detailed description of the embodiments of this application in conjunction with the accompanying drawings. The drawings are provided to further illustrate the embodiments of this application and form part of the specification. They are used together with the embodiments of this application to explain this application and do not constitute a limitation thereof. In the drawings, the same reference numerals generally represent the same components or steps.

[0023] Figure 1 This is a flowchart of a multi-target recognition and localization method for a robot according to an embodiment of this application.

[0024] Figure 2 This is a data flow diagram of a multi-target recognition and localization method for robots according to an embodiment of this application.

[0025] Figure 3 This is a flowchart of sub-step S2 of the multi-target recognition and localization method for robots according to an embodiment of this application.

[0026] Figure 4 This is a flowchart of sub-step S3 of the multi-target recognition and localization method for robots according to an embodiment of this application.

[0027] Figure 5 This is a flowchart of sub-step S33 of the multi-target recognition and localization method for robots according to an embodiment of this application.

[0028] Figure 6 This is a flowchart of sub-step S5 of the multi-target recognition and localization method for robots according to an embodiment of this application.

[0029] Figure 7 This is a flowchart of sub-step S52 of the multi-target recognition and localization method for robots according to an embodiment of this application.

[0030] Figure 8This is a block diagram of a multi-target recognition and localization system for a robot according to an embodiment of this application. Detailed Implementation

[0031] As indicated in this application and claims, unless the context clearly indicates otherwise, the words "a," "an," "an," and / or "the" are not specifically singular and may include plural forms. Generally speaking, the terms "comprising" and "including" only indicate the inclusion of explicitly identified steps and elements, which do not constitute an exclusive list, and the method or apparatus may also include other steps or elements.

[0032] While this application makes various references to certain modules of the systems according to embodiments of this application, any number of different modules can be used and run on user terminals and / or servers. The modules described are merely illustrative, and different aspects of the systems and methods may use different modules.

[0033] Flowcharts are used in this application to illustrate the operations performed by the system according to embodiments of this application. It should be understood that the preceding or following operations are not necessarily performed in exact order. Instead, various steps can be processed in reverse order or simultaneously as needed. Furthermore, other operations can be added to these processes, or one or more steps can be removed from them.

[0034] Hereinafter, exemplary embodiments according to this application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this application, and not all embodiments of this application. It should be understood that this application is not limited to the exemplary embodiments described herein.

[0035] This application proposes a method for multi-target recognition and localization of robots, specifically:

[0036] Figure 1 This is a flowchart of a multi-target recognition and localization method for a robot according to an embodiment of this application. Figure 2 This is a data flow diagram of a multi-target recognition and localization method for robots according to an embodiment of this application. Figure 1 and Figure 2As shown, the multi-target recognition and localization method for the robot includes the following steps: S1, acquiring original video frames and original point clouds collected by camera and radar devices; S2, generating a dense depth map from the original point cloud based on camera extrinsic and intrinsic parameters; S3, performing dynamic region segmentation and slicing strategy generation on the dense depth map to obtain an image slice processing list; S4, adaptively slicing the original video frames based on the image slice processing list to obtain an image slice set and a slice metadata set; S5, performing unified detection and coordinate inversion on the image slice set based on the slice metadata set to obtain frame detection results; S6, performing 3D localization and tracking on the frame detection results based on the robot pose and the original point cloud to obtain the final target list.

[0037] Specifically, in step S1, the original video frames and original point clouds collected by the camera and radar equipment are acquired. Since the robot's multi-target recognition and localization relies on both the target's visual appearance information (for category recognition and feature matching) and three-dimensional spatial position information (for precise coordinate calculation), the original video frames acquired by the camera can intuitively present the visual features of the target object, while the original point cloud collected by the radar equipment can provide information such as the target object's spatial position and shape. Therefore, this application acquires original video frames covering the observation scene using a camera, and simultaneously acquires the corresponding original point cloud using a radar equipment. This provides a comprehensive and original data foundation for subsequent analysis and processing, improving the accuracy and reliability of target object-related processing and meeting the need for precise perception of target objects in complex scenarios.

[0038] In the implementation process, the hardware deployment and spatial calibration of the camera and radar equipment are first completed, and the transformation relationship between their coordinate systems is established through feature matching of the calibration board. Time synchronization is achieved based on a unified clock source, and the sensor clock accuracy is calibrated to the microsecond level using the PTP protocol. At the same time, the start time of camera exposure and radar scanning is controlled by hardware trigger signals. Raw video frames are transmitted in real time in RAW format via Ethernet interface, and radar point cloud data is parsed into a three-dimensional coordinate data stream by a dedicated driver. Finally, the data transmitted to the processing unit is sorted according to the acquisition time sequence, stored in separate folders and associated with the same timestamp, and synchronously supplemented with equipment calibration parameters and environmental metadata to ensure the spatiotemporal alignment accuracy during subsequent data fusion.

[0039] Specifically, in step S2, a dense depth map is generated from the original point cloud based on camera extrinsic and intrinsic parameters. Since the original point cloud is discrete 3D data, belonging to a different data dimension and having no pixel-level correlation with the 2D video frames captured by the camera, it cannot directly provide depth information for subsequent dynamic region segmentation. Therefore, this application further combines camera extrinsic and intrinsic parameters to generate a dense depth map from the original point cloud, aligned with the pixel level of the video frames. This constructs a complete depth information carrier corresponding to the pixel level of the video frames, providing continuous and accurate depth data support for subsequent depth-based dynamic region segmentation and slicing strategies. This avoids region segmentation deviations due to missing depth information and ensures the quality of basic data for multi-scale target processing.

[0040] in, Figure 3 This is a flowchart of sub-step S2 of the multi-target recognition and localization method for robots according to an embodiment of this application. Figure 3 As shown, step S2 includes the following steps: S21, performing coordinate system transformation on each 3D point in the original point cloud based on camera extrinsic parameters to obtain an aligned point cloud; S22, projecting the aligned point cloud based on camera intrinsic parameters to obtain a sparse depth map; S23, performing densification processing on the sparse depth map to obtain the dense depth map.

[0041] Specifically, in step S21, a coordinate system transformation is performed on each 3D point in the original point cloud based on the camera extrinsic parameters to obtain an aligned point cloud. Since the hardware installation positions of the radar and camera differ, the original point cloud is acquired based on the radar coordinate system, while the video frames are generated based on the camera coordinate system. Because their spatial references are different, directly using them would result in a mismatch between 3D points and 2D pixels. Therefore, this application further performs a coordinate system transformation operation on each 3D point in the original point cloud based on pre-calibrated camera extrinsic parameters to unify the point cloud data to the camera coordinate system, eliminating spatial positional deviations between sensors.

[0042] In a specific example of this application, step S21 includes: performing a coordinate system transformation on each 3D point in the original point cloud using the following formula:

[0043]

[0044] in, For camera parameters, For each 3D point in the original point cloud, To ensure that the spatial position of each 3D point in the aligned point cloud is accurately matched with the pixel position in the video frame when the point cloud is subsequently projected onto the image plane, this avoids depth information mapping errors caused by coordinate system misalignment and provides a correct coordinate reference for the spatial association of subsequent depth map generation and target detection results.

[0045] Specifically, in step S22, the aligned point cloud is projected based on camera intrinsic parameters to obtain a sparse depth map. Since the aligned point cloud is still three-dimensional data, it cannot directly correspond one-to-one with the pixels of a two-dimensional video frame. Therefore, this application further performs a two-dimensional projection operation on the aligned point cloud based on imaging parameters such as focal length and principal point contained in the camera intrinsic parameters. This maps each three-dimensional point to a specific pixel position on the image plane and records the depth value corresponding to that pixel, forming a sparse depth map.

[0046] In a specific example of this application, step S22 includes: projecting the aligned point cloud based on camera intrinsic parameters using the following formula:

[0047]

[0048] in, For depth value, For camera internal parameters, For each 3D point in the aligned point cloud, As a homogeneous coordinate vector, it provides a basic framework for subsequent depth map densification processing, ensuring that depth information can be called by subsequent modules in units of pixels, avoiding interruption of the segmentation process due to data format mismatch, while preserving the position and numerical accuracy of valid depth points.

[0049] Specifically, in step S23, the sparse depth map is densified to obtain the dense depth map. Since the sparse depth map only contains the pixel depth values ​​covered by the point cloud projection, there are numerous holes in the uncovered areas. Therefore, this application further employs a depth completion algorithm to fill the hole areas in the sparse depth map, thereby eliminating the discontinuity of depth information and generating a dense depth map where each pixel has a valid depth value. This provides continuous and complete depth information for dynamic region segmentation, ensuring accurate boundary division of far, mid, and near-field regions, avoiding region omissions or misclassifications due to holes, and providing uniform depth support for subsequent slicing strategies, ensuring that targets of different scales are included in the correct processing area.

[0050] In the specific implementation process, firstly, a neighborhood search is performed on the sparse depth map to find the effective depth value within a 5×5 area around each hole pixel, and the average of the effective depth values ​​is calculated as the initial filling value. Secondly, morphological closing operations are applied to the initially filled depth map. A 3×3 cross-shaped structuring element is first used for dilation to further cover the small holes, and then an erosion operation is performed to restore the depth contour of the target. Then, the validity of the processed depth map is verified, and outliers that still exist after filling, such as pixels whose depth difference from the surrounding area exceeds a preset threshold, are removed and reassigned the value to the average of the surrounding effective depths. Finally, a dense depth map with effective depth values ​​for all pixels is output, and the depth continuity of this depth map meets the requirements for region segmentation.

[0051] Specifically, in step S3, dynamic region segmentation and slicing strategy generation are performed on the dense depth map to obtain an image slice processing list. While the dense depth map contains complete pixel-level depth information, targets in different depth regions exhibit significant scale differences in the image. Using a uniform processing method cannot simultaneously adapt to the detection needs of both distant small targets and near large targets, easily leading to missed detections or incomplete recognition. Therefore, this application further dynamically divides the dense depth map according to depth features and generates customized slicing strategies for the target scale characteristics of different regions. This provides a clear operational basis for subsequent image slicing processing, ensuring that targets in each region can be included in the slice range adapted to their scale, avoiding scale adaptation deviations caused by uniform processing. It also lays the foundation for subsequent adaptive slicing and scale normalization, improving the completeness and accuracy of multi-scale target detection.

[0052] in, Figure 4 This is a flowchart of sub-step S3 of the multi-target recognition and localization method for robots according to an embodiment of this application. Figure 4 As shown, step S3 includes the following steps: S31, performing pixel-level depth partitioning and mask generation on the dense depth map based on a depth threshold set to obtain a region mask; S32, generating adaptive slice coordinates based on the region mask and slice parameter set to obtain a preliminary slice list, wherein each element in the preliminary slice list includes a slice rectangle and the region ID corresponding to the slice; S33, calculating a slice scaling factor for each element in the preliminary slice list based on the optimal observation depth to obtain the image slice processing list.

[0053] Specifically, in step S31, based on a depth threshold set, pixel-level depth partitioning and mask generation are performed on the dense depth map to obtain a region mask. Since the depth values ​​in the dense depth map are continuously distributed, it is impossible to directly distinguish the target areas at different distances. Therefore, this application further classifies each pixel in the dense depth map according to its depth value based on the depth threshold set, generating a region mask to mark the region's affiliation. This transforms continuous depth information into discrete, distinguishable region features, clearly defining the spatial boundaries of near, mid, and far-field regions. This provides a spatial positioning basis for subsequent generation of adapted slicing strategies for different regions, avoiding slicing coverage deviations caused by blurred region boundaries, and ensuring that subsequent processing accurately applies to the target region at the corresponding scale.

[0054] In the specific implementation process, the first step is to obtain a set of depth thresholds, which includes upper and lower limits for near-field, mid-field, and far-field regions. For example, the near-field region can be set to 0-10 meters, the mid-field region to 10-40 meters, and the far-field region to 40-120 meters. Then, the dense depth map is traversed pixel by pixel, reading the depth value of each pixel. Pixels with depth values ​​within the near-field threshold range are marked as near-field region IDs (e.g., 0), those within the mid-field threshold range are marked as mid-field region IDs (e.g., 1), and those within the far-field threshold range are marked as far-field region IDs (e.g., 2). Pixels with depth values ​​exceeding the far-field threshold or invalid IDs are marked as invalid IDs. Finally, all pixel region IDs are integrated into a two-dimensional matrix with the same size as the dense depth map, forming a region mask. This mask can intuitively reflect the spatial distribution of each depth region in the image, and the region affiliation of each pixel is unique and clear.

[0055] Specifically, in step S32, adaptive slice coordinates are generated based on the region mask and the slice parameter set to obtain a preliminary slice list. Each element in the preliminary slice list includes a slice rectangle and the region ID corresponding to the slice. Since the region mask only defines the spatial range of each depth region and lacks specific slice operation coordinates, it cannot directly guide image slicing processing. Furthermore, the target scale differences in different regions require different parameters such as slice size and overlap rate, necessitating the generation of targeted slice coordinates. Therefore, this application further combines the spatial distribution of the region mask with the customized parameters of the slice parameter set to generate suitable slice coordinates for each depth region and associate them with the corresponding region ID. This forms a preliminary slice list containing specific operation information, ensuring that the spatial position and size of each slice are adapted to the target scale of the corresponding region. Simultaneously, by associating the region ID with subsequent processing strategies, it avoids incomplete target coverage caused by mismatch between slice and region characteristics, providing basic slice information for subsequent scaling factor calculation.

[0056] In the specific implementation process, the slice size and overlap rate parameters corresponding to each region are first extracted from the slice parameter set. The parameters for the near-field region are a single bounding rectangle generation rule, while the parameters for the mid-field and far-field regions are the gridded slice size and overlap rate. Then, for the near-field region, the smallest bounding rectangle of the near-field ID pixel is found in the region mask, and the coordinates and size of this rectangle are calculated as the slice coordinates for the near-field region. For the mid-field and far-field regions, gridded slices are generated in the corresponding areas of the region mask according to the corresponding slice size and overlap rate. The coordinates of each slice are calculated, and the overlap between the slice and its corresponding region is checked. Slices with an overlap rate below a threshold are discarded. Finally, the coordinates of all valid slices are integrated with their corresponding region IDs to form a preliminary slice list. Each element in the list contains complete slice rectangle information and a region affiliation identifier.

[0057] Specifically, in step S33, based on the optimal observation depth, a slice scaling factor is calculated for each element in the preliminary slice list to obtain the image slice processing list. Since the slices in the preliminary slice list are adapted to the spatial range of the region, the actual scale of the targets within the slices still varies depending on the region's depth. If directly input into the detector, it cannot be guaranteed that all targets are at the detector's optimal recognition scale, which can easily affect detection accuracy. Therefore, this application further uses the ideal depth (optimal observation depth) where the detector performs best as a benchmark to calculate an appropriate scaling factor for each slice in the preliminary slice list. This normalizes the target scale within the slice to the detector's optimal recognition range, ensuring that regardless of whether the slice comes from the near, middle, or far view region, the targets within it can adapt to the detector's receptive field and anchor frame size after scaling, significantly improving detection accuracy. Simultaneously, a complete processing list containing slice coordinates and scaling instructions is formed, providing a clear operational basis for subsequent adaptive slicing.

[0058] in, Figure 5 This is a flowchart of sub-step S33 of the multi-target recognition and localization method for robots according to an embodiment of this application. Figure 5 As shown, step S33 includes the following steps: S331, in response to the region ID corresponding to the slice belonging to the near-field region, the near-field region scaling factor is determined as the scaling factor; S332, in response to the region ID corresponding to the slice belonging to the mid-field region or the far-field region, a depth sub-map is extracted from the dense depth map based on the slice rectangle, and the scaling factor is calculated based on the depth sub-map.

[0059] More specifically, in step S331, in response to the region ID corresponding to the slice belonging to the near-field region, the near-field region scaling factor is determined as the scaling factor. Since the target in the near-field region is close to the robot and occupies a large pixel area in the image, far exceeding the target scale corresponding to the detector's optimal observation depth, and the scale difference of the target in the near-field region is relatively small, there is no need to calculate a dynamic factor separately for each slice; a fixed preset factor is sufficient to meet the scale normalization requirements. Therefore, this application further calls the preset near-field region scaling factor as the scaling factor for the slice when the slice region ID is detected as near-field, for example, setting it to 0.5, which reduces the image size by half. This quickly shrinks the large near-field target to a scale range suitable for the detector, avoiding problems such as the detector's receptive field not being able to cover the near-field target due to its excessively large pixel area, and the loss of key structural information. It also eliminates the dynamic calculation step, improves processing efficiency, and ensures that the near-field target can fully present its global structural features after scaling, meeting the detector's requirement for complete target recognition and reducing the probability of incomplete near-field target recognition.

[0060] More specifically, in step S332, in response to the region ID corresponding to the slice belonging to the mid-range or far-range region, a depth sub-image is extracted from the dense depth map based on the slice bounding box, and a scaling factor is calculated based on the depth sub-image. Since the target in the mid-range and far-range regions is far from the robot and occupies a small pixel area in the image, and the target depth varies significantly between different mid-range and far-range slices (from 10-40 meters in the mid-range to 40-120 meters in the far-range), the target scale varies greatly. If a fixed factor is used, it cannot meet the scale normalization requirements of all slices, and some slices may still have targets that are too small or too large after scaling. Therefore, this application further extracts the depth sub-image of the corresponding region from the dense depth map based on the slice bounding box when the slice region ID is mid-range or far-range, and then calculates the dynamic scaling factor of the slice using the depth sub-image to accurately match the actual depth of the mid-range and far-range target.

[0061] In a specific example of this application, step S332 includes: calculating the scaling factor based on the depth sub-map using the following formula:

[0062]

[0063] in, This is the average of the effective depth values ​​of the depth submap. For optimal observation depth, such as 15 meters, The scaling factor ensures that mid-range and far-range targets can reach the pixel scale corresponding to the detector's optimal observation depth after scaling, significantly enhancing the feature clarity of small distant targets. It avoids problems such as insufficient scaling of distant targets and weak feature omissions caused by fixed factors, or excessive scaling of mid-range targets and feature distortion, thereby improving the detection rate and classification accuracy of mid-range and far-range targets.

[0064] In the aforementioned multi-target recognition and localization method for robots, step S4 involves adaptively slicing the original video frame based on an image slice processing list to obtain an image slice set and a slice metadata set. However, since the image slice processing list clearly defines the bounding box coordinates and specific scaling factors for each slice, but the original video frame remains a complete two-dimensional image, the scale differences between far, medium, and near targets within it are not actually adjusted. Directly inputting this data into the detector will cause feature extraction failure due to extreme target scale. Furthermore, after subsequent unified detection, the detection results within the slices need to be inverted back to the global coordinate system of the original video frame. If key information such as the original slice position and scaling ratio is lacking, the inverted coordinates will be incorrect. Therefore, this application further uses the image slice processing list to perform targeted cropping and scale adjustment on the original video frame, generating an image slice set. Simultaneously, it records the core parameters of each slice to form a slice metadata set. This transforms multi-scale targets into standardized slices that the detector can efficiently process, providing accurate position and scale references for subsequent coordinate inversion. This ensures that the target scale within each slice is within the detector's optimal processing range, significantly improving detection accuracy. At the same time, the position of the detection result in the original frame is accurately traced through metadata, avoiding the loss of target position due to slicing processing, and laying the position benchmark for subsequent multi-target tracking and 3D positioning.

[0065] In the specific implementation process, the image slice processing list is first parsed to extract the slice rectangle coordinates and scaling factor of each element, ensuring that the parameter format is compatible with the image processing module. Next, initial slices are cropped from the synchronized original video frames according to the coordinates. If the coordinates exceed the frame boundaries, they are automatically adjusted to the frame edge to ensure the cropped area is effective. Then, the slice size is adjusted using bilinear interpolation based on the scaling factor, unifying it to a fixed input size for the detector. Near-field slices are reduced in size, far-field slices are enlarged, and mid-field slices are fine-tuned. Then, metadata such as the original coordinates, scaling factor, and region ID are recorded for each slice. Finally, the adjusted slices are aggregated to form an image slice set, and an index association is established between the set and the metadata set. All slices are stored in the computing unit cache for subsequent unified detection and coordinate inversion modules to access.

[0066] In the aforementioned multi-target recognition and localization method for robots, step S5 involves performing unified detection and coordinate inversion on the image slice set based on the slice metadata set to obtain frame detection results. Since each slice in the image slice set undergoes scale normalization, target information needs to be obtained through unified detection. Furthermore, the slice detection results reside in the slice's local coordinate system; without inversion, they would be disconnected from the original video frame coordinates and cannot be directly used for localization. Therefore, this application further combines the cropping and scaling information from the slice metadata set, detecting each slice of the image slice set individually, inverting the results to the original frame coordinate system, and integrating them to obtain the complete detection results of the original video frame. This ensures that targets within multi-scale slices are accurately detected, and the detection box positions perfectly match the original frame, avoiding localization errors caused by coordinate misalignment and providing an accurate two-dimensional foundation for subsequent three-dimensional localization.

[0067] in, Figure 6 This is a flowchart of sub-step S5 of the multi-target recognition and localization method for robots according to an embodiment of this application. Figure 6 As shown, step S5 includes the following steps: S51, extracting a first image slice from the image slice set; S52, inputting the first image slice into an efficient target detector to obtain the detection result within the first slice; S53, performing coordinate inversion on the detection result within the first slice based on the slice metadata corresponding to the first image slice to obtain the first single-frame detection result.

[0068] Specifically, in step S51, a first image slice is extracted from the image slice set. Since the image slice set contains multiple slices adapted to different depth regions, each slice must be processed sequentially to ensure no omissions or confusion in detection. Direct batch processing can easily lead to misalignment between slices and metadata. Therefore, this application further extracts the first slice to be processed from the image slice set as the first image slice. This clarifies the first processing object, establishes the initial operation unit for subsequent single-slice detection and coordinate inversion, ensures the orderliness of slice processing, avoids logical confusion caused by processing multiple slices simultaneously, and facilitates accurate association of processing results with corresponding metadata, laying an orderly processing foundation for subsequent processes.

[0069] In the specific implementation process, the image slice set is first sorted according to the region priority of near-field, mid-field, and far-field to ensure that near-field slices are processed first. Next, the first slice is located based on the sorting result, and its index position in the set is determined. Then, it is verified whether the size and format of this slice meet the detector's input requirements. If not, a backtracking check of the previous slice adjustment process is triggered. After successful verification, this slice is extracted from the set as the first image slice, and its index is recorded to associate it with the corresponding slice metadata. This is then temporarily stored in the detector's input buffer, ready for subsequent detection steps.

[0070] Specifically, in step S52, the first image slice is input into the high-efficiency object detector to obtain the detection results within the first slice. Since the first image slice has undergone scale normalization through adaptive slicing to match the optimal recognition scale of the high-efficiency object detector, the detector needs to extract information such as the category and location of the targets within the slice; otherwise, specific detection data for the targets within the slice cannot be obtained. Therefore, this application further inputs the first image slice into the high-efficiency object detector, and after feature encoding, fine-tuning, and decoding, obtains the detection results of the targets within the first slice. This fully leverages the detector's accurate detection capability at the adapted scale, avoids missed detections and false detections due to scale incompatibility, and ensures the accuracy and reliability of information such as the category and bounding box of the targets within the slice.

[0071] in, Figure 7 This is a flowchart of sub-step S52 of the multi-target recognition and localization method for robots according to an embodiment of this application. Figure 7 As shown, step S52 includes the following steps: S521, inputting the first image slice into the encoder of the high-efficiency object detector to obtain the visual feature encoding vector of the first image block; S522, performing feature adaptive calibration on the visual feature encoding vector of the first image block to obtain the visual enhancement feature encoding vector of the first image block; S523, inputting the visual enhancement feature encoding vector of the first image block into the decoder of the high-efficiency object detector to obtain the detection result within the first slice.

[0072] More specifically, in step S521, the first image slice is input into the encoder of the efficient object detector to obtain the visual feature encoding vector of the first image block. Although the first image slice has been standardized to the appropriate scale, it still exists in the form of a pixel matrix, directly containing a large amount of redundant information (such as background noise), and cannot be directly used for object category judgment and bounding box prediction. Therefore, this application further inputs the first image slice into the encoder of the efficient object detector, extracting the abstract visual features of the slice through operations such as convolution and pooling, and transforming it into a structured visual feature encoding vector. This removes redundant information and retains the core features of the target (such as shape and texture), transforming the original pixel data into a high-dimensional feature representation that can be used for subsequent processing. This provides a semantically valuable information foundation for object detection, avoids low detection efficiency due to high complexity of the original data, and ensures that features of targets at different scales can be effectively captured.

[0073] In the specific implementation process, the encoder of the efficient object detector (such as the backbone network of the YOLO series) is first initialized, and pre-trained weight parameters are loaded to ensure that the encoder is adapted to the target features of the current scene. Next, the first image slice is preprocessed, including pixel value normalization and channel dimension adjustment (e.g., changing from RGB three channels to the input channel format required by the detector). Then, the preprocessed slice is input into the encoder, and shallow detail features, such as edges and textures, are extracted through multiple sets of convolutional layers. These features are then fused with deep convolutional layers through pooling layers to obtain high-level semantic features, such as the combination of target parts. Finally, the high-level semantic features are transformed into a fixed-dimensional visual feature encoding vector for the first image block.

[0074] Specifically, in step S522, the visual feature encoding vector of the first image block is adaptively calibrated to obtain the visual enhancement feature encoding vector of the first image block. Since the visual feature encoding vector extracted by the encoder may contain background features unrelated to the target, and for targets at different scales (far, medium, and near), some key features (such as the faint outline of distant targets) may be submerged in the global features, directly using them for decoding can easily lead to insufficient detection accuracy. Therefore, this application further performs adaptive feature calibration on the visual feature encoding vector of the first image block, strengthening target-related features and suppressing redundant information through attention mechanisms or residual connections to obtain the visual enhancement feature encoding vector, thereby improving the target recognition accuracy of the feature vector.

[0075] In a specific example of this application, step S522 includes: first, extracting the feature value neighborhood distribution of each position in the visual feature encoding vector of the first image block to obtain a set of local distributions of the visual feature values ​​of the first image block, expressed by the following formula:

[0076]

[0077]

[0078]

[0079] in, Encode the visual features of the first image block as a vector. Neighborhood range Eigenvalues ​​within, and For the location index within the neighborhood, A preset threshold is set for the radius of the neighborhood. The number of feature values ​​in the visual feature encoding vector of the first image block. Eigenvalues Local distribution of visual feature values ​​in the first image block It is the set of local distributions of visual feature values ​​of the first image block.

[0080] Specifically, a local window is defined for each feature value, and the set of values ​​within the window is collected to construct its statistical distribution. By drawing on the concept of receptive field, isolated scalar feature values ​​are extended into a high-dimensional data structure that can describe their local environment. This enables the mapping from a one-dimensional feature value sequence describing point information to a high-dimensional distribution space describing the surface information surrounding each feature value. This allows the local contextual attributes of each feature value (such as the structural density of the foreground and the noise interference level of the background) to be quantified. This provides complete local environment data support for subsequent feature extraction of these neighborhood distributions to generate characteristic factors and for high-resolution tuning of feature values. This ensures that the subsequent tuning process can accurately adapt to the feature characteristics of slices at different depths, laying the foundation for improving the expressive power of the final visual enhancement feature encoding vector.

[0081] Then, local distribution feature extraction is performed on the local distribution of visual feature values ​​of each first image block in the set of local distributions of visual feature values ​​of the first image block to obtain a set of local distribution characteristic factors of the first image block, which is expressed by the following formula:

[0082]

[0083]

[0084]

[0085]

[0086]

[0087]

[0088]

[0089] in, for Mean property factor of internal eigenvalues for The characteristic factor of the standard deviation of the internal eigenvalues. for Skewness property factor of internal eigenvalues for Kurtosis factor of intrinsic eigenvalues for Local significance factors of internal eigenvalues The trainable adjustment coefficient, Indicates to Perform local distribution feature extraction. The visual local distribution characteristic factor of the first image block. It is the set of visual local distribution characteristic factors of the first image block.

[0090] Specifically, by using feature engineering techniques to condense and reduce the dimensionality of information, the local distribution of visual feature values ​​of each unstructured first image block is transformed into a set of structured and quantifiable local visual distribution characteristic factors of the first image block. These characteristic factors can become digital signatures or meta-features of the local context of the visual features of the first image slice, quantitatively describing the attributes of the environment in which the original features are located, such as local noise intensity and structural density. This provides precise guiding signals for subsequent high-resolution tuning of the visual feature encoding vector of the first image block based on these characteristic factors, ensuring that the tuning process can be specifically adapted to the feature characteristics of slices at different depths, avoiding the blindness of tuning caused by the inability to directly calculate the original neighborhood distribution, and laying the foundation for improving the expressive power of the final visual enhancement feature encoding vector.

[0091] Finally, based on the set of visual local distribution characteristic factors of the first image block, the eigenvalues ​​of the visual feature encoding vector of the first image block are tuned at high resolution to obtain the visual enhancement feature encoding vector of the first image block, including:

[0092]

[0093]

[0094]

[0095]

[0096] in, for The corresponding adaptive modulation factor, for The corresponding structure-aware enhancement terms, for function, and To modulate the trainable weight matrix and adjust the trainable bias vector, for function, and For structure-aware augmented trainable weight matrix and structure-aware augmented trainable bias vector. Scaling factor The first image block visual enhancement feature encoding vector is the first... Each location feature value, These are the 1st, 2nd, and 3rd features in the visual enhancement feature encoding vector of the first image block, respectively. Each location feature value, This is the visual enhancement feature encoding vector for the first image block.

[0097] Specifically, through dynamic feature modulation and recalibration mechanisms, using characteristic factors as guiding signals, the initial feature values ​​are precisely and individually adjusted to achieve instance-adaptive ultra-high resolution control. This allows the network to dynamically reshape the feature representation of the target within the first image slice based on its internal microstructure. The resulting visual enhancement feature encoding vector for the first image block incorporates considerations of the local signal-to-noise ratio and structural saliency of the target within the slice for each value. This not only significantly improves the expressive power of the features, making distant weak target features clearer and near-field local target structural information more complete, but also significantly enhances the model's robustness to input perturbations and local changes. It guides the model to focus on the most informative and structurally stable features, laying a solid foundation for subsequently inputting them into an efficient target detector decoder to accurately obtain the detection results within the slice, thus ensuring the accuracy of multi-scale target recognition.

[0098] Specifically, in step S523, the visual enhancement feature encoding vector of the first image block is input into the decoder of the efficient object detector to obtain the detection result within the first slice. Since the visual enhancement feature encoding vector of the first image block, although focused on the core features of the target, is still a high-dimensional abstract vector, it cannot directly output interpretable detection results such as the target category and bounding box. Subsequent coordinate inversion needs to be based on specific detection information. Therefore, this application further inputs the visual enhancement feature encoding vector into the decoder of the efficient object detector. Through the decoder's regression and classification branch operations, it is transformed into the detection result within the first slice, containing the target category, bounding box coordinates, and confidence level. This maps abstract features into concrete detection information, directly obtaining key detection data of the target within the first image slice. This ensures that each target has a clear category label and position information within the slice, providing accurate basis for subsequent coordinate inversion based on metadata. Simultaneously, confidence level filtering can initially eliminate false detection results.

[0099] In the specific implementation process, the decoder of the high-efficiency object detector is first initialized, and weight parameters matching the encoder are loaded to activate the classification branch and the bounding box regression branch. Next, the visual enhancement feature encoding vector of the first image block is input into the decoder. The classification branch calculates the probability of the target belonging to each category (e.g., person, vehicle) through a fully connected layer and a softmax function. The regression branch predicts the bounding box coordinates of the target within the slice (e.g., top-left and bottom-right corner coordinates) through a regression layer. Then, combining the classification probability and regression coordinates, a confidence score is calculated for each predicted target, which is the product of the category probability and the bounding box prediction accuracy. Finally, results with confidence scores below a preset threshold are filtered out, and high-confidence results are retained as the detection results within the first slice, including the target category, the bounding box coordinates within the slice, and the confidence score, and output to the coordinate inversion module.

[0100] Specifically, in step S53, based on the slice metadata corresponding to the first image slice, coordinate inversion is performed on the detection results within the first slice to obtain the first single-frame detection result. Since the bounding box of the detection result within the first slice is in the slice's local coordinate system and is independent of the original video frame coordinates, without inversion, the true position of the target in the original frame cannot be reflected. The slice metadata records the slice's cropping coordinates and scaling factor, which are crucial for inversion. Therefore, this application further utilizes the slice metadata corresponding to the first image slice to invert and calculate the bounding box coordinates of its detection results, thereby obtaining the target's coordinates in the original video frame. This maps the local detection results within the slice to the global coordinate system of the original frame, ensuring a complete match between the target position and the original scene, providing an accurate coordinate basis for subsequent multi-slice result integration and 3D localization.

[0101] In the specific implementation process, firstly, the original frame bounding box coordinates and scaling factor are extracted from the slice metadata set using the index of the first image slice. Secondly, the bounding box coordinates of the detection results within the first slice are subjected to an inverse scaling operation, i.e., multiplied by the scaling factor, and then superimposed with the top-left corner offset of the original frame bounding box, to obtain the preliminary coordinates in the original frame coordinate system. Next, it is checked whether the preliminary coordinates are within the original frame size range; if they exceed it, they are adjusted to the frame boundary. Finally, the corrected coordinates are associated with the target category and confidence level to form the first single-frame detection result, which is then stored in conjunction with the index of the first image slice, awaiting integration with the single-frame detection results of other slices.

[0102] In the aforementioned multi-target recognition and localization method for robots, step S6 involves performing 3D localization and tracking on the frame detection results based on the robot's pose and the original point cloud to obtain a final target list. However, since the frame detection results only contain the two-dimensional coordinates and category information of the targets in the original video frames, lacking 3D position data in real physical space, they cannot support higher-level tasks requiring spatial awareness, such as robot path planning and autonomous obstacle avoidance. Furthermore, the identities of targets across frames are not associated, making continuous tracking impossible. Therefore, this application further combines the robot's pose (providing a global coordinate reference) and the original point cloud (providing 3D distance information) to perform 3D coordinate calculation and multi-target tracking on the frame detection results. This yields a final target list containing unique target IDs, global 3D positions, and motion states. This upgrades the 2D detection results to 3D information with spatial localization capabilities. Simultaneously, tracking ensures stable target identities across frames, avoiding misassociation of targets due to ID switching. This provides the robot with accurate and continuous environmental target perception data, ensuring operational safety and decision-making effectiveness.

[0103] In the specific implementation process, the robot's real-time pose (output by the SLAM system, including position and attitude parameters) and the synchronized original point cloud are first invoked to ensure that they are time-sequentially aligned with the current frame detection results. Next, for each 2D target bounding box in the frame detection results, the corresponding pixel region's point cloud cluster is extracted from the original point cloud, and the centroid of the point cloud cluster is calculated as the target's 3D coordinates in the robot coordinate system. Then, the 3D coordinates and target appearance features (extracted from the frame detection results) are input into a multi-target tracker. Cross-frame targets are matched using a data association algorithm (such as the Hungarian algorithm), assigned a unique ID, and their motion state is updated. Finally, the unique IDs, global 3D coordinates (converted from the robot pose to the world coordinate system), category, and confidence level of each target are integrated to form a structured final target list, which is stored in the robot decision system database for use by modules such as path planning and obstacle avoidance.

[0104] Furthermore, a multi-target recognition and localization system for robots is also provided.

[0105] Figure 8 This is a block diagram of a multi-target recognition and localization system for a robot according to an embodiment of this application. Figure 8 As shown, the multi-target recognition and localization system 100 for robots according to an embodiment of this application includes: a raw data acquisition module 110, used to acquire raw video frames and raw point clouds acquired by a camera and radar device; a depth map generation module 120, used to generate a dense depth map from the raw point cloud based on camera extrinsic and intrinsic parameters; a depth map dynamic segmentation module 130, used to perform dynamic region segmentation and slicing strategy generation on the dense depth map to obtain an image slice processing list; an adaptive slicing module 140, used to adaptively slice the raw video frames based on the image slice processing list to obtain an image slice set and a slice metadata set; a unified detection and coordinate inversion module 150, used to perform unified detection and coordinate inversion on the image slice set based on the slice metadata set to obtain a frame detection result; and a three-dimensional localization and tracking module 160, used to perform three-dimensional localization and tracking on the frame detection result based on the robot pose and the raw point cloud to obtain a final target list.

[0106] The basic principles of the present invention have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in the present invention are merely examples and not limitations, and should not be considered as essential features of each embodiment of the present invention. Furthermore, the specific details of the above embodiments are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the present invention to the necessity of employing the specific details described above.

[0107] In the above embodiments, the descriptions of each embodiment have their own emphasis. Parts not detailed or described in a particular embodiment can be referred to in the relevant descriptions of other embodiments. In the several embodiments provided by this invention, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative; for example, the unit division is only a logical functional division, and other division methods may exist in actual implementation. The units described as separate components may or may not be physically separated. The components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0108] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be embraced within the present invention. No appended diagram markings in the claims should be construed as limiting the scope of the claims.

[0109] Furthermore, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units stated in a system claim may also be implemented by a single unit through software or hardware.

[0110] Finally, it should be noted that the above description has been given for illustrative and descriptive purposes. Furthermore, the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although modifications or equivalent substitutions may be made to the technical solutions with reference to preferred embodiments, they will not depart from the spirit and scope of the technical solutions of the present invention.

Claims

1. A method for multi-target recognition and localization of a robot, characterized in that, include: Acquire raw video frames and raw point clouds collected by cameras and radar devices; A dense depth map is generated from the original point cloud based on camera extrinsic and intrinsic parameters. Dynamic region segmentation and slicing strategies are applied to dense depth maps to generate an image slicing list, including: Based on a set of depth thresholds, pixel-level depth partitioning and mask generation are performed on dense depth maps to obtain region masks. An adaptive slice coordinate generation is performed based on the region mask and slice parameter set to obtain a preliminary slice list. Each element in the preliminary slice list includes a slice rectangle and the region ID corresponding to the slice. Based on the optimal observation depth, a slice scaling factor is calculated for each element in the preliminary slice list to obtain the image slice processing list, including: in response to the region ID corresponding to the slice belonging to the near-field region, the near-field region scaling factor is determined as the scaling factor; in response to the region ID corresponding to the slice belonging to the mid-field region or the far-field region, a depth sub-map is extracted from the dense depth map based on the slice rectangle, and a scaling factor is calculated based on the depth sub-map. Based on the image slice processing list, the original video frames are adaptively sliced ​​to obtain an image slice set and a slice metadata set; Based on the slice metadata set, a unified detection and coordinate inversion are performed on the image slice set to obtain the frame detection result; Based on the robot's pose and the original point cloud, the frame detection results are used for 3D localization and tracking to obtain the final target list.

2. The multi-target recognition and localization method for robots according to claim 1, characterized in that, A dense depth map is generated from the original point cloud based on camera extrinsic and intrinsic parameters, including: Based on camera extrinsic parameters, coordinate system transformation is performed on each 3D point in the original point cloud to obtain an aligned point cloud; The aligned point cloud is projected based on camera intrinsic parameters to obtain a sparse depth map. The sparse depth map is densified to obtain the dense depth map.

3. The multi-target recognition and localization method for robots according to claim 2, characterized in that, The coordinate system transformation of each 3D point in the original point cloud based on camera extrinsic parameters is performed to obtain an aligned point cloud. This includes: performing a coordinate system transformation on each 3D point in the original point cloud using the following formula: ； in, For camera parameters, For each 3D point in the original point cloud, For each 3D point in the aligned point cloud.

4. The multi-target recognition and localization method for robots according to claim 2, characterized in that, Projecting the aligned point cloud onto a sparse depth map based on camera intrinsic parameters includes: projecting the aligned point cloud onto the camera intrinsic parameters using the following formula: ； in, For depth value, For camera internal parameters, For each 3D point in the aligned point cloud, It is a homogeneous coordinate vector.

5. The multi-target recognition and localization method for robots according to claim 1, characterized in that, Extracting a depth submap from a dense depth map based on sliced ​​rectangles, and calculating a scaling factor based on the depth submap, includes: calculating the scaling factor based on the depth submap using the following formula: ； in, This is the average of the effective depth values ​​of the depth submap. For optimal observation depth, This is the scaling factor.

6. The multi-target recognition and localization method for robots according to claim 1, characterized in that, Based on the slice metadata set, a unified detection and coordinate inversion process is performed on the image slice set to obtain frame detection results, including: Extract the first image slice from the image slice set; The first image slice is input into the high-efficiency object detector to obtain the detection results within the first slice; Based on the slice metadata corresponding to the first image slice, coordinate inversion is performed on the detection results within the first slice to obtain the first single-frame detection result.

7. The multi-target recognition and localization method for robots according to claim 6, characterized in that, The first image slice is input into the high-efficiency object detector to obtain the detection results within the first slice, including: The first image slice is input into the encoder of the high-efficiency object detector to obtain the visual feature encoding vector of the first image block; The visual feature encoding vector of the first image block is adaptively calibrated to obtain the visual enhancement feature encoding vector of the first image block; The visual enhancement feature encoding vector of the first image block is input into the decoder of the efficient object detector to obtain the detection result within the first slice.

8. A multi-target recognition and localization system for a robot, characterized in that, include: The raw data acquisition module is used to acquire raw video frames and raw point clouds collected by cameras and radar devices; The depth map generation module is used to generate a dense depth map from the original point cloud based on camera extrinsic and intrinsic parameters. The depth map dynamic segmentation module is used to dynamically segment dense depth maps and generate slicing strategies to obtain an image slice processing list, including: Based on a set of depth thresholds, pixel-level depth partitioning and mask generation are performed on dense depth maps to obtain region masks. An adaptive slice coordinate generation is performed based on the region mask and slice parameter set to obtain a preliminary slice list. Each element in the preliminary slice list includes a slice rectangle and the region ID corresponding to the slice. Based on the optimal observation depth, a slice scaling factor is calculated for each element in the preliminary slice list to obtain the image slice processing list, including: in response to the region ID corresponding to the slice belonging to the near-field region, the near-field region scaling factor is determined as the scaling factor; in response to the region ID corresponding to the slice belonging to the mid-field region or the far-field region, a depth sub-map is extracted from the dense depth map based on the slice rectangle, and a scaling factor is calculated based on the depth sub-map. The adaptive slicing module is used to adaptively slice the original video frames based on the image slice processing list to obtain a set of image slices and a set of slice metadata. The unified detection and coordinate inversion module is used to perform unified detection and coordinate inversion on the image slice set based on the slice metadata set to obtain the frame detection result; The 3D localization and tracking module is used to perform 3D localization and tracking on the frame detection results based on the robot's pose and the original point cloud to obtain the final target list.

Images