Multi-source sensor fusion-based man-machine interaction method for intelligent robot with body

Abstract

The invention relates to a multi-source sensor fusion-based man-machine interaction method for an intelligent robot with a body, which belongs to the field of robot intelligent control, realizes high-precision and high-efficiency fusion of point clouds and images through calibration parameters and a self-adaptive weight algorithm, integrates a YOLO model and a semantic segmentation network, and improves the fusion accuracy of the intelligent robot with the body. Fine identification and motion trend pre-judgment of multiple types of targets are realized in combination with a trajectory prediction model; meanwhile, a visual explanation function is creatively added, previews of an original image, a point cloud fusion result and a target recognition result are provided, a radar map is supported to visually display point cloud distribution characteristics, and a user can visually evaluate a processing effect; in addition, the system also has a point cloud fusion algorithm test function, can calculate multiple indexes such as fusion time and errors, and provides data support for algorithm optimization. Through fusion of point cloud radar information and image data, the target identification accuracy and decision-making efficiency are further improved, and the reliability and robustness of the system in a complex scene are enhanced.

Description

Technical Field

[0001] This invention belongs to the field of robot intelligent control, and specifically proposes a human-computer interaction method for embodied intelligent robots based on multi-source sensor fusion. Background Technology

[0002] In cutting-edge fields such as autonomous driving, intelligent transportation systems, and navigation for embodied intelligent robots, environmental perception, as a core technology, directly determines the safety and reliability of the system. The KITTI dataset, as an industry-standard testing benchmark, has become an important tool for algorithm research and model evaluation due to its rich image, point cloud, calibration, and label data.

[0003] Traditional target detection methods often rely on data from a single sensor, which has significant limitations. While image-based detection can acquire rich texture and semantic information, it lacks depth data, making it difficult to accurately locate the target's 3D coordinates and pose. Point cloud-based methods can directly present 3D spatial information, but the data is sparse and uneven, and semantic information is missing, resulting in low efficiency in target recognition and classification.

[0004] To overcome the limitations of single sensors, multi-sensor fusion technology has become a research hotspot. Among these, point cloud and image fusion has wide applications, achieving semantic annotation and 3D localization by projecting 3D point clouds onto a 2D image plane. However, this technology still faces many challenges: low efficiency in point cloud data processing and visualization, insufficient accuracy in point cloud image registration, limited performance of detection models in complex scenes, and the urgent need to address the problems of data imbalance and multi-scale target detection. Summary of the Invention

[0005] Against this backdrop, this invention, based on the YOLOv5 model, integrates radar point cloud and image information, achieving not only high-precision point cloud image fusion but also innovatively incorporating visualization functions and trajectory prediction, optimizing the human-computer interaction experience, and providing a new direction for the development of environmental perception technology in fields such as robot movement.

[0006] This invention aims to solve the technical challenges of motion prediction in the field of embodied intelligence: addressing the accuracy bottleneck and computational efficiency issues of point cloud and image fusion in complex environments, the real-time challenges of target recognition and semantic segmentation, and the accuracy and real-time issues of robot trajectory prediction in dynamic scenes, it proposes an integrated solution. By constructing a precise coordinate transformation matrix between point cloud and image through calibration parameters, the three-dimensional point cloud is projected onto a two-dimensional image plane. Simultaneously, an adaptive weight fusion algorithm is designed to optimize the efficiency of multimodal data integration. The invention integrates the YOLO target detection model and semantic segmentation network to achieve refined classification of road targets (pedestrians, obstacles, etc.), and combines this with a trajectory prediction model to predict motion trends.

[0007] This invention enhances a robot's environmental perception capabilities through multi-source sensor data fusion, and specifically adds a visualization and explanation module for the fusion process to the embodied intelligence system. By dynamically displaying the semantically related regions of point clouds and images using attention heatmaps, it intuitively presents the feature matching logic of road targets. Simultaneously, through the visualization rendering of trajectory prediction results, it assists human operators in understanding the robot's basis for predicting the movement trajectories of traffic participants, providing interpretable support for intelligent driving decisions.

[0008] A human-computer interaction method for embodied intelligent robots based on multi-source sensor fusion includes the following steps:

[0009] S1: Load the object detection model YOLOv5, initialize the dataset path, file list, main component and layout; create an instance of the MainWindow class that inherits from QMainWindow, set the window title and initial size, initialize the vertical layout to organize interface elements, create a horizontal layout to place processing options; create option buttons and link their click events to the corresponding processing methods;

[0010] S2: Acquire camera sensor data and LiDAR sensor data; acquire camera image files and LiDAR binary point cloud data corresponding to the current index, combined with calibration file data used to describe the relationship between the camera, LiDAR, and sensors; perform preprocessing, converting the image from BGR format to RGB format and adjusting the image size; normalize the image pixel data and add batch dimensions to adapt to the model's input requirements;

[0011] S3: Call the point cloud fusion model to perform point cloud fusion, realize radar point cloud coordinate transformation, project LiDAR point cloud data onto camera image, generate sparse depth map and record fusion time and fusion error at the same time.

[0012] S4: Convert the fused image to BGR format, read the point cloud data and calibration information, call the detector.detect method to perform target recognition and semantic segmentation, and record the detection time;

[0013] S5: Clean up visualization resources, obtain image color information and calibration information from effective point cloud locations, and create a visualization window;

[0014] S6: Use Kalman filtering to predict car trajectory; when a car is detected for the first time, initialize the state and covariance matrix of the Kalman filter, where the state includes the car's position and speed, and the covariance matrix represents the uncertainty of the state; update the state of the Kalman filter based on the current observations and correct the predicted state.

[0015] The beneficial effects achieved by this invention are as follows:

[0016] (1) The technical solution of this invention achieves high-precision and high-efficiency fusion of point cloud and image by calibration parameters and adaptive weight algorithm, integrates YOLO model and semantic segmentation network, and combines trajectory prediction model to achieve fine recognition of multi-category targets and prediction of motion trends; (2) At the same time, it innovatively adds visualization interpretation function, provides preview of original image, point cloud fusion result and target recognition result, supports radar chart visualization to display point cloud distribution characteristics, and facilitates users to intuitively evaluate the processing effect; (3) In addition, it also has point cloud fusion algorithm testing function, which can calculate multiple indicators such as fusion time and error, and provide data support for algorithm optimization; (4) By fusing point cloud radar information and image data, the target recognition accuracy and decision efficiency are further improved, and the reliability and robustness of the system in complex scenarios are enhanced. Attached Figure Description

[0017] Figure 1 This is a flowchart illustrating the operation steps of the human-computer interaction method in a specific embodiment of the present invention.

[0018] Figure 2 This is a flowchart illustrating the operational steps of target recognition and semantic segmentation in a specific embodiment of the present invention. Detailed Implementation

[0019] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings.

[0020] A human-computer interaction method for embodied intelligent robots based on multi-source sensor fusion includes the following steps:

[0021] S1: Load the object detection model YOLOv5, and initialize the dataset path, file list, main components, and layout.

[0022] S2: Select the dataset, validate the directory structure, and update the original image preview.

[0023] S3: Call the fusion.to_fusion function to perform point cloud fusion, realize radar point cloud coordinate transformation, and record fusion time and fusion error.

[0024] S4: Convert the fused image to BGR format, read the point cloud data and calibration information, call the detector.detect method to perform target recognition and semantic segmentation, and record the detection time.

[0025] S5: Clean up visualization resources, obtain image color information and calibration information from valid point cloud locations, and create a visualization window.

[0026] In step S3, the improved point cloud fusion model is described in the following details:

[0027] S31: Initialize a sparse depth image with all zeros, sparse_depth_image, and fill the depth image with the corresponding depth values ​​based on the projected point cloud coordinates.

[0028] S32: Performs color rendering on the sparse depth map, normalizes the original depth values ​​for Jet color mapping, and enhances visual continuity through interpolation.

[0029] S33: Perform gradient calculation on the depth map and draw highlighted outlines at abrupt changes in depth. Statistical quantile processing is used to avoid interference from outliers.

[0030] The operational steps of object recognition and semantic segmentation will be described in detail below:

[0031] S41: Use the locally saved YOLOv5 model for target recognition, and adopt the pre-trained DeepLabv3-ResNet101 model for road segmentation tasks.

[0032] S42: Obtain the mapping relationship from category ID to name, associate the point cloud data with the point cloud fusion result corresponding to the camera calibration parameters, realize 3D coordinate localization, and enhance environmental perception capabilities. Convert the model output into a Pandas data frame, dynamically assign colors, extract bounding box coordinates, confidence scores, and labels, and draw the target recognition result map.

[0033] S43: In semantic segmentation, the original input image is adjusted to 256×256 pixels, normalized, and converted into a PyTorch tensor. A road mask is created based on the model output to remove small regions with an area less than 100 pixels, reducing noise. The mask is then adjusted to the original image size.

[0034] S44: Project ground points onto the image plane to generate a ground point cloud mask. Combine the road mask obtained from semantic segmentation with the ground point cloud projection mask to enhance the accuracy of road detection. Combine the target recognition result map and the semantic segmentation map to output the comprehensive detection result.

[0035] S51: Extract the class activation map (CAM) of the YOLOv5 model using Grad-CAM.

[0036] S52: Overlay the CAM with the depth map obtained by fusing the point cloud to generate a semantic association heatmap.

[0037] S53: Display the original image, fusion result, and heat map in different areas of the visualization interface.

[0038] This method employs a pixel-level fusion scheme between LiDAR point clouds and camera images. It transforms the point cloud from the LiDAR coordinate system to the camera coordinate system using an extrinsic parameter matrix, achieving a true conversion between the two systems. Then, it projects the points from the camera coordinate system onto the image plane using an intrinsic parameter matrix, forming a fused image with depth information. For sparse depth map generation, depth values ​​are filled in by projecting point cloud coordinates, preserving the original data accuracy and exhibiting low computational complexity, making it suitable for real-time scenarios. Pseudo-color rendering is used, with depth values ​​normalized using Jet color mapping. Bilinear interpolation enhances visual continuity, and gradient calculations are used to draw highlighted outlines at abrupt depth changes, improving the scene's three-dimensionality.

[0039] End-to-end multimodal data fusion process: This process protects the entire workflow from coordinate transformation between LiDAR point clouds and images, depth map generation to target detection, especially the cascaded projection algorithm from the LiDAR system to the image system. It emphasizes a dynamic switching strategy between sparse and dense depth maps, selecting the optimal fusion mode based on real-time requirements.

[0040] Innovative fusion of semantic and geometric information: This involves protecting the method for associating YOLOv5 target detection results with point cloud depth information, achieving target spatial localization through the mapping of bounding box coordinates to 3D point clouds. It also includes a visualization scheme that protects Grad-CAM overlay with depth maps to generate semantic association heatmaps, enhancing environmental understanding capabilities.

[0041] Real-time optimization and engineering implementation: Protects the multi-threaded batch processing mechanism, achieving ≤6ms processing time for a single frame of 100,000 point clouds, and supports efficient parsing of the KITTI dataset. Protects the modular design architecture, including decoupled interfaces for data input, core processing, and visualization modules, facilitating expansion with other sensors.

[0042] The above description is only a preferred embodiment of the present invention. The scope of protection of the present invention is not limited to the above embodiments. Any equivalent modifications or changes made by those skilled in the art based on the content disclosed in the present invention should be included within the scope of protection set forth in the claims.

Claims

1. A human-computer interaction method for embodied intelligent robots based on multi-source sensor fusion, characterized in that: The method includes the following steps: S1: Load the object detection model YOLOv5, initialize the dataset path, file list, main components and layout; create a main window class instance, set the window title and initial size, initialize the vertical layout to organize interface elements, create a horizontal layout to place processing options; create option buttons and link their click events to the corresponding processing methods; S2: Acquire camera sensor data and LiDAR sensor data; acquire camera image files and LiDAR binary point cloud data corresponding to the current index, combined with calibration file data used to describe the relationship between the camera, LiDAR, and sensors; Preprocessing is performed to convert the image from BGR format to RGB format and adjust the image size; the image pixel data is normalized and batch dimension is added to adapt to the model's input requirements; S3: Call the point cloud fusion model to perform point cloud fusion, realize radar point cloud coordinate transformation, project LiDAR point cloud data onto camera image, generate sparse depth map and record fusion time and fusion error at the same time. S4: Convert the fused image to BGR format, read the point cloud data and calibration information, call the detector.detect method to perform target recognition and semantic segmentation, and record the detection time; S5: Clean up visualization resources, obtain image color information and calibration information from effective point cloud locations, and create a visualization window; S6: Use Kalman filtering for vehicle trajectory prediction; When a car is detected for the first time, the state and covariance matrix of the Kalman filter are initialized, where the state includes the car's position and speed, and the covariance matrix represents the uncertainty of the state; the state of the Kalman filter is updated based on the current observations, and the predicted state is corrected.

2. The human-computer interaction method for embodied intelligent robots based on multi-source sensor fusion according to claim 1, characterized in that: In step S3, the specific steps of the improved point cloud fusion model are as follows: S31: Initialize a sparse depth map sparse_depth_image with all zeros based on the height h and width w of the image. Call the function to obtain the point cloud mask projected onto the image and the point cloud array data with the shape (M, 3) after projection, where M represents the number of points falling within the image range. Obtain the x and y coordinates of the projected point cloud and convert them to integer types. Then use these coordinates as indices to fill the corresponding depth values ​​into the depth map. S32: Perform input checks on the sparse depth map to ensure that the input is a single-channel depth map, normalize the original depth values ​​for Jet color mapping, convert the normalized depth map into RGB colors, and then convert it into integers in the range of [0,255] to enhance visual continuity through interpolation; S33: Calculate the gradient of the depth map using the Sobel operator; calculate the horizontal gradient Gx and the vertical gradient Gy of the depth map respectively, using the formula... Obtain the gradient magnitude; draw a highlighted outline at the depth abrupt change; calculate the quantile of the gradient magnitude through statistical quantile processing, select the 95th quantile as the threshold, and regard points with gradient magnitude greater than the threshold as depth abrupt change points, thereby avoiding outlier interference.

3. The human-computer interaction method for embodied intelligent robots based on multi-source sensor fusion according to claim 1, characterized in that: The specific steps of step S4 are as follows: S41: Use the locally stored YOLOv5 object detection model for object recognition, and adopt the pre-trained DeepLabv3-ResNet101 model. This model is pre-trained on a large-scale image dataset and simulates learning the feature representations of different objects in the image, including color, texture, and shape. The road segmentation task is achieved through this model, separating the road region in the image from other regions. S42: Obtain the mapping relationship from category ID to name, associate the point cloud data with the point cloud fusion result corresponding to the camera calibration parameters, and realize 3D coordinate positioning; obtain the model output through image recognition inference, extract the coordinates, confidence and category index of the bounding box, convert it into a Pandas data frame, dynamically assign colors, extract the bounding box coordinates, confidence and label, and draw the target recognition result map; S43: In semantic segmentation, the original input image is adjusted to 256×256 pixels, converted to a PyTorch tensor after data conversion and normalization, and obtained according to model inference. The predicted category index is then defined, and a road category index is defined to create a road mask. Small regions with an area of ​​less than 100 pixels are removed to reduce noise. The mask is then adjusted to the original image size. S44: Project the ground points onto the image plane to generate a ground point cloud mask; combine the road mask obtained from semantic segmentation with the ground point cloud projection mask to generate a semantic segmentation map; combine the target recognition result map and the semantic segmentation map to output the detection result.

4. The human-computer interaction method for embodied intelligent robots based on multi-source sensor fusion according to claim 1, characterized in that: The specific steps of step S5 are as follows: S51: Extract the class activation map (CAM) of the YOLOv5 model using the gradient-weighted class activation map Grad-CAM. The formula is as follows: CAM(x,y)=∑ k w k ·f k (x,y) Among them, w k For class weights, f k (x,y) represents the activation value of the feature map, k represents the kth feature map, and x and y represent the horizontal and vertical positions of the spatial coordinates, respectively. S52: Overlay the CAM with the depth map after fusing the point cloud to generate a semantic association heatmap; S53: Display the original image, the fusion result of the original image and radar point cloud, and the heat map in different areas in the visualization interface.

Images