A robot target detection method, device and medium based on fusion perception

Abstract

This specification discloses a robot target detection method, device, and medium based on fusion perception, relating to the field of robotics. The method includes: using a multi-sensor fusion device of an inspection robot to collect in real time multiple frames of point cloud data and multiple frames of image data corresponding to the environment to be detected; performing region filtering on each frame of point cloud data to generate multiple corresponding point cloud region sets, and performing region filtering on each frame of image data to generate multiple image region blocks corresponding to each frame of image data; generating target fusion features of the multi-fusion data to be detected based on the multiple point cloud region sets and the multiple image region blocks corresponding to each frame of image data to be detected using a pre-trained target detection model, the target detection model including a three-layer neural network structure; and performing target detection on the environment to be detected based on the target fusion features of the multi-fusion data to be detected, so as to identify target obstacles in the environment to be detected.

Description

Technical Field

[0001] This specification relates to the field of robotics technology, and in particular to a robot target detection method, device and medium based on fusion perception. Background Technology

[0002] In the current construction of various smart scenarios, intelligent transportation systems have become an active research area, effectively improving system efficiency and decision-making. Intelligent driving technology needs to be integrated into the research field of outdoor security robots to enhance the safety performance of robot inspection and other activities. Specifically, target detection, such as trajectory prediction and obstacle detection, can increase the robot's reaction time in complex scenarios, thereby improving its driving safety.

[0003] Currently, LiDAR and cameras are two crucial sensors for obstacle detection in autonomous driving. Multimodal fusion solutions using LiDAR and cameras are becoming increasingly popular in the field. However, both point clouds and images require consideration of temporal effects and sensor registration. Existing fusion methods primarily rely on hard correlation between LiDAR points and image pixels, using a calibration matrix to connect image pixels to the point cloud. However, the calibration matrix always deviates significantly from the actual values, leading to substantial and irreversible errors during fusion. In summary, existing technologies are susceptible to the influence of temporal timing and device calibration during multimodal fusion, resulting in inaccurate target detection results. Summary of the Invention

[0004] This specification provides one or more embodiments of a robot target detection method, device, and medium based on fusion perception, to solve the following technical problem: existing technologies are easily affected by timing and device calibration when performing multimodal fusion, resulting in inaccurate target detection results.

[0005] One or more embodiments of this specification employ the following technical solutions:

[0006] This specification provides one or more embodiments of a robot target detection method based on fusion perception. The method includes: acquiring multi-fusion data to be detected in real time corresponding to the environment to be detected using a multi-sensor fusion device of an inspection robot, wherein the multi-fusion data to be detected includes multiple frames of point cloud data and multiple frames of image data to be detected; performing region filtering on each frame of the point cloud data to be detected to generate multiple point cloud region sets corresponding to each frame of the point cloud data to be detected, and performing region filtering on each frame of the image data to be detected to generate multiple image region blocks corresponding to each frame of the image data to be detected; generating target fusion features of the multi-fusion data to be detected based on the multiple point cloud region sets corresponding to each frame of the point cloud data to be detected and the multiple image region blocks corresponding to each frame of the image data to be detected using a pre-trained target detection model, wherein the target detection model includes a three-layer neural network structure; and performing target detection on the environment to be detected according to the target fusion features of the multi-fusion data to be detected to identify target obstacles in the environment to be detected.

[0007] Further, the system performs region filtering on each frame of the point cloud data to be detected, generating multiple point cloud region point sets corresponding to each frame of the point cloud data to be detected. Specifically, this includes: acquiring single point cloud information for each point cloud data in each frame of the point cloud data to be detected, wherein the single point cloud information includes point cloud location data and point cloud attribute information for each point cloud data; determining at least one point cloud key point that meets the requirements from multiple point cloud data based on the point cloud attribute information in the single point cloud information of each point cloud data; determining multiple point cloud region points that meet the requirements corresponding to each point cloud key point based on the point cloud location information of each point cloud data, wherein the distance between each point cloud region point and the point cloud key point is not greater than a preset distance threshold; and using each point cloud key point and the multiple point cloud region points that meet the requirements corresponding to each point cloud key point as point cloud region point sets to generate multiple point cloud region point sets corresponding to each frame of the point cloud data to be detected.

[0008] The process of filtering regions for each frame of the image data to be detected to generate multiple image region blocks corresponding to each frame of the image data to be detected includes: splitting each frame of the image data to be detected according to a preset ratio to obtain multiple sub-regions of the image to be detected; obtaining pixel information of each pixel in each sub-region of the image to be detected; determining the average pixel value of each sub-region of the image to be detected based on the pixel information of each sub-region of the image to be detected; filtering the multiple sub-regions of the image to be detected according to the average pixel value of each sub-region of the image to be detected, and selecting multiple specified sub-regions of the image to be detected that meet the requirements as multiple image region blocks corresponding to each frame of the image data to be detected.

[0009] Furthermore, before generating the target fusion features of the multi-fusion data to be detected based on multiple point cloud region sets corresponding to each frame of the point cloud data to be detected and multiple image region blocks corresponding to each frame of the image data to be detected using a pre-trained target detection model, the method further includes: constructing a first-layer detection model, a second-layer detection model, and a third-layer detection model based on a preset network structure; wherein, the first-layer detection model is used to model multiple frames of point cloud data to be detected or multiple frames of image data to be detected, and determine the intra-feature correlation of the same frame of data; the second-layer detection model is used to determine the extra-feature correlation between adjacent frames based on the intra-feature correlation of the same frame of data, and obtain latent features under different modalities; the third-layer detection model is used to fuse the latent features under different modalities based on the inter-modal self-attention mechanism, and obtain the target fusion features; based on the first-layer detection model, the second-layer detection model, and the third-layer detection model, an initial target detection model is determined, and the target detection model is obtained based on the initial target detection model.

[0010] Further, based on a pre-trained target detection model, and using multiple point cloud region sets corresponding to each frame of the point cloud data to be detected and multiple image region blocks corresponding to each frame of the image data to be detected, target fusion features of the multi-fusion data to be detected are generated. Specifically, this includes: inputting each point cloud region set into the first-layer detection model and outputting a first specified correlation vector corresponding to each point cloud region set, wherein the first specified correlation vector is used to represent the intra-feature correlation in each frame of the point cloud data to be detected; inputting each image region block into the first-layer detection model and outputting a first preset correlation feature vector corresponding to each image region block, wherein the first preset correlation feature vector is used to represent the intra-feature correlation in each frame of the image data to be detected; and inputting the point cloud data to be detected in adjacent frames into the first-layer detection model. The first specified correlation feature vectors are input into the second layer detection model to output the second specified correlation feature vector of the point cloud data to be detected in adjacent frames. The second specified correlation feature vector is used to represent the extrinsic correlation between the point cloud data to be detected in adjacent frames. The first preset correlation feature vectors corresponding to the image data to be detected in adjacent frames are input into the second layer detection model to output the second preset correlation feature vector of the image data to be detected in adjacent frames. The second preset correlation feature vector is used to represent the extrinsic correlation between the image data to be detected in adjacent frames. The second preset correlation feature vector and the second specified correlation feature vector are input into the third layer detection model to output the target fusion feature of the multi-fusion data to be detected.

[0011] Further, based on the initial target detection model, the target detection model is obtained, specifically including: acquiring historical point cloud data and historical image data of the robot in the application scenario using a LiDAR and image acquisition device installed on the robot; acquiring a common point cloud dataset, downsampling the common point cloud dataset to obtain sampled point cloud data that meets the requirements, wherein the data format of the sampled point cloud data that meets the requirements is the same as the data format of the historical point cloud data acquired by the LiDAR; constructing a point cloud dataset based on the sampled point cloud data and the historical point cloud data; constructing an image dataset based on the historical image data; and using the point cloud dataset and the image dataset as training datasets to train the initial target detection model to obtain a target detection model that meets the requirements.

[0012] Furthermore, based on the point cloud attribute information in the single point cloud information of each point cloud data, at least one point cloud key point that meets the requirements is determined from multiple point cloud data. Specifically, this includes: dividing the point cloud data in each frame of the point cloud data to be detected into multiple sets of point cloud data; obtaining the point cloud attribute information in the single point cloud information of each point cloud data, wherein the point cloud attribute information includes laser reflection intensity; and based on the laser reflection intensity of each point cloud data, determining the specified point cloud data with the highest intensity in each set of point cloud data as the point cloud key point.

[0013] Further, the point cloud dataset and the image dataset are used as training datasets, specifically including: performing data augmentation processing on each point cloud data in the point cloud dataset to generate a specified point cloud data after augmentation for each point cloud data; performing image denoising processing and image augmentation processing on each image data in the image dataset to generate a specified image data after processing for each image data; and generating a training dataset based on multiple specified point cloud data and multiple specified image data.

[0014] This specification provides one or more embodiments of a robot target detection device based on fusion perception, comprising:

[0015] At least one processor; and,

[0016] A memory communicatively connected to the at least one processor; wherein,

[0017] The memory stores instructions that can be executed by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to:

[0018] The inspection robot uses a multi-sensor fusion device to collect real-time multi-fusion data corresponding to the environment to be inspected. This multi-fusion data includes multiple frames of point cloud data and multiple frames of image data. For each frame of point cloud data, region filtering is performed to generate multiple point cloud region sets corresponding to each frame. Similarly, for each frame of image data, region filtering is performed to generate multiple image region blocks corresponding to each frame. Using a pre-trained target detection model, target fusion features are generated based on the multiple point cloud region sets and image region blocks corresponding to each frame of point cloud data and image data. The target detection model includes a three-layer neural network structure. Based on these target fusion features, target detection is performed on the environment to identify target obstacles.

[0019] This specification provides one or more embodiments of a non-volatile computer storage medium storing computer-executable instructions, wherein the computer-executable instructions are configured as follows:

[0020] The inspection robot uses a multi-sensor fusion device to collect real-time multi-fusion data corresponding to the environment to be inspected. This multi-fusion data includes multiple frames of point cloud data and multiple frames of image data. For each frame of point cloud data, region filtering is performed to generate multiple point cloud region sets corresponding to each frame. Similarly, for each frame of image data, region filtering is performed to generate multiple image region blocks corresponding to each frame. Using a pre-trained target detection model, target fusion features are generated based on the multiple point cloud region sets and image region blocks corresponding to each frame of point cloud data and image data. The target detection model includes a three-layer neural network structure. Based on these target fusion features, target detection is performed on the environment to identify target obstacles.

[0021] The above-mentioned at least one technical solution adopted in the embodiments of this specification can achieve the following beneficial effects: Through the above technical solution, the collected multi-fusion data are filtered by region, and the filtered point cloud region set and image region block are input into the multi-layer neural network model. The model extracts the features of the internal correlation and the external correlation of adjacent frames of the point cloud region set and image region block. The results of the internal correlation features and external correlation features are fused to achieve target recognition. There is no need to calibrate the multi-sensor device, avoiding the influence of time sequence and device calibration, and further improving the accuracy of target detection results. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments or prior art of this specification, the drawings used in the description of the embodiments or prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. In the drawings:

[0023] Figure 1 A flowchart illustrating a robot target detection method based on fusion perception provided in the embodiments of this specification;

[0024] Figure 2 This is a schematic diagram of the structure of a robot target detection device based on fusion perception, provided as an embodiment of this specification. Detailed Implementation

[0025] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this specification, and not all embodiments. Based on the embodiments of this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this specification.

[0026] In the current construction of various smart scenarios, intelligent transportation systems have become an active research area, effectively improving system efficiency and decision-making. Intelligent driving technology needs to be integrated into the research field of outdoor security robots to enhance the safety performance of robot inspection and other activities. Specifically, target detection, such as trajectory prediction and obstacle detection, can increase the robot's reaction time in complex scenarios, thereby improving its driving safety.

[0027] Currently, LiDAR and cameras are two crucial sensors for obstacle detection in autonomous driving. Multimodal fusion solutions using LiDAR and cameras are becoming increasingly popular in the field. However, both point clouds and images require consideration of temporal effects and sensor registration. Existing fusion methods primarily rely on hard correlation between LiDAR points and image pixels, using a calibration matrix to connect image pixels to the point cloud. However, the calibration matrix always deviates significantly from the actual values, leading to substantial and irreversible errors during fusion. In summary, existing technologies are susceptible to the influence of temporal timing and device calibration during multimodal fusion, resulting in inaccurate target detection results.

[0028] This specification provides a robot target detection method based on fusion perception. It should be noted that the execution entity in this specification embodiment can be a server or any device with data processing capabilities. Figure 1 A flowchart illustrating a robot target detection method based on fusion perception provided in this specification is shown below. Figure 1 As shown, the main steps include the following:

[0029] Step S101: The multi-sensor fusion device of the inspection robot collects multi-sensor fusion data corresponding to the environment to be inspected in real time.

[0030] The multi-fusion data to be detected includes multiple frames of point cloud data to be detected and multiple frames of image data to be detected.

[0031] In one embodiment of this specification, a multi-sensor fusion device is installed on the head of the inspection robot. This device includes a Velodyne-16-line LiDAR and a USB camera. The Velodyne-16-line LiDAR collects point cloud data corresponding to the environment to be inspected. Since the inspection robot's data acquisition process is real-time, the collected point cloud data consists of multiple frames, i.e., multiple frames of point cloud data to be inspected. Furthermore, video images are acquired using the USB camera mounted on the inspection robot, and the acquired video data is converted into multiple frames of image data to be inspected.

[0032] Step S102: Perform region filtering on each frame of point cloud data to be detected to generate multiple point cloud region point sets corresponding to each frame of point cloud data to be detected; perform region filtering on each frame of image data to be detected to generate multiple image region blocks corresponding to each frame of image data to be detected.

[0033] For each frame of the point cloud data to be detected, region filtering is performed to generate multiple point cloud region point sets corresponding to each frame of the point cloud data to be detected. Specifically, this includes: obtaining single point cloud information for each point cloud data in each frame of the point cloud data to be detected, wherein the single point cloud information includes point cloud location data and point cloud attribute information for each point cloud data; determining at least one point cloud key point that meets the requirements from multiple point cloud data based on the point cloud attribute information in the single point cloud information of each point cloud data; determining multiple point cloud region points that meet the requirements corresponding to each point cloud key point based on the point cloud location information of each point cloud data, wherein the distance between each point cloud region point and the point cloud key point is not greater than a preset distance threshold; and using each point cloud key point and the multiple point cloud region points that meet the requirements corresponding to each point cloud key point as point cloud region point sets to generate multiple point cloud region point sets corresponding to each frame of the point cloud data to be detected.

[0034] In one embodiment of this specification, due to the large number of points in the point cloud data to be detected, inputting all points would result in a large computational load, leading to slow measurement speed. Furthermore, the point cloud data contains a significant amount of redundant data, which could easily cause inaccurate identification due to inputting redundant data. Therefore, it is necessary to pre-divide each frame of the point cloud data to be detected into regional point sets. First, the single point cloud information of each point cloud data in each frame of the point cloud data to be detected is obtained. The single point cloud information includes the point cloud location data and point cloud attribute information of each point cloud data. Based on the point cloud attribute information in the single point cloud information of each point cloud data, at least one point cloud key point that meets the requirements is determined from multiple point cloud data. Using each point cloud key point as the center, all point cloud data within a preset distance threshold from the point cloud key point are designated as regional point cloud points. It should be noted that the distance between each point cloud regional point and the point cloud key point is no greater than the preset distance threshold. This distance threshold can be preset according to the user's needs for the robot. A larger distance threshold results in a larger regional point set and a smaller number of regional point sets corresponding to each frame of the point cloud data to be detected. Each point cloud keypoint and its corresponding multiple compliant point cloud region points are used as a point cloud region point set to generate multiple point cloud region point sets corresponding to each frame of the point cloud data to be detected.

[0035] Based on the point cloud attribute information in the single point cloud information of each point cloud data, at least one point cloud key point that meets the requirements is determined from multiple point cloud data. Specifically, this includes: dividing the point cloud data in each frame of the point cloud data to be detected into multiple sets of point cloud data; obtaining the point cloud attribute information in the single point cloud information of each point cloud data, wherein the point cloud attribute information includes laser reflection intensity; and based on the laser reflection intensity of each point cloud data, determining the specified point cloud data with the highest intensity in each set of point cloud data as the point cloud key point.

[0036] In one embodiment of this specification, when determining key points in a point cloud, the point cloud data in each frame of the point cloud data to be detected is divided equally. This division can be based on the size of the point cloud data or the number of points. For example, assuming each frame of the point cloud data to be detected has a size of 4*4, the point cloud data in each frame can be divided into 16 groups of 1*1 points. Assuming each frame contains 100 points, it can be divided into 10 groups, each containing 10 points. After obtaining multiple groups of point cloud data, the point cloud attribute information, including laser reflection intensity, is obtained from the single point cloud information of each point cloud data. Within each group of point cloud data, the point cloud data with the highest intensity is selected as the key point.

[0037] For each frame of the image data to be detected, region filtering is performed to generate multiple image region blocks corresponding to each frame of the image data to be detected. Specifically, this includes: splitting each frame of the image data to be detected according to a preset ratio to obtain multiple sub-regions of the image to be detected; obtaining the pixel information of each pixel in each sub-region of the image to be detected; determining the average pixel value of each sub-region of the image to be detected based on the pixel information of each pixel in each sub-region of the image to be detected; filtering the multiple sub-regions of the image to be detected according to the average pixel value of each sub-region of the image to be detected, and selecting multiple specified sub-regions of the image to be detected that meet the requirements as multiple image region blocks corresponding to each frame of the image data to be detected.

[0038] In one embodiment of this specification, to avoid redundant data in the image data, it is necessary to perform region filtering on each frame of the image data to be detected. Multiple image region blocks are generated. First, the image data to be detected is divided according to a preset ratio to obtain multiple sub-regions. This division can be random or equal division according to certain rules. For example, assuming the image data to be detected is 4*4 data, it can be divided into four 2*2 sub-regions, or into sixteen 1*1 sub-regions, or it can be randomly divided. The pixel value of each pixel in each sub-region is obtained, and the average pixel value of each sub-region is calculated. Based on the average pixel value of each sub-region, a preset pixel threshold is compared, and the sub-regions with values ​​greater than the preset pixel threshold are selected as image region blocks. It should be noted that the preset pixel threshold can be set according to the average pixel value corresponding to redundant environments in the environment.

[0039] Step S103: Using a pre-trained target detection model, target fusion features of the multi-fusion data to be detected are generated based on multiple point cloud region point sets corresponding to each frame of point cloud data to be detected and multiple image region blocks corresponding to each frame of image data to be detected.

[0040] Before generating the target fusion features of the multi-fusion data to be detected based on multiple point cloud region sets corresponding to each frame of the point cloud data to be detected and multiple image region blocks corresponding to each frame of the image data to be detected using a pre-trained target detection model, the method further includes: constructing a first-layer detection model, a second-layer detection model, and a third-layer detection model based on a preset network structure; wherein, the first-layer detection model is used to model multiple frames of point cloud data or multiple frames of image data to be detected and determine the intra-feature correlation of the same frame of data; the second-layer detection model is used to determine the extra-feature correlation between adjacent frames based on the intra-feature correlation of the same frame of data and obtain latent features under different modalities; the third-layer detection model is used to fuse the latent features under different modalities based on the self-attention mechanism between modalities to obtain the target fusion features; based on the first-layer detection model, the second-layer detection model, and the third-layer detection model, an initial target detection model is determined, and the target detection model is obtained based on the initial target detection model.

[0041] In one embodiment of this specification, considering the need for multi-sensor temporal alignment, image feature extraction abandons convolutional networks, which have strong local feature extraction capabilities. Both modal features are extracted using a Transformer structure as the encoder. A three-layer detection model is constructed using the Transformer network structure, with all three layers adhering to the Transformer framework. That is, the object detection model comprises a three-layer neural network structure, divided into three levels. The multimodal inputs are encoded by independent Transformer streams in the first layer, a Transformer layer is stacked in the middle to encode the temporal correlation between frames, and the outputs are connected and fused by the last Transformer. A multi-head self-attention (MSA) block is introduced to capture temporal dependencies. A cross-attention module is added after the self-attention mechanism of the last Transformer network structure to realize the interaction between point cloud and image features. Network inference is performed, mapping the two inputs to the final 3D bounding box output.

[0042] In other words, the first-layer detection model is used to model multiple frames of point cloud data or multiple frames of image data to be detected, determining the intra-feature correlation of the same frame. For point cloud data generated by LiDAR, the first-layer Transformer model models the correlation between point clouds in each frame, that is, the pairwise correlation of point clouds in local regions, thereby learning the local features of the context on a single point cloud data. For image data, the first-layer Transformer model models the intra-image information, that is, the correlation between the local regions divided in a single frame image, and its input is the image region blocks divided into multiple blocks.

[0043] The second-layer detection model is used to determine the extra-feature correlation between adjacent frames based on the intra-feature correlation of data in the same frame, thus obtaining latent features under different modalities. The second-layer Transformer model models the inter-frame correlation of point clouds, that is, it learns the contextual content features of the scene point clouds between two frames, and the Transformer model can perform long-distance temporal learning of point clouds. The second-layer Transformer model models the correlation between video frames, that is, it learns the image correlation features between two frames.

[0044] The reason for this design is to take into account the contextual information in the scene point cloud. Many objects still exist in the scene even after the changes between frames. For example, in a scene where the target object is not moving violently, the feature changes of the target object are minimal in the images or point clouds between frames. This allows us to use the correlation between frames to make a judgment. On the other hand, it also prevents the modeling difficulties caused by the unfolded sequence being too long.

[0045] The third-layer detection model uses a self-attention mechanism between modalities to fuse latent features from different modalities, obtaining target fusion features. The correlation and complementarity of various modalities can effectively identify some special objects. It should be noted that special objects here refer to target objects that do not move violently or undergo instantaneous displacement in the scene. Therefore, the designed third-layer Transformer model utilizes the self-attention mechanism between modalities to model the correlation between features from different modalities. That is, the latent features output from the point cloud are fused with the latent features output from the image. Higher weights are assigned to features from modalities that detect anomalies, while lower weights are assigned to features from other modalities that detect anomalies before fusion. In addition, a cross-attention module is used to refine and fuse target features using information from the image and point cloud. An initial target detection model is determined through the first-layer detection model, the second-layer detection model, and the third-layer detection model. Based on this initial target detection model, the target detection model is obtained.

[0046] Based on the initial target detection model, the target detection model is obtained, specifically including: acquiring historical point cloud data and historical image data of the robot in the application scenario using a LiDAR and image acquisition device installed on the robot; acquiring a public point cloud dataset, downsampling the public point cloud dataset to obtain sampled point cloud data that meets the requirements, wherein the data format of the sampled point cloud data that meets the requirements is the same as the data format of the historical point cloud data acquired by the LiDAR; constructing a point cloud dataset based on the sampled point cloud data and the historical point cloud data; constructing an image dataset based on the historical image data; and using the point cloud dataset and the image dataset as a training dataset to train the initial target detection model to obtain a target detection model that meets the requirements.

[0047] Using the point cloud dataset and the image dataset as training datasets, specifically including: performing data augmentation processing on each point cloud data in the point cloud dataset to generate a specified point cloud data after augmentation for each point cloud data; performing image denoising processing and image augmentation processing on each image data in the image dataset to generate a specified image data after processing for each image data; and generating a training dataset based on multiple specified point cloud datasets and multiple specified image datasets.

[0048] In one embodiment of this specification, since the LiDAR installed on the robot is a Velodyne-16-line LiDAR, its historical data consists of park scenes, which suffers from a limited number of points and a single scene. Therefore, the training dataset is expanded using the KITTI-64-line point cloud public dataset. First, the KITTI-64-line point cloud public dataset is acquired and downsampled to obtain 16-line point cloud data. The 16-line point cloud data acquired by the 16-line LiDAR and the 16-line point cloud data from the public dataset are used as the point cloud dataset. Historical image data acquired by the image acquisition device mounted on the robot is used as the image dataset. The point cloud dataset and the image dataset are used as the training dataset to train the initial target detection model, resulting in a target detection model that meets the requirements.

[0049] In one embodiment of this specification, in order to make the data in the dataset more robust, increase the permutation invariance and prevent overfitting, a point cloud data enhancement module is used to perform positive enhancement processing on the point cloud data. Basic point cloud data enhancement includes random rotation, random scaling, random discarding, random translation and random perturbation. Image data also needs to be preprocessed, including noise removal and contrast enhancement.

[0050] Using a pre-trained target detection model, based on multiple point cloud region sets corresponding to each frame of the point cloud data to be detected and multiple image region blocks corresponding to each frame of the image data to be detected, target fusion features of the multi-fusion data to be detected are generated. Specifically, this includes: inputting each point cloud region set into the first-layer detection model and outputting a first specified correlation vector corresponding to each point cloud region set, wherein the first specified correlation vector is used to represent the intra-feature correlation in each frame of the point cloud data to be detected; inputting each image region block into the first-layer detection model and outputting a first preset correlation feature vector corresponding to each image region block, wherein the first preset correlation feature vector is used to represent the intra-feature correlation in each frame of the image data to be detected; and merging the target fusion features of adjacent frames of the point cloud data to be detected. Multiple first-specified correlation feature vectors are input into the second-layer detection model, which outputs a second-specified correlation feature vector for the point cloud data to be detected in adjacent frames. The second-specified correlation feature vector is used to represent the extrinsic correlation between the point cloud data to be detected in adjacent frames. Multiple first-specified correlation feature vectors corresponding to the image data to be detected in adjacent frames are input into the second-layer detection model, which outputs a second-specified correlation feature vector for the image data to be detected in adjacent frames. The second-specified correlation feature vector and the second-specified correlation feature vector are input into the third-layer detection model, which outputs the target fusion feature of the multi-fusion data to be detected.

[0051] In one embodiment of this specification, for point cloud data, each point cloud region set is input into the first-layer detection model, and a first specified correlation vector corresponding to each point cloud region set is output. The first specified correlation vector is used to represent the intra-feature correlation in each frame of the point cloud data to be detected. Here, intra-feature correlation refers to the pairwise correlation of point clouds in each region set. For image data, each image region block is input into the first-layer detection model, and a first preset correlation feature vector corresponding to each image region block is output. Here, the first preset correlation feature vector is used to represent the intra-feature correlation in each frame of the image data to be detected. Here, intra-feature correlation refers to the correlation within each image region block.

[0052] Since each frame of point cloud data to be detected includes multiple region point sets, each frame of point cloud data to be detected corresponds to multiple first-specified correlation feature vectors. The multiple first-specified correlation feature vectors corresponding to two adjacent frame point cloud data are input into the second-layer detection model, which outputs a second-specified correlation feature vector for the two adjacent frame point cloud data. The second-specified correlation feature vector represents the extrinsic correlation between the two adjacent frame point cloud data. Similarly, each frame of image data to be detected includes multiple image region blocks, each image region block corresponding to a first-preset correlation feature vector. Therefore, each frame of image data to be detected corresponds to multiple first-preset correlation feature vectors. The multiple first-preset correlation feature vectors corresponding to two adjacent frame image data are input into the second-layer detection model, which outputs a second-preset correlation feature vector for the two adjacent frame image data. The second-preset correlation feature vector represents the extrinsic correlation between the two adjacent frame image data. The second-preset correlation feature vector and the second-specified correlation feature vector are input into the third-layer detection model, which outputs the target fusion feature of the multi-fusion data to be detected.

[0053] Step S104: Based on the target fusion features of the multi-fusion data to be detected, target detection is performed on the environment to be detected in order to identify target obstacles in the environment to be detected.

[0054] In one embodiment of this specification, target detection is performed on the environment to be detected based on the target fusion features of the multi-fusion data to be detected, so as to identify target obstacles in the environment to be detected.

[0055] The above technical solution involves filtering the collected multi-fusion data into different regions, inputting the filtered point cloud regions and image regions into a multi-layer neural network model, and extracting features from the internal correlation and external correlation of the point cloud regions and image regions. The results of internal correlation features and external correlation features are then fused to achieve target recognition. This eliminates the need for calibration of multi-sensor devices, avoids the influence of time series and device calibration, and further improves the accuracy of target detection results.

[0056] This specification also provides an embodiment of a robot target detection device based on fusion perception, such as... Figure 2As shown, the device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enable the at least one processor to: acquire, in real time, multi-sensor fusion data corresponding to the environment to be detected via the multi-sensor fusion device of the inspection robot, wherein the multi-fusion data to be detected includes multiple frames of point cloud data and multiple frames of image data to be detected; perform region filtering on each frame of the point cloud data to be detected to generate multiple point cloud region sets corresponding to each frame of the point cloud data to be detected, and perform region filtering on each frame of the image data to be detected to generate multiple image region blocks corresponding to each frame of the image data to be detected; generate target fusion features of the multi-fusion data to be detected based on the multiple point cloud region sets corresponding to each frame of the point cloud data to be detected and the multiple image region blocks corresponding to each frame of the image data to be detected using a pre-trained target detection model, wherein the target detection model includes a three-layer neural network structure; and perform target detection on the environment to be detected based on the target fusion features of the multi-fusion data to be detected, so as to identify target obstacles in the environment to be detected.

[0057] This specification also provides a non-volatile computer storage medium storing computer-executable instructions. These instructions are configured to: acquire, in real-time, multi-sensor fusion data corresponding to the environment to be detected using a multi-sensor fusion device of an inspection robot; wherein the multi-fusion data includes multiple frames of point cloud data and multiple frames of image data to be detected; perform region filtering on each frame of the point cloud data to generate multiple point cloud region sets corresponding to each frame; perform region filtering on each frame of the image data to be detected to generate multiple image region blocks corresponding to each frame; generate target fusion features of the multi-fusion data to be detected based on the multiple point cloud region sets and the multiple image region blocks corresponding to each frame of the image data to be detected using a pre-trained target detection model; wherein the target detection model includes a three-layer neural network structure; and perform target detection on the environment to be detected based on the target fusion features of the multi-fusion data to be detected, thereby identifying target obstacles in the environment to be detected.

[0058] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the embodiments of apparatus, devices, and non-volatile computer storage media are basically similar to the method embodiments, so the descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0059] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0060] The devices, media, and methods provided in the embodiments of this specification are one-to-one correspondences. Therefore, the devices and media also have similar beneficial technical effects as their corresponding methods. Since the beneficial technical effects of the methods have been described in detail above, the beneficial technical effects of the devices and media will not be repeated here.

[0061] Those skilled in the art will understand that embodiments of this specification can be provided as methods, systems, or computer program products. Therefore, this specification may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this specification may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0062] This specification is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this specification. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create a machine for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0063] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0064] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0065] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0066] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0067] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0068] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0069] The above description is merely one or more embodiments of this specification and is not intended to limit this specification. Various modifications and variations can be made to the one or more embodiments of this specification by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principle of one or more embodiments of this specification should be included within the scope of the claims of this specification.

Claims

1. A robot target detection method based on fusion perception, characterized in that, The method includes: The inspection robot uses a multi-sensor fusion device to collect real-time multi-fusion data of the environment to be inspected, which includes multiple frames of point cloud data and multiple frames of image data to be inspected. For each frame of the point cloud data to be detected, perform region filtering to generate multiple point cloud region point sets corresponding to each frame of the point cloud data to be detected. For each frame of the image data to be detected, perform region filtering to generate multiple image region blocks corresponding to each frame of the image data to be detected. By using a pre-trained target detection model, target fusion features of the multi-fusion data to be detected are generated based on multiple point cloud region point sets corresponding to each frame of the point cloud data to be detected and multiple image region blocks corresponding to each frame of the image data to be detected. The target detection model includes a three-layer neural network structure. Based on the target fusion features of the multi-fusion data to be detected, target detection is performed on the environment to be detected to identify target obstacles in the environment to be detected; For each frame of the point cloud data to be detected, region filtering is performed to generate multiple point cloud region point sets corresponding to each frame of the point cloud data to be detected, specifically including: Obtain single point cloud information for each point cloud data in each frame of the point cloud data to be detected, wherein the single point cloud information includes point cloud location data and point cloud attribute information for each point cloud data; Based on the point cloud attribute information in the single point cloud information of each point cloud data, at least one point cloud key point that meets the requirements is determined from multiple point cloud data. Based on the point cloud location data of each point cloud data, multiple point cloud region points that meet the requirements are determined for each point cloud key point, wherein the distance between each point cloud region point and the point cloud key point is not greater than a preset distance threshold. Each point cloud key point and the multiple point cloud region points corresponding to each point cloud key point that meet the requirements are used as a point cloud region point set to generate multiple point cloud region point sets corresponding to the point cloud data to be detected in each frame. Each frame of the image data to be detected is subjected to region filtering to generate multiple image region blocks corresponding to each frame of the image data to be detected, specifically including: Each frame of the image data to be detected is divided according to a preset ratio to obtain multiple sub-regions of the image to be detected; Obtain pixel information for each pixel in each sub-region of the image to be detected; Based on the pixel information of each pixel in each sub-region of the image to be detected, the average pixel value of each sub-region of the image to be detected is determined; Based on the average pixel value of each of the image sub-regions to be detected, multiple image sub-regions to be detected are filtered, and multiple specified image sub-regions that meet the requirements are used as multiple image region blocks corresponding to each frame of image data to be detected. Before generating the target fusion features of the multi-fusion data to be detected based on multiple point cloud region point sets corresponding to each frame of the point cloud data to be detected and multiple image region blocks corresponding to each frame of the image data to be detected using a pre-trained target detection model, the method further includes: Based on the preset network structure, a first-layer detection model, a second-layer detection model, and a third-layer detection model are constructed. The first layer detection model is used to model multiple frames of point cloud data or multiple frames of image data to be detected, and to determine the intra-feature correlation of the same frame of data. The second-layer detection model is used to determine the extra-feature correlation between adjacent frames based on the intra-feature correlation of data in the same frame, and to obtain the latent features under different modalities; The third-layer detection model is used to fuse latent features from different modalities based on the intermodal self-attention mechanism to obtain target fused features; Based on the first layer detection model, the second layer detection model, and the third layer detection model, an initial target detection model is determined, and the target detection model is obtained based on the initial target detection model.

2. The robot target detection method based on fusion perception according to claim 1, characterized in that, Using a pre-trained target detection model, based on multiple point cloud region sets corresponding to each frame of the point cloud data to be detected and multiple image region blocks corresponding to each frame of the image data to be detected, target fusion features of the multi-fusion data to be detected are generated, specifically including: Each point cloud region point set is input into the first layer detection model, and a first specified correlation vector corresponding to each point cloud region point set is output, wherein the first specified correlation vector is used to represent the intra-feature correlation in each frame of the point cloud data to be detected; Each image region block is input into the first layer detection model, and a first preset correlation feature vector corresponding to each image region block is output, wherein the first preset correlation feature vector is used to represent the intra-feature correlation in each frame of the image data to be detected; Multiple first-specified correlation vectors corresponding to the point cloud data to be detected in adjacent frames are input into the second-layer detection model, and the second-specified correlation feature vector of the point cloud data to be detected in adjacent frames is output, wherein the second-specified correlation feature vector is used to represent the out-of-feature correlation between the point cloud data to be detected in adjacent frames. Multiple first preset correlation feature vectors corresponding to the image data to be detected in adjacent frames are input into the second layer detection model, and the second preset correlation feature vector of the image data to be detected in adjacent frames is output. The second preset correlation feature vector is used to represent the out-of-feature correlation between the image data to be detected in adjacent frames. The second preset correlation feature vector and the second specified correlation feature vector are input into the third-layer detection model, and the target fusion feature of the multi-fusion data to be detected is output.

3. The robot target detection method based on fusion perception according to claim 1, characterized in that, Based on the initial target detection model, the target detection model is obtained, specifically including: By using a lidar and image acquisition device mounted on the robot, historical point cloud data and historical image data of the robot in the application scenario are obtained. A public point cloud dataset is acquired, and the public point cloud dataset is downsampled to obtain sampled point cloud data that meets the requirements from the public point cloud dataset. The data format of the sampled point cloud data that meets the requirements is the same as the data format of the historical point cloud data acquired by the lidar. Based on the sampled point cloud data and the historical point cloud data, a point cloud dataset is constructed; Based on the historical image data, an image dataset is constructed; The point cloud dataset and the image dataset are used as training datasets to train the initial object detection model, thereby obtaining an object detection model that meets the requirements.

4. The robot target detection method based on fusion perception according to claim 1, characterized in that, Based on the point cloud attribute information in the single point cloud information of each point cloud data, at least one key point that meets the requirements is determined from multiple point cloud data, specifically including: The point cloud data in each frame of the point cloud data to be detected is divided equally to obtain multiple sets of point cloud data. Obtain point cloud attribute information from the single point cloud information of each point cloud data, wherein the point cloud attribute information includes laser reflection intensity; Based on the laser reflection intensity of each point cloud data, the specified point cloud data with the highest intensity is determined in each group of point cloud data and used as the key point of the point cloud.

5. The robot target detection method based on fusion perception according to claim 3, characterized in that, Using the point cloud dataset and the image dataset as training datasets, specifically including: Perform data augmentation processing on each point cloud data in the point cloud dataset to generate a specified point cloud data after augmentation for each point cloud data. Perform image denoising and image enhancement processing on each image data in the image dataset to generate a specified image data after processing each image data; A training dataset is generated based on multiple specified point cloud data and multiple specified image data.

6. A robot target detection device based on fusion perception, characterized in that, The device includes: At least one processor; and, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method as described in any one of claims 1-5.

7. A non-volatile computer storage medium storing computer-executable instructions, characterized in that, The computer-executable instructions are configured to perform the method as described in any one of claims 1-5.

Images