A target detection method and device for a robot

Abstract

The application provides a target detection method and device for a robot. The method comprises the following steps: acquiring environment point cloud data and scene description data collected by a robot environment collection device; performing data enhancement on the environment point cloud data and the scene description data to obtain enhanced data; performing feature extraction on the enhanced data to obtain feature data; performing alignment processing on the feature data to obtain aligned data; and performing target detection according to the aligned data to obtain a target detection result. The application acquires environment point cloud data and scene description data collected by a robot environment collection device, then performs data enhancement on the environment point cloud data and the scene description data to obtain enhanced data, then performs feature extraction on the enhanced data to obtain feature data, then performs alignment processing on the feature data to obtain aligned data, and finally performs target detection according to the aligned data to obtain a target detection result, which is beneficial to improving the accuracy of the target detection result.

Description

Technical Field

[0001] This invention relates to the field of target detection technology, and also to a target detection method and apparatus for robots. Background Technology

[0002] With the development of robotics technology, intelligent mobile robots have made significant progress and are now being practically applied in people's daily lives and work, such as tunnel inspection robots and computer room inspection robots. The smooth movement of intelligent mobile robots relies on the judgment and recognition of targets in the environment, as well as the identification of obstacles and fall detection information, to avoid obstacles and prevent falls, adapting to different work scenarios and environments. However, in existing technologies, when robots detect targets in the environment, modal differences and sensor noise lead to large alignment errors, affecting the accuracy of target detection. Summary of the Invention

[0003] The technical problem to be solved by the present invention is to provide a target detection method and apparatus for robots, so as to improve the accuracy of target detection in robots.

[0004] To solve the above-mentioned technical problems, the technical solution of the present invention is as follows:

[0005] A first aspect of the present invention provides a target detection method for a robot, comprising:

[0006] Acquire environmental point cloud data and scene description data collected by the robot's environmental acquisition equipment;

[0007] Data augmentation is performed on the environmental point cloud data and the scene description data to obtain augmented data;

[0008] Feature extraction is performed on the enhanced data to obtain feature data;

[0009] The feature data is aligned to obtain aligned data;

[0010] Target detection is performed based on the alignment data to obtain the target detection result.

[0011] Optionally, acquire environmental point cloud data and scene description data collected by the robot's environmental acquisition device, including:

[0012] Environmental point cloud data is collected by an image acquisition device fixed at the first preset position of the robot;

[0013] Voice data is collected by a voice acquisition device fixed at the second preset position of the robot;

[0014] Scene description data is obtained based on the voice data.

[0015] Optionally, data augmentation is performed on the environmental point cloud data and the scene description data to obtain augmented data, including:

[0016] Based on the environmental point cloud data and the preset search radius and minimum number of neighboring points, outlier points are determined;

[0017] The environmental point cloud data is filtered based on the outliers to obtain filtered point cloud data;

[0018] The selected point cloud data and the scene description data are augmented according to a preset data augmentation method to obtain augmented data.

[0019] Optionally, feature extraction is performed on the enhanced data to obtain feature data, including:

[0020] The environmental point cloud data in the enhanced data is standardized to obtain standardized data;

[0021] The scene description data in the enhanced data is transformed into a sub-word sequence to obtain sub-word sequence data;

[0022] Feature data is obtained by extracting features from the standardized data and the sub-word sequence data.

[0023] Optionally, the feature data is aligned to obtain aligned data, including:

[0024] The feature data is subjected to dimensionality reduction processing to obtain dimensionality-reduced data;

[0025] The similarity is calculated by performing similarity calculation on the dimensionality-reduced data to obtain the similarity score.

[0026] The dimensionality-reduced data is aligned based on the similarity to obtain aligned data.

[0027] Optionally, target detection is performed based on the alignment data to obtain target detection results, including:

[0028] Based on the alignment data, candidate bounding boxes are obtained;

[0029] The candidate bounding boxes are adjusted to obtain the adjusted candidate bounding boxes;

[0030] Target detection is performed based on the adjusted candidate bounding boxes to obtain the target detection results.

[0031] Optionally, the method further includes:

[0032] Based on the target detection results, the robot's movement path is determined.

[0033] A second aspect of the present invention provides a target detection device for a robot, comprising:

[0034] The acquisition module is used to acquire environmental point cloud data and scene description data collected by the robot's environmental acquisition device;

[0035] The processing module is used to perform data augmentation on the environmental point cloud data and the scene description data to obtain augmented data; to extract features from the augmented data to obtain feature data; to perform alignment processing on the feature data to obtain aligned data; and to perform target detection based on the aligned data to obtain target detection results.

[0036] A third aspect of the present invention provides a computing device, comprising: a processor and a memory storing a computer program, wherein the computer program, when executed by the processor, performs the method described in the first aspect.

[0037] A fourth aspect of the present invention provides a computer-readable storage medium storing instructions that, when executed on a computer, cause the computer to perform the method as described in the first aspect.

[0038] The above-described solution of the present invention has at least the following beneficial effects:

[0039] The above-mentioned solution of the present invention acquires environmental point cloud data and scene description data collected by a robot environment acquisition device, then performs data augmentation on the environmental point cloud data and scene description data to obtain augmented data, then extracts features from the augmented data to obtain feature data, performs alignment processing on the feature data to obtain aligned data, and finally performs target detection based on the aligned data to obtain target detection results, which helps to improve the accuracy of target detection results. Attached Figure Description

[0040] Figure 1 This is a flowchart illustrating the target detection method for robots in an embodiment of the present invention;

[0041] Figure 2 This is a schematic diagram of the target detection device for robots in an embodiment of the present invention. Detailed Implementation

[0042] Exemplary embodiments of the invention will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the invention are shown in the drawings, it should be understood that the invention may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this invention will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art.

[0043] like Figure 1 As shown, an embodiment of the present invention proposes a target detection method for robots, comprising the following steps:

[0044] Step 101: Obtain environmental point cloud data and scene description data collected by the robot's environmental acquisition device;

[0045] Step 102: Perform data augmentation on the environmental point cloud data and the scene description data to obtain augmented data;

[0046] Step 103: Extract features from the enhanced data to obtain feature data;

[0047] Step 104: Align the feature data to obtain aligned data;

[0048] Step 105: Perform target detection based on the alignment data to obtain the target detection result.

[0049] The target detection method for robots according to embodiments of the present invention acquires environmental point cloud data and scene description data collected by the robot's environmental acquisition device, then performs data augmentation on the environmental point cloud data and scene description data to obtain augmented data, then extracts features from the augmented data to obtain feature data, performs alignment processing on the feature data to obtain aligned data, and finally performs target detection based on the aligned data to obtain target detection results, which helps to improve the accuracy of target detection results.

[0050] In an optional embodiment of the present invention, step 101 includes:

[0051] Step 1011: Collect environmental point cloud data using an image acquisition device fixed at the first preset position of the robot;

[0052] Specifically, the image acquisition device can be a 3D LiDAR, suitable for large-scale outdoor scenes, providing high-precision, long-range point cloud data; or an RGB-D camera (3D camera), suitable for close-range indoor scenes, simultaneously providing RGB (red, green, blue, a color standard) images and depth maps; or a structured light camera, suitable for static scenes, calculating depth by projecting structured light patterns. In specific embodiments, the type of image acquisition device can be selected according to the specific type of robot. The first preset position can be at least one position on the top, middle, and bottom of the robot. Different positions can be selected to fix the image acquisition device according to the user and type of the robot, so as to collect high-quality environmental point cloud data. In one specific embodiment, the first preset position is the top of the robot or the end of the robotic arm, ensuring that the field of view covers the target area.

[0053] Step 1012: Collect voice data using a voice acquisition device fixed at the second preset position of the robot;

[0054] Specifically, the robot is equipped with a microphone array as a voice acquisition device to collect voice data. The second preset position can be any position on the robot. Different positions can be selected to fix the voice acquisition device according to the user and type of the robot, so as to collect high-quality voice data.

[0055] Step 1013: Obtain scene description data based on the voice data.

[0056] Specifically, after converting the speech data into text, semantic labels are added to each 3D scene in the collected environmental point cloud data based on the text, either through manual annotation or automatic generation (such as using a multimodal language model with visual capabilities to generate scene descriptions) or through annotation tools (inputting object categories, locations, and attributes, etc.). For example, "There is a laptop and a coffee cup on the desk."

[0057] In an optional embodiment of the present invention, step 102 includes:

[0058] Step 1021: Determine outlier points based on the environmental point cloud data, the preset search radius, and the minimum number of neighboring points;

[0059] Step 1022: Filter the environmental point cloud data based on the outliers to obtain filtered point cloud data;

[0060] Step 1023: Perform data augmentation on the selected point cloud data and the scene description data according to the preset data augmentation method to obtain augmented data.

[0061] Specifically, the preset search radius and minimum number of neighboring points in step 1021 can be preset as follows: the preset search radius r is 0.05m, and the minimum number of neighboring points... The value is 5. For each point in the environmental point cloud data. All of its neighborhood that satisfy the condition point As target points, the number of target points is: ;like Then point Outliers are identified; in step 1022, all outliers in the environmental point cloud data are deleted, and the remaining environmental point cloud data is the filtered point cloud data.

[0062] In step 1023, when performing data augmentation on the selected point cloud data, the corresponding preset data augmentation method can be at least one of random rotation (e.g., ±30° around the Z-axis), translation (±0.1m), or adding Gaussian noise; when performing data augmentation on the scene description data, the corresponding preset data augmentation method can be at least one of synonym replacement (e.g., replacing "cup" with "water cup") or attribute perturbation (e.g., changing "red" to "blue"); the augmented point cloud data and the augmented scene description data are used together as augmented data for subsequent processing. By simulating sensor errors and expanding text diversity through augmented data, it is beneficial to improve the accuracy of target detection in the future.

[0063] In an optional embodiment of the present invention, step 103 includes:

[0064] Step 1031: Standardize the environmental point cloud data in the enhanced data to obtain standardized data;

[0065] Specifically, through The environmental point cloud data in the augmented data is downsampled to obtain downsampled environmental point cloud data; then through... The downsampled environmental point cloud data is standardized to obtain standardized data. Among them, Let N be the average coordinates of all points in the downsampled environmental point cloud data, and N be the number of points in the augmented environmental point cloud data. To enhance the k-th environmental point cloud data in the dataset, , The i-th preset parameter is a cube or rectangular region in three-dimensional space, determined by the voxel size (resolution). For the i-th standardized data, To reduce the mean of the sampled environmental point cloud data, To reduce the standard deviation of the sampling environment point cloud data.

[0066] Step 1032: Perform sub-word sequence transformation on the scene description data in the enhanced data to obtain sub-word sequence data;

[0067] Specifically, the scene description data in the augmented data is split into multiple words or sub-words, each corresponding to a unique sequence identifier in a pre-defined vocabulary. Then, all words or sub-words are arranged in order according to this unique sequence identifier, with sub-words at the beginning of sentences marked as sentence beginnings and sub-words at the end of sentences marked as sentence endings, resulting in sub-word sequence data. ,in, L represents the number of words or subwords, and T represents the set of words or subwords, i.e., the subword sequence data.

[0068] Step 1033: Extract features from the standardized data and the sub-word sequence data to obtain feature data.

[0069] Specifically, the steps for feature extraction from standardized data include: selecting using farthest point sampling. Each center point For each center point Search radius Neighborhood points Calculate relative coordinates ;pass , Mapping relative coordinates to a high-dimensional space yields the mapped high-dimensional features; through , For all points in the neighborhood Perform element-wise max pooling to obtain aggregated local features; finally, through... Generate a set of point-level features ,in, .

[0070] Where N is the total number of points in the standardized data; The number of center points selected by sampling from the farthest point. ; For each of the center points sampled from the farthest point, 3 represents the XYZ coordinates of the point, and d represents the additional feature dimension of the point; For the search radius, Center point The set of neighborhood points containing all points that satisfy... point ; Relative coordinates, i.e., neighborhood points Relative to the center point Coordinate offset; These are the high-dimensional features after mapping; For mapping functions; For hyperparameters; These are local features after aggregation; It is a set of point-level features; Point-level features; It is a composite function. These are learnable parameters.

[0071] The steps for feature extraction from sub-word sequence data include: inputting the sub-word sequence data into a pre-defined encoding model (such as the BERT model) and obtaining the output of each layer. ,in This represents the hidden state of layer l. For the dimension; take the word or sub-word features from the last or second-to-last layer as the feature vector: .

[0072] in, The hidden state of layer l; T represents the sub-word sequence data; M represents the actual number of words or sub-words in the sub-word sequence data; For dimensions; These are the eigenvectors.

[0073] Here, feature data includes point-level features and feature vectors.

[0074] In an optional embodiment of the present invention, step 104 includes:

[0075] Step 1041: Perform dimensionality reduction processing on the feature data to obtain dimensionality-reduced data;

[0076] Specifically, through Dimensionality reduction is performed on the point-level features in the feature data to obtain the dimensionality-reduced point-level features; among which, These are the point-level features after dimensionality reduction. This is a neural network structure that performs feature transformation and dimensionality reduction on input data through multi-layer nonlinear transformations. As an aggregation feature, , Point-level features The number of center points selected by sampling from the farthest point.

[0077] pass The dimensionality reduction process is performed on the feature vectors in the feature data to obtain the dimensionality-reduced feature vectors; where, The feature vectors after dimensionality reduction. This is a neural network structure that performs feature transformation and dimensionality reduction on input data through multi-layer nonlinear transformations. These are the eigenvectors.

[0078] Here, the dimensionality-reduced data includes the point-level features and the feature vectors after dimensionality reduction.

[0079] Step 1042: Calculate the similarity of the dimensionality-reduced data to obtain the similarity score;

[0080] Specifically, through Calculate the similarity between the dimensionality-reduced point-level features and the dimensionality-reduced feature vectors; where, Let be the similarity between the dimensionality-reduced point-level features of the i-th point and the dimensionality-reduced feature vector of the i-th point. It is an exponential function. The point-level features of the i-th point after dimensionality reduction. The feature vectors after dimensionality reduction. The bandwidth parameter, used to control the similarity decay rate, can be 0.5. Based on the similarity between the point-level features and the feature vectors after dimensionality reduction, a similarity set can be obtained. .

[0081] Step 1043: Align the dimensionality-reduced data according to the similarity to obtain aligned data.

[0082] Specifically, from the similarity set The key point is to select the preset number (e.g., 100). ;pass Alignment processing is performed to obtain aligned data. Among them, For the m-th key point, The point-level features of the i-th point after dimensionality reduction. Let be the similarity between the dimensionality-reduced point-level features of the i-th point and the dimensionality-reduced feature vector of the i-th point. This is the a-th aligned data.

[0083] In an optional embodiment of the present invention, step 105 includes:

[0084] Step 1051: Obtain candidate bounding boxes based on the alignment data;

[0085] Step 1052: Adjust the candidate bounding box to obtain the adjusted candidate bounding box;

[0086] Step 1053: Perform target detection based on the adjusted candidate bounding boxes to obtain target detection results.

[0087] Specifically, through The process involves obtaining a discrimination value; identifying points in the environmental point cloud data with a discrimination value greater than the discriminant value as foreground points and points with a discrimination value less than the discriminant value as background points; generating candidate bounding boxes for each foreground point; pooling the point cloud features within each candidate bounding box to generate a fixed-length feature vector; predicting the offset of the candidate bounding box using the fixed-length feature vector and an MLP (Multi-Level Processing) algorithm; adjusting the candidate bounding boxes based on the offset to minimize the error between the candidate bounding boxes and the ground truth boxes; and finally, predicting the object category probability using a Softmax (Normalized Exponential Function) classifier, which yields the object detection result.

[0088] here, To distinguish values, , To preset hyperparameters, This is the a-th aligned data.

[0089] In an optional embodiment of the present invention, the method further includes:

[0090] Step 106: Determine the robot's movement path based on the target detection results.

[0091] Specifically, the detected target object is located at (x, y, z). A fast expanding random tree algorithm is used to plan a collision-free path from the robot's current position to the target. Candidate grasping points are generated through a grasping pose estimation network, and the grasping point with the highest score is selected for robotic arm control. It should be noted that existing algorithms can be used to calculate the grasping path, allowing the robot to move to the target object and grasp it.

[0092] A specific embodiment of the target detection method for robots according to this invention includes:

[0093] Step 111: Obtain environmental point cloud data and scene description data collected by the robot's environmental acquisition device;

[0094] Environmental point cloud data is collected by a 3D LiDAR or 3D camera fixed on the top of the robot or the end of the robotic arm; voice data is collected by a microphone array mounted on the robot, and after the voice data is converted into text, semantic tags are added to each 3D scene in the collected environmental point cloud data according to the text through manual annotation, automatic generation or annotation tools, to obtain scene description data.

[0095] Step 112: Perform data augmentation on the environmental point cloud data and the scene description data to obtain augmented data;

[0096] Enhanced data is obtained by performing data augmentation on environmental point cloud data and scene description data using corresponding data augmentation methods.

[0097] Step 113: Extract features from the enhanced data to obtain feature data;

[0098] The environmental point cloud data is standardized, and the scene description data is transformed into a word sequence. Then, feature extraction is performed on the standardized data and the word sequence data respectively to obtain feature data.

[0099] Step 114: Align the feature data to obtain aligned data;

[0100] The environmental point cloud data and scene description data are aligned by calculating similarity.

[0101] Step 115: Perform target detection based on the alignment data to obtain the target detection result.

[0102] Candidate bounding boxes are obtained by distinguishing between foreground and background points in the environmental point cloud data. After adjustment, object detection is performed to obtain the object detection result. The object detection result can be any object.

[0103] The target detection method for robots in this invention improves the quality of collected data, reduces data errors, and increases the accuracy of final target detection by performing various processing and alignment on the collected environmental point cloud data and scene description data before finally performing target detection.

[0104] like Figure 2 As shown, an embodiment of the present invention provides a target detection device 200 for a robot, comprising:

[0105] The acquisition module 201 is used to acquire environmental point cloud data and scene description data collected by the robot's environmental acquisition device;

[0106] The processing module 202 is used to perform data augmentation on the environmental point cloud data and the scene description data to obtain augmented data; to extract features from the augmented data to obtain feature data; to perform alignment processing on the feature data to obtain aligned data; and to perform target detection based on the aligned data to obtain target detection results.

[0107] Optionally, acquire environmental point cloud data and scene description data collected by the robot's environmental acquisition device, including:

[0108] Environmental point cloud data is collected by an image acquisition device fixed at the first preset position of the robot;

[0109] Voice data is collected by a voice acquisition device fixed at the second preset position of the robot;

[0110] Scene description data is obtained based on the voice data.

[0111] Optionally, data augmentation is performed on the environmental point cloud data and the scene description data to obtain augmented data, including:

[0112] Based on the environmental point cloud data and the preset search radius and minimum number of neighboring points, outlier points are determined;

[0113] The environmental point cloud data is filtered based on the outliers to obtain filtered point cloud data;

[0114] The selected point cloud data and the scene description data are augmented according to a preset data augmentation method to obtain augmented data.

[0115] Optionally, feature extraction is performed on the enhanced data to obtain feature data, including:

[0116] The environmental point cloud data in the enhanced data is standardized to obtain standardized data;

[0117] The scene description data in the enhanced data is transformed into a sub-word sequence to obtain sub-word sequence data;

[0118] Feature data is obtained by extracting features from the standardized data and the sub-word sequence data.

[0119] Optionally, the feature data is aligned to obtain aligned data, including:

[0120] The feature data is subjected to dimensionality reduction processing to obtain dimensionality-reduced data;

[0121] The similarity is calculated by performing similarity calculation on the dimensionality-reduced data to obtain the similarity score.

[0122] The dimensionality-reduced data is aligned based on the similarity to obtain aligned data.

[0123] Optionally, target detection is performed based on the alignment data to obtain target detection results, including:

[0124] Based on the alignment data, candidate bounding boxes are obtained;

[0125] The candidate bounding boxes are adjusted to obtain the adjusted candidate bounding boxes;

[0126] Target detection is performed based on the adjusted candidate bounding boxes to obtain the target detection results.

[0127] Optionally, the processing module 202 is further configured to:

[0128] Based on the target detection results, the robot's movement path is determined.

[0129] The target detection device for robots in this invention acquires environmental point cloud data and scene description data collected by the robot's environmental acquisition device, then performs data augmentation on the environmental point cloud data and scene description data to obtain augmented data, then extracts features from the augmented data to obtain feature data, performs alignment processing on the feature data to obtain aligned data, and finally performs target detection based on the aligned data to obtain target detection results, which helps to improve the accuracy of target detection results.

[0130] It should be noted that this device corresponds to the method described above, and all implementations in the method embodiments described above are applicable to the embodiments of this device and can achieve the same technical effect. Further details are omitted in this embodiment.

[0131] This invention also provides a computing device, including: a processor and a memory storing a computer program, wherein the computer program, when executed by the processor, performs the method as described in any of the above embodiments. All implementations in the above method embodiments are applicable to the embodiments of this device and can achieve the same technical effects. Further details are omitted in this embodiment.

[0132] This invention also provides a computer-readable storage medium storing instructions that, when executed on a computer, cause the computer to perform the method as described in any of the above embodiments. All implementations in the above method embodiments are applicable to the embodiments of this device and can achieve the same technical effects. Further details are omitted in this embodiment.

[0133] It should be noted that in the apparatus and method of the present invention, the components or steps can obviously be decomposed and / or recombined. These decompositions and / or recombinations should be considered equivalent solutions of the present invention. Furthermore, the steps for performing the above series of processes can naturally be performed in the order described and in chronological order, but are not necessarily required to be performed in chronological order. Some steps can be performed in parallel, overlapping, or independently of each other.

[0134] It should be noted that in the above embodiments, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments described above is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0135] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A target detection method for robots, characterized in that, include: Acquire environmental point cloud data and scene description data collected by the robot's environmental acquisition equipment; Data augmentation is performed on the environmental point cloud data and the scene description data to obtain augmented data; Feature extraction is performed on the enhanced data to obtain feature data; The feature data is aligned to obtain aligned data; Target detection is performed based on the alignment data to obtain the target detection result; The enhanced data is subjected to feature extraction to obtain feature data, including: The environmental point cloud data in the enhanced data is standardized to obtain standardized data; through... Downsampling is performed on the environmental point cloud data in the augmented data to obtain downsampled environmental point cloud data; through The downsampled environmental point cloud data is standardized to obtain standardized data; among them, Let N be the average coordinates of all points in the downsampled environmental point cloud data, and N be the number of points in the augmented environmental point cloud data. To enhance the k-th environmental point cloud data in the dataset, , For the i-th preset parameter, For the i-th standardized data, To reduce the mean of the sampled environmental point cloud data, To reduce the standard deviation of the sampling environment point cloud data; The scene description data in the enhanced data is transformed into a sub-word sequence to obtain sub-word sequence data. The scene description data in the enhanced data is then split into multiple words or sub-words, each word or sub-word corresponding to a unique sequence identifier in a preset vocabulary. All words or sub-words are arranged in order according to this unique sequence identifier, and the sub-word at the beginning of the sentence is marked as the beginning of the sentence, and the sub-word at the end of the sentence is marked as the end of the sentence, thus obtaining the sub-word sequence data. ,in, L represents the number of words or subwords. Feature extraction is performed on the standardized data and the word sequence data to obtain feature data; the feature data includes point-level features and feature vectors; wherein, the step of feature extraction on the standardized data includes: using farthest point sampling to select... Each center point For each center point Search radius Neighborhood points Calculate relative coordinates ;pass , Mapping relative coordinates to a high-dimensional space yields the mapped high-dimensional features; through , For all points in the neighborhood Perform element-wise max pooling to obtain aggregated local features; through Generate a set of point-level features ,in, ; Where N is the total number of points in the standardized data; The number of center points selected by sampling from the farthest point. ; For each of the center points sampled from the farthest point, 3 represents the XYZ coordinates of the point, and d represents the additional feature dimension of the point; For the search radius, Center point The set of neighborhood points containing all points that satisfy point ; Relative coordinates, i.e., neighborhood points Relative to the center point Coordinate offset; These are the high-dimensional features after mapping; For mapping functions; For hyperparameters; These are local features after aggregation; It is a set of point-level features; Point-level features; It is a composite function. These are learnable parameters; The steps for feature extraction from sub-word sequence data include: inputting the sub-word sequence data into a pre-defined encoding model and obtaining the output of each layer. ,in This represents the hidden state of layer l. For the dimension; take the word or sub-word features from the last or second-to-last layer as the feature vector: ; in, The hidden state of layer l; T represents the sub-word sequence data; M represents the actual number of words or sub-words in the sub-word sequence data; For dimensions; For feature vectors; The alignment process for the feature data to obtain aligned data includes: The feature data is subjected to dimensionality reduction processing to obtain dimensionality-reduced data; the dimensionality-reduced data includes dimensionality-reduced point-level features and dimensionality-reduced feature vectors; wherein, through Dimensionality reduction is performed on the point-level features in the feature data to obtain the dimensionality-reduced point-level features; among which, These are the point-level features after dimensionality reduction. This is a neural network structure that performs feature transformation and dimensionality reduction on input data through multi-layer nonlinear transformations. As an aggregation feature, , Point-level features The number of center points selected by sampling from the farthest point; pass The dimensionality reduction process is performed on the feature vectors in the feature data to obtain the dimensionality-reduced feature vectors; where, The feature vectors after dimensionality reduction. For feature vectors; The similarity score is obtained by performing similarity calculations on the dimensionality-reduced data; through... Calculate the similarity between the dimensionality-reduced point-level features and the dimensionality-reduced feature vectors; where, Let be the similarity between the dimensionality-reduced point-level features of the i-th point and the dimensionality-reduced feature vector of the i-th point. It is an exponential function. The point-level features of the i-th point after dimensionality reduction. The feature vectors after dimensionality reduction. The bandwidth parameter is used to control the rate of similarity decay. The dimensionality-reduced data is aligned based on the similarity to obtain aligned data; from the similarity set... The key point in selecting the preset number ;pass Alignment processing is performed to obtain aligned data; among which, For the m-th key point, The point-level features of the i-th point after dimensionality reduction. Let be the similarity between the dimensionality-reduced point-level features of the i-th point and the dimensionality-reduced feature vector of the i-th point. This is the a-th aligned data.

2. The target detection method for robots according to claim 1, characterized in that, Acquire environmental point cloud data and scene description data collected by the robot's environmental acquisition equipment, including: Environmental point cloud data is collected by an image acquisition device fixed at the first preset position of the robot; Voice data is collected by a voice acquisition device fixed at the second preset position of the robot; Scene description data is obtained based on the voice data.

3. The target detection method for robots according to claim 1, characterized in that, Data augmentation is performed on the environmental point cloud data and the scene description data to obtain augmented data, including: Based on the environmental point cloud data and the preset search radius and minimum number of neighboring points, outlier points are determined; The environmental point cloud data is filtered based on the outliers to obtain filtered point cloud data; The selected point cloud data and the scene description data are augmented according to a preset data augmentation method to obtain augmented data.

4. The target detection method for robots according to claim 1, characterized in that, Target detection is performed based on the alignment data to obtain target detection results, including: Based on the alignment data, candidate bounding boxes are obtained; The candidate bounding boxes are adjusted to obtain the adjusted candidate bounding boxes; Target detection is performed based on the adjusted candidate bounding boxes to obtain the target detection results.

5. The target detection method for robots according to claim 1, characterized in that, Also includes: Based on the target detection results, the robot's movement path is determined.

6. A target detection device for a robot, characterized in that, include: The acquisition module is used to acquire environmental point cloud data and scene description data collected by the robot's environmental acquisition device; The processing module is used to perform data augmentation on the environmental point cloud data and the scene description data to obtain augmented data; Feature extraction is performed on the enhanced data to obtain feature data; The feature data is aligned to obtain aligned data; Target detection is performed based on the alignment data to obtain the target detection result; The enhanced data is subjected to feature extraction to obtain feature data, including: The environmental point cloud data in the enhanced data is standardized to obtain standardized data; through... Downsampling is performed on the environmental point cloud data in the augmented data to obtain downsampled environmental point cloud data; through The downsampled environmental point cloud data is standardized to obtain standardized data; among them, Let N be the average coordinates of all points in the downsampled environmental point cloud data, and N be the number of points in the augmented environmental point cloud data. To enhance the k-th environmental point cloud data in the dataset, , For the i-th preset parameter, For the i-th standardized data, To reduce the mean of the sampled environmental point cloud data, To reduce the standard deviation of the sampling environment point cloud data; The scene description data in the enhanced data is transformed into a sub-word sequence to obtain sub-word sequence data. The scene description data in the enhanced data is then split into multiple words or sub-words, each word or sub-word corresponding to a unique sequence identifier in a preset vocabulary. All words or sub-words are arranged in order according to this unique sequence identifier, and the sub-word at the beginning of the sentence is marked as the beginning of the sentence, and the sub-word at the end of the sentence is marked as the end of the sentence, thus obtaining the sub-word sequence data. ,in, L represents the number of words or subwords. Feature extraction is performed on the standardized data and the word sequence data to obtain feature data; the feature data includes point-level features and feature vectors; wherein, the step of feature extraction on the standardized data includes: using farthest point sampling to select... Each center point For each center point Search radius Neighborhood points Calculate relative coordinates ;pass , Mapping relative coordinates to a high-dimensional space yields the mapped high-dimensional features; through , For all points in the neighborhood Perform element-wise max pooling to obtain aggregated local features; through Generate a set of point-level features ,in, ; Where N is the total number of points in the standardized data; The number of center points selected by sampling from the farthest point. ; For each of the center points sampled from the farthest point, 3 represents the XYZ coordinates of the point, and d represents the additional feature dimension of the point; For the search radius, Center point The set of neighborhood points containing all points that satisfy point ; Relative coordinates, i.e., neighborhood points Relative to the center point Coordinate offset; These are the high-dimensional features after mapping; For mapping functions; For hyperparameters; These are local features after aggregation; It is a set of point-level features; Point-level features; It is a composite function. These are learnable parameters; The steps for feature extraction from sub-word sequence data include: inputting the sub-word sequence data into a pre-defined encoding model and obtaining the output of each layer. ,in This represents the hidden state of layer l. For the dimension; take the word or sub-word features from the last or second-to-last layer as the feature vector: ; in, The hidden state of layer l; T represents the sub-word sequence data; M represents the actual number of words or sub-words in the sub-word sequence data; For dimensions; For feature vectors; The alignment process for the feature data to obtain aligned data includes: The feature data is subjected to dimensionality reduction processing to obtain dimensionality-reduced data; the dimensionality-reduced data includes dimensionality-reduced point-level features and dimensionality-reduced feature vectors; wherein, through Dimensionality reduction is performed on the point-level features in the feature data to obtain the dimensionality-reduced point-level features; among which, These are the point-level features after dimensionality reduction. This is a neural network structure that performs feature transformation and dimensionality reduction on input data through multi-layer nonlinear transformations. As an aggregation feature, , Point-level features The number of center points selected by sampling from the farthest point; pass The dimensionality reduction process is performed on the feature vectors in the feature data to obtain the dimensionality-reduced feature vectors; where, The feature vectors after dimensionality reduction. For feature vectors; The similarity score is obtained by performing similarity calculations on the dimensionality-reduced data; through... Calculate the similarity between the dimensionality-reduced point-level features and the dimensionality-reduced feature vectors; where, Let be the similarity between the dimensionality-reduced point-level features of the i-th point and the dimensionality-reduced feature vector of the i-th point. It is an exponential function. The point-level features of the i-th point after dimensionality reduction. The feature vectors after dimensionality reduction. The bandwidth parameter used to control the rate of similarity decay; The dimensionality-reduced data is aligned based on the similarity to obtain aligned data; from the similarity set... The key point in selecting the preset number ;pass Alignment processing is performed to obtain aligned data; among which, For the m-th key point, The point-level features of the i-th point after dimensionality reduction. Let be the similarity between the dimensionality-reduced point-level features of the i-th point and the dimensionality-reduced feature vector of the i-th point. This is the a-th aligned data.

7. A computing device, characterized in that, include: A processor, a memory storing a computer program, wherein the computer program, when executed by the processor, performs the method as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The system stores instructions that, when executed on a computer, cause the computer to perform the method as described in any one of claims 1 to 5.

Images