Autonomous mobile robot and detection method fusing radar information and camera information

Abstract

A detection method is applied to an autonomous mobile robot with a camera and a radar, and comprises the following steps: obtaining a camera image and obtaining radar point cloud information; obtaining a depth map; performing semantic segmentation processing on the camera image to obtain a semantic score of each pixel, and superposing the semantic scores to the depth map to generate a virtual key instance; acquiring a plurality of first features and a plurality of second features of the virtual key instance and the radar point cloud information; performing aerial view feature fusion processing on the plurality of first features and the plurality of second features to generate first bounding box information corresponding to the target object; performing region-of-interest feature fusion processing on the plurality of first features and the plurality of second features to generate second bounding box information; and combining the first bounding box information and the second bounding box information to output a detection result. The disclosure also relates to an autonomous mobile robot.

Description

TECHNICAL FIELD

[0001] The present application relates to an autonomous mobile robot, in particular to a detection method for detecting objects of a specified type by an autonomous mobile robot. BACKGROUND

[0002] In recent years, the development of autonomous mobile robots (AMRs) has been rapid, and the market demand has also increased significantly. Such AMRs are mainly applied in environments (such as factories, restaurants, or hotels) full of heterogeneous objects (such as humans, luggage, tables, or cabinets). Therefore, how to quickly and accurately detect objects in the surrounding environment and on the moving path and perform corresponding actions is crucial for AMRs.

[0003] There are many detection methods applied to autonomous mobile robots, such as Voxel-Net, Voxel-RCNN, Point painting, SFD, and VirConv, but each has its own advantages and disadvantages. For example, the calculation of SFD is very large, and although VirConv has improved SFD and is currently the best detection method, it uses all virtual points detected by radar for calculation, which causes a considerable computational burden, affecting the speed of detection by autonomous mobile robots, and also causing a large amount of energy consumption by autonomous mobile robots. SUMMARY

[0004] The present application provides an autonomous mobile robot and a detection method that fuses radar information and camera information, which combines radar information, camera information, and a deep learning model to achieve accurate detection of objects of a specified type, not only reducing the amount of calculation required for detection, but also improving the safety of autonomous mobile robots due to improved accuracy.

[0005] In one embodiment, the detection method that fuses radar information and camera information of the present application is applied to an autonomous mobile robot having a camera, a radar, and a processor, and includes:

[0006] Step a) obtains a camera image of an environment by the camera and obtains radar point cloud information of the environment by the radar;

[0007] Step b) obtaining, by the processor, a depth map based on the camera image, performing a semantic segmentation process on the camera image to obtain a semantics score corresponding to each pixel being a specified type, and painting the semantics score onto the depth map to generate a virtual key instance (VKI), wherein the virtual key instance records a plurality of dense points in three-dimensional space for displaying a target object of the specified type;

[0008] Step c) performing, by the processor, feature extraction on the virtual key instance and the radar point cloud information to obtain a plurality of virtual key instance features and a plurality of radar point cloud information features;

[0009] Step d) performing, by the processor, a bird's eye view feature fusion process on the plurality of virtual key instance features and the plurality of radar point cloud information features in a two-dimensional plane by a virtual instance focused fusion (VIFF) model to generate a first bounding box information corresponding to the target object in the radar point cloud information;

[0010] Step e) performing, by the virtual instance focused fusion model, a region of interest feature fusion process on the plurality of virtual key instance features and the plurality of radar point cloud information features in a three-dimensional space to generate a second bounding box information, wherein the second bounding box information describes the first bounding box information; and

[0011] Step f) combining, by the virtual instance focused fusion model, the first bounding box information and the second bounding box information to output a detection result for describing the target object in the radar point cloud information.

[0012] As described above, wherein the specified type is a human type, and the target object is a human.

[0013] As described above, wherein step b) includes performing, by the processor, a depth completion process on the camera image and the radar point cloud information to generate the depth map, or directly obtaining the depth map from the camera image generated by the camera as an RGB-D camera.

[0014] As described above, wherein step b) includes generating a target object mask by the processor after performing the semantic segmentation process on the camera image, and filtering the depth map by the target object mask to reduce the complexity of the depth map, wherein the processor performs the semantic segmentation process using a deep learning model trained based on data of the specified type.

[0015] As described above, step b) further includes the processor overlaying the semantic score onto the depth map to generate overlaid information, and converting the overlaid information to a radar coordinate system used by the radar to generate the virtual key instance.

[0016] As described above, step c) includes the processor performing a 3D feature acquisition program on the radar point cloud information and the virtual key instance to obtain multiple 3D features of the radar point cloud information and multiple 3D features of the virtual key instance, and in step e), the virtual instance focusing fusion model performs the region of interest feature fusion processing on the multiple 3D features of the radar point cloud information and the multiple 3D features of the virtual key instance.

[0017] As described above, step c) is followed by step c1): the processor performs a dimensionality reduction process on the multiple radar point cloud information 3D features and the multiple virtual key instance 3D features to generate multiple radar point cloud information 2D features and multiple virtual key instance 2D features, wherein in step d), the virtual instance focusing fusion model is to perform the bird's-eye view feature fusion process on the multiple radar point cloud information 2D features and the multiple virtual key instance 2D features.

[0018] As described above, step d) includes the following steps to achieve the bird's-eye view feature fusion process:

[0019] Step d1) Concatenate the two-dimensional features of the multiple radar point cloud information and the corresponding two-dimensional features of the multiple virtual key instances along the channel dimension to generate multiple concatenated features, and perform channel attention processing on the multiple concatenated features to generate multiple processed features.

[0020] Step d2) Perform a spatial attention process on the multiple processed features to generate multiple fused bird's-eye view features; and

[0021] Step d3) Generate the first bounding box information based on the multiple fused bird's-eye view features.

[0022] As described above, step e) includes the following steps to achieve the feature fusion processing of the region of interest:

[0023] Step e1) Perform a first self-attention process on the three-dimensional features of the multiple radar point cloud information to obtain a first vector representation;

[0024] Step e2) Perform a first frequency self-attention process on the first vector representation to generate a first query vector (Q), a first key vector (K) and a first value vector (V);

[0025] Step e3) Perform a second self-attention process on the 3D features of the multiple virtual key instances to obtain a second vector representation;

[0026] Step e4) Perform a second frequency domain self-attention process on the second vector representation to generate a second query vector (Q), a second key vector (K), and a second value vector (V);

[0027] Step e5) Perform a first cross-attention process on the first value vector, the first key vector, and the second query vector to generate a third vector representation, and perform a second cross-attention process on the second value vector, the second key vector, and the first query vector to generate a fourth vector representation;

[0028] Step e6) Perform a global max pooling process on the third vector representation, the fourth vector representation, and the output of the second frequency domain self-attention processing to produce a first output;

[0029] Step e7) concatenate the first output and the output of the first frequency domain self-attention processing to generate a fused three-dimensional region of interest feature; and

[0030] Step e8) Generate the second bounding box information based on the fused bird's-eye view features and the fused 3D region of interest features.

[0031] As described above, the first frequency domain self-attention processing is a frequency domain self-attention processing that focuses on the low-frequency domain ((Frequency Self-Attention(Low-frequency),Fsal), and the second frequency domain self-attention processing is a frequency domain self-attention processing that focuses on the full value ((Frequency Self-Attention(All Value),Fsav).

[0032] In one embodiment, the autonomous mobile robot of this application includes:

[0033] A camera, configured to capture camera images of an environment;

[0034] A radar is configured to acquire radar point cloud information of the environment; and

[0035] A processor, electrically connected to the camera and the radar, is configured to execute a plurality of instructions to perform the following actions:

[0036] Action a) Obtain a depth map based on the camera image, perform a semantic segmentation process on the camera image to obtain a semantic score for each pixel corresponding to a specified type, and superimpose the semantic score onto the depth map to generate a virtual key instance, wherein the virtual key instance records a plurality of dense points in three-dimensional space used to display a target object of the specified type.

[0037] Action b) Execute a feature acquisition procedure on the virtual key instance and the radar point cloud information to obtain multiple virtual key instance features and multiple radar point cloud information features;

[0038] Action c) Perform a two-dimensional bird's-eye view feature fusion process on the features of the multiple virtual key instances and the features of the multiple radar point cloud information using a virtual instance focusing fusion model to generate a first bounding box information corresponding to the target object in the radar point cloud information;

[0039] Action d) The virtual instance focusing fusion model performs a three-dimensional region-of-interest feature fusion process on the features of multiple virtual key instances and the features of multiple radar point cloud information to generate a second bounding box information, wherein the second bounding box information describes the first bounding box information; and

[0040] Action e) The virtual instance focuses the fusion model to combine the first bounding box information and the second bounding box information to output a detection result describing the target object in the radar point cloud information.

[0041] As described above, the specified type is human, and the target object is human.

[0042] As described above, the action a) includes performing a depth completion process on the camera image and the radar point cloud information to generate the depth map, or directly obtaining the depth map from the camera image generated by the camera, which is an RGB-D camera.

[0043] As described above, the action a) includes generating a target object mask after performing the semantic segmentation process on the camera image, and filtering the depth map by the target object mask to reduce the complexity of the depth map, wherein the processor is configured to perform the semantic segmentation process using a deep learning model trained on the specified type of data.

[0044] As described above, action a) further includes overlaying the semantic score onto the depth map to generate overlaid information, and converting the overlaid information to a radar coordinate system used by the radar to generate the virtual key instance.

[0045] As described above, action b) includes performing a 3D feature acquisition procedure on the radar point cloud information and the virtual key instance to obtain multiple three-dimensional features of the radar point cloud information and multiple three-dimensional features of the virtual key instance, and in action d), the virtual instance focusing fusion model performs the region of interest feature fusion processing on the multiple three-dimensional features of the radar point cloud information and the multiple three-dimensional features of the virtual key instance.

[0046] As described above, action b) is followed by action b1): performing a dimensionality reduction process on the multiple radar point cloud information 3D features and the multiple virtual key instance 3D features to generate multiple radar point cloud information 2D features and multiple virtual key instance 2D features, wherein in action c), the virtual instance focusing fusion model is to perform the bird's-eye view feature fusion process on the multiple radar point cloud information 2D features and the multiple virtual key instance 2D features.

[0047] As described above, action c) includes the following actions to achieve the bird's-eye view feature fusion process:

[0048] Action c1) The two-dimensional features of the multiple radar point cloud information and the corresponding two-dimensional features of the multiple virtual key instances are stitched together along the channel dimension to generate multiple stitched features, and a channel attention process is performed on the multiple stitched features to generate multiple processed features.

[0049] Action c2) Performs a spatial attention process on the multiple processed features to generate multiple fused bird's-eye view features; and

[0050] Action c3) generates the first bounding box information based on the multiple fused bird's-eye view features.

[0051] As described above, action d) includes the following actions to achieve the feature fusion processing of the region of interest:

[0052] Action d1) performs a first self-attention process on the three-dimensional features of the multiple radar point cloud information to obtain a first vector representation;

[0053] Action d2) performs a first frequency domain self-attention process on the first vector representation to generate a first query vector, a first key vector and a first value vector;

[0054] Action d3) performs a second self-attention process on the 3D features of the multiple virtual key instances to obtain a second vector representation;

[0055] Action d4) performs a second frequency domain self-attention process on the second vector representation to generate a second query vector, a second key vector, and a second value vector;

[0056] Action d5) performs a first cross-attention process on the first value vector, the first key vector, and the second query vector to generate a third vector representation, and performs a second cross-attention process on the second value vector, the second key vector, and the first query vector to generate a fourth vector representation;

[0057] Action d6) performs a global max pooling process on the third vector representation, the fourth vector representation, and the output of the second frequency domain self-attention processing to produce a first output;

[0058] Action d7) concatenates the first output and the output of the first frequency domain self-attention processing to generate a fused three-dimensional region of interest feature; and

[0059] Action d8) generates the second bounding box information based on the fused bird's-eye view features and the fused 3D region of interest features.

[0060] As described above, the first frequency domain self-attention processing is a frequency domain self-attention processing that focuses on the low frequency domain, and the second frequency domain self-attention processing is a frequency domain self-attention processing that focuses on the full value.

[0061] Compared to related technologies, this application integrates radar and camera information, solving the problem of poor detection results caused by the sparsity of radar information. Furthermore, by improving the voxel-based detection method, it overcomes the shortcomings of high computational load and slow detection speed in previous methods. In addition, this application incorporates semantic information during processing, which can further improve detection accuracy and enhance the safety of autonomous mobile robots. Attached Figure Description

[0062] Figure 1 This is an embodiment of a block diagram of the autonomous mobile robot of this application;

[0063] Figure 2 This is an embodiment of the detection architecture of this application;

[0064] Figure 3 This is an embodiment of the flowchart of the detection method of this application;

[0065] Figure 4 This is an embodiment of the virtual instance-focused fusion model of this application;

[0066] Figure 5 This is an embodiment of the workflow diagram for the bird's-eye view feature fusion processing of this application;

[0067] Figure 6 This is an embodiment of the workflow diagram for the feature fusion processing of the region of interest in this application;

[0068] Figure 7 This is an example illustrating the detection results of this application.

[0069] In the attached figures, the following labels are used:

[0070] 1: Autonomous mobile robot

[0071] 11: Processor

[0072] 12: Camera

[0073] 121: Camera Images

[0074] 13: Radar

[0075] 131: Radar Point Cloud Information

[0076] 14: Mobile mechanism

[0077] 15: Human-computer interface

[0078] 2: Semantic Key Instance Selection Model

[0079] 21: Deep Completion Processing

[0080] 22: Depth Map

[0081] 23: Semantic Segmentation Processing

[0082] 24: Target object masking

[0083] 25: Semantic score

[0084] 26: Information after overlay

[0085] 3: Virtual Instance Focused Fusion Model

[0086] 31: Bird's-eye view feature fusion processing

[0087] 310: Integrating features from bird's-eye view

[0088] 311: Channel Attention Processing

[0089] 312: Spatial Attention Processing

[0090] 32: Focus on regional feature fusion processing

[0091] 320: Integrating 3D Region of Interest Features

[0092] 321, 322: Self-attention processing

[0093] 323, 324: Frequency Domain Self-Attention Processing

[0094] 325:326: Cross-attention processing

[0095] 4: Virtual Key Instance

[0096] 5: Feature Acquisition and Processing

[0097] 511: 3D Features of Radar Point Cloud Information

[0098] 512: Two-dimensional features of radar point cloud information

[0099] 521: 3D Features of Virtual Key Instances

[0100] 522: Two-dimensional features of virtual key instances

[0101] 6: Dimensionality reduction

[0102] 7: Detection Results

[0103] 8: Target object

[0104] S31~S38: Detection Steps

[0105] S51~S54, S61~S68: Working Steps Detailed Implementation

[0106] This application discloses an autonomous mobile robot (AMR) that integrates radar and camera information for detection. By fusing radar and camera information, it solves the problem of insufficient accuracy caused by the sparsity of radar information in previous detection methods that relied solely on radar information. Furthermore, the detection method in this application improves upon voxel-based detection methods, addressing the issues of high computational cost and slow detection speed caused by referencing too much information in previous methods, thereby reducing the energy consumption of the autonomous mobile robot.

[0107] Furthermore, this application incorporates semantic information when processing camera information, thereby further improving detection accuracy. Moreover, the increased detection accuracy also enhances the safety of the autonomous mobile robot.

[0108] Please refer to the following first. Figure 1 This is an embodiment of the block diagram of the autonomous mobile robot of this application. For example... Figure 1 As shown, the autonomous mobile robot 1 of this application mainly includes a processor 11, a camera 12, and a radar 13. The processor 11 is electrically connected to the camera 12 and the radar 13, controlling the camera 12 to take pictures in the environment where the autonomous mobile robot 1 is located to obtain camera information (e.g., ...). Figure 2 The camera image 121 shown is used to control the radar 13 to sense the environment in order to obtain radar information (e.g., the radar image 121 shown). Figure 2The radar point cloud information shown is 131. In one embodiment, radar 13 may be a millimeter-wave radar or an optical radar (Lidar).

[0109] In one embodiment, the processor 11 can be implemented by a Central Processing Unit (CPU), a Microcontroller Unit (MCU), a Programmable Logic Controller (PLC), a System on Chip (SoC), or a Field Programmable Gate Array (FPGA). In this application, the processor 11 executes a number of instructions to internally establish a Semantic Key Instance Selection (SKIS) model 2 and a Virtual Instance Focused Fusion (VIFF) model 3. Both SKIS model 2 and VIFF model 3 are software-implemented models. SKIS model 2 is used to generate a number of dense points (i.e., a Virtual Key Instance (VKI)) that can display objects of a specified type in three-dimensional space based on camera information. VIFF model 3 is used to fuse radar information and the VKI to generate detection results for target objects of a specified type (described in detail later).

[0110] It is worth mentioning that this application fuses camera and radar information on the feature set using VIFF model 3, thus the final detection result can combine the advantages of camera 12 and radar 13. For example, even if the autonomous mobile robot 1 is in a dim environment and camera 12 malfunctions, radar 13 will not malfunction due to the dimness, so the technical solution of this application can still output reliable detection results. For example, if it is determined from the content of VKI that camera 12 is malfunctioning (e.g., VKI is a 0 matrix), VIFF model 3 can automatically increase the weight of radar features and decrease the weight of camera features during fusion, so that the failure of camera 12 will not lead to an invalid detection result.

[0111] like Figure 1As shown, the autonomous mobile robot 1 of this application may also have a movement mechanism 14 electrically connected to the processor 11. In some embodiments, the movement mechanism 14 may include, for example, directional wheels and / or steering wheels, and the autonomous mobile robot 1 stands on the ground via the movement mechanism 14. In this way, the processor 11 can generate corresponding control commands based on the detection results and control the movement mechanism 14, thereby realizing the movement of the autonomous mobile robot 1.

[0112] like Figure 1 As shown, the autonomous mobile robot 1 of this application may also have a human-machine interface 15 electrically connected to the processor 11. In some embodiments, the human-machine interface 15 may be, for example, a physical button, a touch button, a touch screen, or a voice input interface. By operating the human-machine interface 15, the user can set up the autonomous mobile robot 1 and issue relevant commands.

[0113] This application also discloses a detection method that integrates radar information and camera information. In this application, the processor 11 of the autonomous mobile robot 1 can execute a plurality of instructions to internally execute the SKIS model 2 and VIFF model 3, and implement the various specific steps of the detection method described below based on the SKIS model 2 and VIFF model 3.

[0114] Please also refer to Figure 2 and Figure 3 ,in Figure 2 This is a schematic diagram of an embodiment of the detection architecture of this application. Figure 3 This is an embodiment of the detection method flowchart of this application.

[0115] like Figure 2 As shown, the detection method of this application involves an autonomous mobile robot 1 continuously acquiring camera images 121 and radar point cloud information 131 via camera 12 and radar 13, respectively, while moving in the environment. The processor 11 of the autonomous mobile robot 1 processes the camera images 121 (and radar point cloud information 131) using its internal SKIS model 2 to generate virtual key instances 4. These virtual key instances 4 record a plurality of dense points that can represent objects (e.g., humans) of a specified type (e.g., human type) in the environment in three-dimensional space. Next, the processor 11 of the autonomous mobile robot 1 processes the virtual key instances 4 and the radar point cloud information 131 using its internal VIFF model 3 to generate a detection result 7 that indicates objects of a specified type in the radar point cloud information 131. In one embodiment, the detection result 7 is a bounding box that can encompass the target object in the radar point cloud information 131, but this is not a limitation.

[0116] In the following paragraphs of the instruction manual, it will be combined with Figure 2 architecture and Figure 3The process will be explained together.

[0117] First, the autonomous mobile robot 1 moves through the environment via the mobility mechanism 14. During this movement, the camera 12 acquires camera images 121 of the environment, and the radar 13 acquires radar point cloud information 131 of the environment (step S31). Next, the processor 11 processes the camera images 121 to obtain a corresponding depth map 22 (step S32). In this application, the depth map 22 records a group of dense pixels (e.g., including 500x1200 pixels) used to represent depth.

[0118] In one embodiment, the camera 12 of the autonomous mobile robot 1 is an RGB camera. In this embodiment, the processor 11 performs depth completion processing 21 on the camera image 121 and radar point cloud information 131 to generate the depth map 22. Specifically, the processor 11 may include a detection model, which is a deep learning model pre-trained using data of a specified type (e.g., human type). In this embodiment, the processor 11 uses this detection model to perform recognition processing on the camera image 121 to obtain images of target objects (e.g., humans) belonging to a specified type from the camera image 121. Then, the processor 11 performs depth completion processing 21 on the acquired image and radar point cloud information 131 to generate the depth map 22.

[0119] In another embodiment, the camera 12 of the autonomous mobile robot 1 is an RGB-D camera. In this embodiment, the processor 11 can directly obtain the depth map 22 from the camera image 121 without performing depth completion processing 21.

[0120] In another embodiment, the processor 11 may include a detection model, which is a deep learning model pre-trained using data of a specified type (e.g., human type). Since the radar point cloud information 131 already contains depth information, in this embodiment, the processor 11 can use this detection model to identify the radar point cloud information 131 in step S32, thereby directly obtaining the depth map 22 corresponding to the target object of the specified type from the radar point cloud information 131.

[0121] In summary, the processor 11 of this application can obtain the depth map 22 based solely on camera image 121, solely on radar point cloud information 131, or simultaneously based on both camera image 121 and radar point cloud information 131, but this is not a limitation.

[0122] After step S32, the processor 11 further performs semantic segmentation processing 23 on the camera image 121 to obtain semantic scores for each pixel in the camera image 121 relative to a specified type (step S33). Specifically, as described above, the processor 11 includes a detection model that can identify target objects in the camera image 121. Based on the relationship between each pixel in the camera image 121 and the target object (e.g., no target object, partial target object, and specific target object), the processor 11 can assign a semantic score 25 to each pixel. Finally, the processor 11 uses point painting technology to overlay the semantic scores 25 onto the depth map 22 to generate the virtual key instance 4 (step S34). In this application, the virtual key instance 4 records a complex number of dense points that represent the specified type of target object in three-dimensional space.

[0123] It is worth mentioning that after the processor 11 overlays the semantic score 25 onto the depth map 22, it can obtain the overlaid information 26. By converting the overlaid information 26 to the radar coordinate system (not labeled in the figure) used by the radar 13, the processor 11 can obtain the virtual key instance 4.

[0124] In one embodiment, when the processor 11 performs the semantic segmentation processing 23 on the camera image 121, it identifies the camera image 121 using the detection model to generate a target object mask 24. The target object mask 24 contains only images of objects of a specified type identified by the detection model. In this way, the processor 11 can use the target object mask 24 to filter the depth map 22, reducing the complexity of the depth map 22 (i.e., removing information from the depth map 22 that does not belong to the target object), reducing the number of points that need to be calculated, and improving the processing speed. Through this filtering procedure, the accuracy of the virtual key instance 4 finally output by the SKIS model 2 can be further optimized.

[0125] Since the lighting conditions in the environment where the autonomous mobile robot 1 is located may change continuously, in one embodiment, the processor 11 can detect the lighting conditions in the environment (e.g., too dark, normal, or overexposed) and switch between different methods to generate the depth map 22 and / or the target object mask 24 at any time depending on the lighting conditions (e.g., obtained only from camera image 121, obtained only from radar point cloud information 131, or obtained by combining camera image 121 and radar point cloud information 131).

[0126] Next, the processor 11 performs feature acquisition processing 5 on the virtual key instance 4 and the radar point cloud information 131 respectively to obtain multiple virtual key instance features and multiple radar point cloud information features (step S35). Then, the processor 11 processes the multiple virtual key instance features and multiple radar point cloud information features through the VIFF model 3 to generate detection results 7.

[0127] Specifically, the feature acquisition process 5 employs a 3D feature acquisition program with a 3D backbone. After executing the 3D feature acquisition program on the virtual key instance 4 and the radar point cloud information 131 respectively, the processor 11 can obtain multiple 3D features of the radar point cloud information and multiple 3D features of the virtual key instance. Furthermore, by performing dimensionality reduction processing 6 on the multiple 3D features of the radar point cloud information and multiple 3D features of the virtual key instance, the processor 11 can generate multiple 2D features of the radar point cloud information and multiple 2D features of the virtual key instance. Specifically, the dimensionality reduction processing 6 compresses the height information in the multiple 3D features of the radar point cloud information and multiple 3D features of the virtual key instance, thereby converting the 3D features presented in 3D space (X,Y,Z) into 2D features presented in 2D plane (X,Y) (also known as bird's-eye view (BEV) features).

[0128] like Figure 2 As shown, the VIFF model 3 of this application also performs bird's-eye view feature fusion (BEV Feature Fusion) processing 31 and region of interest feature fusion (3D ROI Feature Fusion) processing 32 based on software design. The bird's-eye view feature fusion processing 31 is used to process the two-dimensional features of the multiple radar point cloud information and the two-dimensional features of the multiple virtual key instances, while the region of interest feature fusion processing 32 is used to process the three-dimensional features of the multiple radar point cloud information and the three-dimensional features of the multiple virtual key instances.

[0129] Specifically, the processor 11 performs a two-dimensional bird's-eye view feature fusion process 31 on multiple virtual key instance features and multiple radar point cloud information features (i.e., multiple virtual key instance two-dimensional features and multiple radar point cloud information two-dimensional features) using the VIFF model 3 to generate first bounding box information corresponding to the target object in the radar point cloud information 131 (step S36). In one embodiment, the first bounding box information records a coarse and unrefined detection box (e.g., no orientation and fixed length and width), and this detection box roughly covers the target object in the radar point cloud information 131.

[0130] Next, the processor 11 performs a three-dimensional region of interest feature fusion processing 32 on the multiple virtual key instance features and multiple radar point cloud information features (i.e., multiple virtual key instance 3D features and multiple radar point cloud information 3D features) using the VIFF model 3, to generate second bounding box information used to describe the first bounding box information (step S37). In one embodiment, the second bounding box information records the orientation and specific information such as length and width of the detection box.

[0131] Finally, the processor 11 combines the first bounding box information and the second bounding box information through the VIFF model 3 to output the detection result 7 (step S38) describing the target object in the radar point cloud information 131.

[0132] like Figure 2 As shown, the VIFF model 3 of this application performs a two-stage detection. First, the bird's-eye view feature fusion processing 31 performs the first-stage detection to generate a coarse detection box for the target object; then, the region of interest feature fusion processing 32 performs the second-stage detection to optimize the orientation, length, width and other information of this detection box, and finally outputs a detection box that accurately surrounds the target object (e.g., a human) in three-dimensional space (i.e., the detection result 7).

[0133] Please also refer to Figure 7 This is an embodiment illustrating the detection results of this application. For example... Figure 7 As shown, when the autonomous mobile robot 1 moves in the environment, and the camera 12 and radar 13 acquire camera images 121 and radar point cloud information 131 respectively, the target object 8 in the environment can be identified through the SKIS model 2. Figure 7 (Taking humans as an example); and by further processing the features through the VIFF model 3, a detection result 7 (i.e., bounding box) of one or more target objects 8 in the environment can be generated on the radar point cloud information 131.

[0134] Please also refer to Figure 4 , Figure 5 and Figure 6 ,in Figure 4 This is an embodiment of the virtual instance-focused fusion model of this application. Figure 5 This is an embodiment of the workflow diagram for the bird's-eye view feature fusion processing of this application, and Figure 6 This is an example of a workflow diagram for the feature fusion processing of the region of interest in this application.

[0135] Figure 4 The structure of VIFF model 3 of this application is disclosed. (By...) Figure 4It can be seen that the VIFF model 3 is used to perform a two-stage fusion process that includes bird’s-eye view feature fusion processing 31 and region of interest feature fusion processing 32. Bird’s-eye view feature fusion processing 31 is a fusion process performed on a two-dimensional plane, while region of interest feature fusion processing 32 is a process performed in three-dimensional space.

[0136] The following will combine Figure 4 as well as Figure 5 The feature fusion processing of the bird's-eye view is explained in section 31.

[0137] First, as mentioned above, after obtaining multiple 3D features 511 of radar point cloud information and multiple 3D features 521 of virtual key instances through feature acquisition processing 5, processor 11 needs to perform dimensionality reduction processing 6 on the multiple 3D features 511 of radar point cloud information and multiple 3D features 521 of virtual key instances to obtain multiple 2D features 512 of radar point cloud information and multiple 2D features 522 of virtual key instances in a 2D plane (step S51). Next, processor 11 uses the multiple 2D features 512 of radar point cloud information and multiple 2D features 522 of virtual key instances as input to bird's-eye view feature fusion processing 31.

[0138] The bird's-eye view feature fusion processing 31 concatenates multiple radar point cloud information two-dimensional features 512 and multiple virtual key instance two-dimensional features 522 along the channel dimension to generate multiple concatenated features. Then, channel-attention processing 311 is performed on these concatenated features to generate multiple processed features (step S52). Next, the bird's-eye view feature fusion processing 31 performs spatial-attention processing 312 on the multiple processed features after channel-attention processing 311 to generate multiple fused bird's-eye view features 310 (step S53). Finally, the bird's-eye view feature fusion processing 31 can generate the first bounding box information based on the multiple fused bird's-eye view features 310 (step S54). As mentioned above, the first bounding box information records the detection boxes that roughly cover the target objects in the radar point cloud information 131. Therefore, the bird's-eye view feature fusion processing 31 will incorporate the first bounding box information and / or the fused bird's-eye view features 310 into the region of interest feature fusion processing 32 as a three-dimensional region proposal, so that the region of interest feature fusion processing 32 knows which regions should be focused on.

[0139] Specifically, channel attention processing 311 allows the model to adjust and learn the importance of different feature channels, thus determining which channels within a feature are important. Spatial attention processing 312 complements channel attention processing 311 to improve model performance. Specifically, spatial attention processing 312 uses a heatmap of the virtual key instance 4 to enhance the two-dimensional feature 512 of the radar point cloud information, and allows the model to adjust and learn the importance of different spatial locations within the feature, thus determining which spaces within the feature are important.

[0140] The channel attention processing 311 and spatial attention processing 312 are common techniques in artificial intelligence models, and their working principles will not be elaborated here.

[0141] The following will combine Figure 6 as well as Figure 5 The feature fusion processing of the region of interest is explained in section 32.

[0142] As shown in the figure, after obtaining multiple radar point cloud information 3D features 511 and multiple virtual key instance 3D features 521, the processor 11 directly uses the multiple radar point cloud information 3D features 511 and multiple virtual key instance 3D features 521 as input to the region of interest feature fusion processing 32. The region of interest feature fusion processing 32 processes the multiple radar point cloud information 3D features 511 and multiple virtual key instance 3D features 521 respectively, including performing a first self-attention processing 321 on the multiple radar point cloud information 3D features 511 to obtain a first vector representation (step S61) and performing a second self-attention processing 322 on the multiple virtual key instance 3D features 521 to obtain a second vector representation (step S63).

[0143] Specifically, the region of interest feature fusion processing 32 of this application adopts bidirectional self-attention features, which can reduce the noise generated by the depth completion processing 21 from the front end, and can compensate for the incomplete correction of the camera 12 and radar 13.

[0144] Next, the region feature fusion processing 32 performs a first frequency domain self-attention processing 323 on the first vector representation generated by the first self-attention processing 321 to generate a first query vector (Query, Q), a first key vector (Key, K), and a first value vector (Value, V) (step S62), and performs a second frequency domain self-attention processing 324 on the second vector representation generated by the second self-attention processing 322 to generate a second query vector (Query, Q), a second key vector (Key, K), and a second value vector (Value, V) (step S64).

[0145] In one embodiment, the first frequency domain self-attention processing 323 is a frequency domain self-attention processing (Fsal) that focuses on the low-frequency domain, and the second frequency domain self-attention processing 324 is a frequency domain self-attention processing (Fsav) that focuses on the full value, but is not limited thereto. The first frequency domain self-attention processing 323 and the second frequency domain self-attention processing 324 decompose the frequency domain into different parts (e.g., high frequency and low frequency), and apply different self-attention processing to the features of different parts respectively.

[0146] Next, the region feature fusion processing 32 performs a first cross-attention processing 325 on the first value vector, the first key vector generated by the first frequency domain self-attention processing 323 and the second query vector generated by the second frequency domain self-attention processing 324 to generate a third vector representation, and performs a second cross-attention processing 326 on the second value vector, the second key vector generated by the second frequency domain self-attention processing 324 and the first query vector generated by the first frequency domain self-attention processing 323 to generate a fourth vector representation (step S65).

[0147] Cross-attention processing allows a model to focus on features of a different sequence while processing features of one sequence.

[0148] Next, the feature fusion processing 32 focuses on the third vector representation generated by the first cross-attention processing 325, the fourth vector representation generated by the second cross-attention processing 326, and the output of the second frequency domain self-attention processing 324, and performs global max pooling processing to produce the first output (step S66). Specifically, global max pooling processing retains the most salient and active information at each position from the feature representations from different processing paths.

[0149] Finally, the attention region feature fusion processing 32 concatenates the first output of the global max pooling processing and the output of the first frequency domain self-attention processing 323 to generate a fused 3D attention region feature 320 (step S67). Furthermore, the attention region feature fusion processing 32 then generates second bounding box information based on the fused bird's-eye view feature 310 output by the bird's-eye view feature fusion processing 31 and / or the first bounding box information, as well as the fused 3D attention region feature 320 (step S68). Ultimately, as... Figure 3 As shown in step S38, the processor 11 can generate a detection result 7 indicating the target object in the radar point cloud information 131 based on the first bounding box information and the second bounding box information.

[0150] The self-attention processing, frequency domain self-attention processing, cross-attention processing, global max pooling processing, query vector, key vector, and value vector mentioned above are common techniques in the field of artificial intelligence, and will not be elaborated further here.

[0151] As described above, the autonomous mobile robot 1 of this application can acquire camera images 121 through camera 12, acquire radar point cloud information 131 through radar 13, and implement the various specific steps of the detection method of this application by executing a number of instructions through processor 11. Specifically, processor 11 executes a number of instructions to perform the following actions in sequence:

[0152] Action 1: Obtain depth map 22 based on camera image 121, perform semantic segmentation processing 23 on camera image 121 to obtain semantic score 25 for each pixel corresponding to a specified type, and superimpose semantic score 25 onto depth map 22 to generate virtual key instance 4, wherein virtual key instance 4 records multiple dense points in three-dimensional space used to display target objects of a specified type.

[0153] Action 2: Execute a feature acquisition program on virtual key instance 4 and radar point cloud information 131 to obtain features of multiple virtual key instances and features of multiple radar point cloud information.

[0154] Action 3: Perform bird's-eye view feature fusion processing 31 on a two-dimensional plane on multiple virtual key instance features and multiple radar point cloud information features through VIFF model 3 to generate the first bounding box information of the target object corresponding to the radar point cloud information 131.

[0155] Action 4: The VIFF model 3 performs a three-dimensional spatial feature fusion process on the features of multiple virtual key instances and multiple radar point cloud information features to generate second bounding box information, wherein the second bounding box information describes the first bounding box information.

[0156] Action 5: The VIFF model 3 combines the first bounding box information and the second bounding box information to output the detection result 7, which describes the target object in the radar point cloud information 131.

[0157] Specifically, by executing complex instructions, the autonomous mobile robot 1 of this application can achieve the following: Figure 3 , Figure 5 and Figure 7 The flowchart shows the specific steps.

[0158] The applicant tested the average accuracy (AP) of the detection method, Voxel R-CNN, and VirConv-L based on the JRDB dataset. Intersection over Union (IoU) thresholds of 0.5, 0.25, and 0.25 were used for easy, medium, and hard modes (the IoU threshold was the same as for the medium mode; the difficulty difference stemmed from the characteristics of the object itself). The experimental results are shown in Tables 1 and 2 below.

[0159]

[0160] Table 1

[0161] Table 1 above shows the results of 3D pedestrian detection on the JDDB validation set. As can be seen from Table 1, the accuracy and computation time of the algorithm in this application significantly outperform VirConv-L. Furthermore, although the computation time of the algorithm in this application is still slower than Voxel R-CNN, its accuracy surpasses Voxel R-CNN in easy, medium, and difficult modes.

[0162]

[0163] Table 2

[0164] Table 2 above shows the results of bird's-eye view (BEV) pedestrian detection on the JDDB validation set. As can be seen from Table 2, the algorithm presented in this application outperforms VirConv-L in both accuracy and computation time. Furthermore, although the computation time of the algorithm presented in this application is still slower than Voxel R-CNN, it outperforms Voxel R-CNN in accuracy across easy, medium, and difficult modes.

[0165] The technical solution of this application solves the problem of poor detection results for target objects (such as humans) by fusing radar and camera information. Furthermore, by using the SKIS and VIFF models, this application reduces the amount of information required, thereby overcoming the shortcomings of high computational load and slow detection speed in previous methods. Moreover, the SKIS model of this application further improves detection accuracy by incorporating semantic information during processing.

Claims

1. A detection method integrating radar information and camera information, applied to an autonomous mobile robot having a camera, a radar, and a processor, characterized in that, include: Step a) The camera acquires a camera image of the environment, and the radar acquires radar point cloud information of the environment; Step b) The processor obtains a depth map based on the camera image, performs a semantic segmentation process on the camera image to obtain a semantic score for each pixel corresponding to a specified type, and superimposes the semantic score onto the depth map to generate a virtual key instance, wherein the virtual key instance records a plurality of dense points in three-dimensional space used to display a target object of the specified type. Step c) The processor performs a feature acquisition program on the virtual key instance and the radar point cloud information to obtain multiple virtual key instance features and multiple radar point cloud information features; Step d) The processor performs a bird's-eye view feature fusion process on a two-dimensional plane on the features of the multiple virtual key instances and the features of the multiple radar point cloud information through a virtual instance focusing fusion model to generate a first bounding box information corresponding to the target object in the radar point cloud information. Step e) The virtual instance focusing fusion model performs a region of interest feature fusion processing in three-dimensional space on the features of the multiple virtual key instances and the features of the multiple radar point cloud information to generate a second bounding box information, wherein the second bounding box information describes the first bounding box information; and Step f) The virtual instance focusing fusion model combines the first bounding box information and the second bounding box information to output a detection result describing the target object in the radar point cloud information.

2. The detection method according to claim 1, characterized in that, The specified type is human, and the target object is human.

3. The detection method according to claim 1, characterized in that, Step b) includes the processor performing a depth completion process on the camera image and the radar point cloud information to generate the depth map, or the depth map being directly obtained from the camera image generated by the camera, which is an RGB-D camera.

4. The detection method according to claim 1, characterized in that, Step b) includes the processor performing the semantic segmentation process on the camera image to generate a target object mask, and the target object mask filtering the depth map to reduce the complexity of the depth map, wherein the processor uses a deep learning model trained on the specified type of data to perform the semantic segmentation process.

5. The detection method according to claim 4, characterized in that, Step b) further includes having the processor overlay the semantic score onto the depth map to generate overlaid information, and converting the overlaid information to a radar coordinate system used by the radar to generate the virtual key instance.

6. The detection method according to claim 1, characterized in that, Step c) includes the processor performing a 3D feature acquisition procedure on the radar point cloud information and the virtual key instance to obtain multiple 3D features of the radar point cloud information and multiple 3D features of the virtual key instance, and in step e), the virtual instance focusing fusion model performs the region of interest feature fusion processing on the multiple 3D features of the radar point cloud information and the multiple 3D features of the virtual key instance.

7. The detection method according to claim 6, characterized in that, Step c) is followed by step c1): the processor performs a dimensionality reduction process on the multiple radar point cloud information 3D features and the multiple virtual key instance 3D features to generate multiple radar point cloud information 2D features and multiple virtual key instance 2D features, wherein in step d), the virtual instance focusing fusion model is to perform the bird's-eye view feature fusion process on the multiple radar point cloud information 2D features and the multiple virtual key instance 2D features.

8. The detection method according to claim 7, characterized in that, Step d) includes the following steps to achieve the bird's-eye view feature fusion processing: Step d1) The two-dimensional features of the multiple radar point cloud information and the corresponding two-dimensional features of the multiple virtual key instances are stitched together along the channel dimension to generate multiple stitched features, and a channel attention process is performed on the multiple stitched features to generate multiple processed features. Step d2) Perform a spatial attention process on the multiple processed features to generate multiple fused bird's-eye view features; and Step d3) Generate the first bounding box information based on the multiple fused bird's-eye view features.

9. The detection method according to claim 8, characterized in that, Step e) includes the following steps to achieve the feature fusion processing of the region of interest: Step e1) Perform a first self-attention process on the three-dimensional features of the multiple radar point cloud information to obtain a first vector representation; Step e2) Perform a first frequency domain self-attention process on the first vector representation to generate a first query vector, a first key vector, and a first value vector; Step e3) Perform a second self-attention process on the 3D features of the multiple virtual key instances to obtain a second vector representation; Step e4) Perform a second frequency domain self-attention process on the second vector representation to generate a second query vector, a second key vector, and a second value vector; Step e5) Perform a first cross-attention process on the first value vector, the first key vector, and the second query vector to generate a third vector representation, and perform a second cross-attention process on the second value vector, the second key vector, and the first query vector to generate a fourth vector representation; Step e6) Perform a global max pooling process on the third vector representation, the fourth vector representation, and the output of the second frequency domain self-attention processing to produce a first output; Step e7) The first output and the output of the first frequency domain self-attention processing are concatenated to generate a fused three-dimensional region of interest feature; and Step e8) Generate the second bounding box information based on the fused bird's-eye view features and the fused 3D region of interest features.

10. The detection method according to claim 9, characterized in that, The first frequency domain self-attention processing is a frequency domain self-attention processing that focuses on the low frequency domain, and the second frequency domain self-attention processing is a frequency domain self-attention processing that focuses on the full value.

11. An autonomous mobile robot, characterized in that, include: A camera, configured to capture camera images of an environment; A radar is configured to acquire radar point cloud information of the environment; and A processor, electrically connected to the camera and the radar, is configured to execute a plurality of instructions to perform the following actions: Action a) Obtain a depth map based on the camera image, perform a semantic segmentation process on the camera image to obtain a semantic score for each pixel corresponding to a specified type, and superimpose the semantic score onto the depth map to generate a virtual key instance, wherein the virtual key instance records a plurality of dense points in three-dimensional space used to display a target object of the specified type. Action b) Execute a feature acquisition procedure on the virtual key instance and the radar point cloud information to obtain multiple virtual key instance features and multiple radar point cloud information features; Action c) Perform a two-dimensional bird's-eye view feature fusion process on the features of the multiple virtual key instances and the features of the multiple radar point cloud information using a virtual instance focusing fusion model to generate a first bounding box information corresponding to the target object in the radar point cloud information; Action d) The virtual instance focusing fusion model performs a three-dimensional region of interest feature fusion processing on the features of the multiple virtual key instances and the features of the multiple radar point cloud information to generate a second bounding box information, wherein the second bounding box information describes the first bounding box information; and Action e) The virtual instance focuses the fusion model to combine the first bounding box information and the second bounding box information to output a detection result describing the target object in the radar point cloud information.

12. The autonomous mobile robot according to claim 11, characterized in that, The specified type is human, and the target object is human.

13. The autonomous mobile robot according to claim 11, characterized in that, The action a) includes performing a depth completion process on the camera image and the radar point cloud information to generate the depth map, or directly obtaining the depth map from the camera image generated by the camera, which is an RGB-D camera.

14. The autonomous mobile robot according to claim 11, characterized in that, The action a) includes performing the semantic segmentation process on the camera image to generate a target object mask, and filtering the depth map by the target object mask to reduce the complexity of the depth map, wherein the processor is configured to perform the semantic segmentation process using a deep learning model trained on the specified type of data.

15. The autonomous mobile robot according to claim 14, characterized in that, The action a) also includes overlaying the semantic score onto the depth map to generate overlaid information, and converting the overlaid information to a radar coordinate system used by the radar to generate the virtual key instance.

16. The autonomous mobile robot according to claim 11, characterized in that, Action b) includes performing a 3D feature acquisition procedure on the radar point cloud information and the virtual key instance to obtain multiple 3D features of the radar point cloud information and multiple 3D features of the virtual key instance, and in action d), the virtual instance focusing fusion model performs the region of interest feature fusion processing on the multiple 3D features of the radar point cloud information and the multiple 3D features of the virtual key instance.

17. The autonomous mobile robot according to claim 16, characterized in that, The action b) is followed by an action b1): performing a dimensionality reduction process on the multiple radar point cloud information 3D features and the multiple virtual key instance 3D features to generate multiple radar point cloud information 2D features and multiple virtual key instance 2D features, wherein in the action c), the virtual instance focusing fusion model is to perform the bird's-eye view feature fusion process on the multiple radar point cloud information 2D features and the multiple virtual key instance 2D features.

18. The autonomous mobile robot according to claim 17, characterized in that, Action c) includes the following actions to achieve the feature fusion processing of the bird's-eye view: Action c1) The two-dimensional features of the multiple radar point cloud information and the corresponding two-dimensional features of the multiple virtual key instances are stitched together along the channel dimension to generate multiple stitched features, and a channel attention process is performed on the multiple stitched features to generate multiple processed features. Action c2) Performs a spatial attention process on the multiple processed features to generate multiple fused bird's-eye view features; and Action c3) generates the first bounding box information based on the multiple fused bird's-eye view features.

19. The autonomous mobile robot according to claim 18, characterized in that, Action d) includes the following actions to achieve feature fusion processing of the region of interest: Action d1) performs a first self-attention process on the three-dimensional features of the multiple radar point cloud information to obtain a first vector representation; Action d2) performs a first frequency domain self-attention process on the first vector representation to generate a first query vector, a first key vector and a first value vector; Action d3) performs a second self-attention process on the 3D features of the multiple virtual key instances to obtain a second vector representation; Action d4) performs a second frequency domain self-attention process on the second vector representation to generate a second query vector, a second key vector, and a second value vector; Action d5) performs a first cross-attention process on the first value vector, the first key vector, and the second query vector to generate a third vector representation, and performs a second cross-attention process on the second value vector, the second key vector, and the first query vector to generate a fourth vector representation; Action d6) performs a global max pooling process on the third vector representation, the fourth vector representation, and the output of the second frequency domain self-attention processing to produce a first output; Action d7) concatenates the first output and the output of the first frequency domain self-attention processing to generate a fused three-dimensional region of interest feature; and Action d8) generates the second bounding box information based on the fused bird's-eye view features and the fused 3D region of interest features.

20. The autonomous mobile robot according to claim 19, characterized in that, The first frequency domain self-attention processing is a frequency domain self-attention processing that focuses on the low frequency domain, and the second frequency domain self-attention processing is a frequency domain self-attention processing that focuses on the full value.

Images