Perception method and device based on multi-radar and camera fusion, and vehicle

Abstract

The invention belongs to the technical field of unmanned driving, and particularly relates to a sensing method and device based on multi-radar and camera fusion and a vehicle, and the method comprises the steps: obtaining the original point cloud data of a plurality of radar sensors and the image data of a camera, carrying out the space-time alignment of the original point cloud data and the image data, and obtaining the image data of the camera; generating a depth map of the image by using the data after space-time alignment; the depth map is input into a wavelet DSIRes2Net module, multi-scale information is extracted, and output features are formed; the output features are sent to a feature fusion module for feature integration, and the three branches are spliced in the channel dimension when the model is output; the spliced features are sent to three task decoders, obstacle instance segmentation, drivable region segmentation and depth prediction are carried out through different encoders, and loss between different tasks is balanced in a self-adaptive manner, so that image results are obtained to present different regions of an obstacle module and a drivable region boundary. The method is used for solving the problem of poor multi-sensor fusion perception.

Description

A perception method, device, and vehicle based on multi-radar and camera fusion Technical Field

[0001] This invention belongs to the field of autonomous driving technology, specifically relating to a perception method, device, and vehicle based on the fusion of multiple radars and cameras. Background Technology

[0002] With the continuous development of autonomous driving technology, it has been widely applied in various industries, such as logistics, industrial parks, and airports. The application of autonomous driving technology in recent years demonstrates its ability to improve road traffic safety, its effectiveness in promoting low-carbon development, and its role in aligning with national low-carbon policies. The coal mining industry, in order to keep pace with low-carbon policies, has gradually introduced various low-carbon technologies. Mining area transportation roads are relatively simple, and the driving trajectories of mining trucks are relatively straightforward, making the implementation of autonomous driving for mining trucks relatively easy in terms of application scenarios. In 2019, the "Several Opinions on Promoting the Intelligent and Rapid Development of Coal Mining" was issued, which highlighted "open-pit unmanned truck systems." This policy has prompted enterprises to vigorously develop autonomous driving technology for mining trucks.

[0003] In the unmanned driving technology of mining trucks, environmental perception is the foundation of driving. By acquiring information about the road environment around the vehicle through various sensors, a solid foundation is laid for subsequent extensions of technologies such as obstacle avoidance, path planning, and decision control.

[0004] Currently, mining trucks generate significant dust during operation. LiDAR detects dust particles in the air, generating echoes that trigger obstacle collision alarms. Furthermore, the bumpy and difficult terrain of mining roads, coupled with the potential for false alarms from millimeter-wave radar due to frequency shifts, makes it difficult for a single sensor to provide the wide field of view required for mining trucks. Therefore, multiple radars and cameras are used to construct a comprehensive road environment perception system. Consequently, a multi-task perception method for the road environment of unmanned mining trucks based on the fusion of multi-source information from radar and cameras is currently lacking. Summary of the Invention

[0005] To address the shortcomings of existing technologies, a perception method, device, and vehicle based on the fusion of multiple radars and cameras are provided to solve the problem of poor perception through multi-sensor fusion.

[0006] The technical solution of this invention to solve the above-mentioned technical problems is as follows: A perception method based on multi-radar and camera fusion, comprising: S1, acquiring raw point cloud data from multiple radar sensors and image data from cameras, performing spatiotemporal alignment on the raw point cloud data and image data, and generating a depth map of the image from the spatiotemporally aligned data; S2, inputting the depth map into a wavelet DSIRes2Net module to extract multi-scale information and form output features; S3, sending the output features into a feature fusion module for feature integration to obtain four feature maps of different sizes, one of which retains the original channel dimension as a base mask, and the other three feature maps are designed with bounding boxes, categories, and masks respectively, and then the three branches are stitched together in the channel dimension during model output; S4, sending the stitched features into three task decoders, performing obstacle instance segmentation, drivable area segmentation, and depth prediction respectively through different encoders, and using Gradnorm adaptively balancing the loss between different tasks to ensure that the multi-task branches can converge normally, so that the image result presents different regions of obstacle modules and drivable area boundaries.

[0007] In one embodiment, step S1 specifically includes: S101, periodically acquiring and updating the latest data from multiple radar sensors and cameras at the same time point, using a timestamp synchronization algorithm to ensure that the point cloud data and image data are time-consistent; S102, converting the measurement coordinates of the radar and vision system into vehicle coordinates in space through rotation and translation matrices, thereby achieving time and space alignment, and generating a depth map of the image from the aligned data.

[0008] In one embodiment, step S101 specifically includes: initializing the data and timestamps of the radar and camera, acquiring the data and timestamps of the lidar, millimeter-wave radar and camera respectively, and when the timestamps of the three are aligned, keeping each data item; when the timestamps are not aligned, waiting for the timestamps of other data to be aligned.

[0009] In one embodiment, step S102 specifically includes: initializing the matching radius; based on traversing millimeter-wave radar targets, traversing whether lidar targets and camera targets are within the matching radius; if within the matching radius, saving the matching result in a list, ultimately obtaining consistent time data; establishing the relationship between the measurement coordinates and the vehicle coordinates as follows: The aligned 3D coordinates are converted to 2D coordinates, and the Z-coordinate is extracted from the 3D coordinates as the depth value to form a depth map; among them, the radar and camera measurement coordinates are... The vehicle coordinates are Let the rotation matrix be R and the translation matrix be T. The rotation matrix R is expressed as: The translation matrix T is expressed as: The specific values ​​in the rotation and translation matrices are obtained by extrinsic parameter calibration.

[0010] In one embodiment, step S2 includes: S201, inputting the depth map as a first feature into the input... The standard convolution expands the features to n times the original number of channels, and the size is... The original features, after feature upscaling, yield a size of The second feature; S202, divide the upgraded second feature into n feature subsets evenly, and divide the feature subsets from bottom to top into The first feature subset Output directly without convolution operation, other feature subsets After adding it to the previous feature subset in turn, proceed with... Depthwise separable convolution and denoted as The corresponding output feature subset is obtained. Correspondingly, the output feature subsets from bottom to top are respectively This subset of output features is used as a third feature to extract multi-scale information: .

[0011] In one embodiment, step S2 further includes: S203, dividing each output feature subset Introducing a CWT convolution module, using a set of low-pass filters Calculate the average value of neighboring pixels to extract low-frequency information, using a set of high-pass filters. Calculate interpolation between adjacent pixels to extract high-frequency information: ; Perform a 2D Haar wavelet transform, combine the results in two dimensions to form four filters, and then perform depthwise convolution with a stride of 2. The output is... : Then, through inverse wavelet transform (IWT) transpose convolution, we obtain: Then, wavelet transform (WT) is applied to the low-frequency and high-frequency information obtained from convolution, followed by depthwise convolution with small kernels on different frequency maps. Finally, IWT is used to output the global feature Y. W is a The weight tensor of the depthwise convolution kernel outputs a subset of global features, which together form the global feature set as the fourth feature; then... The standard convolution reduces the dimensionality of the fourth feature, restoring it to its original number of channels; by adding skip connections between the input and output, the first and fourth features are combined to obtain the fifth feature, which is then passed to the output feature.

[0012] In one embodiment, step S4 includes three tasks: obstacle instance segmentation, drivable region segmentation, and depth prediction, each corresponding to a different decoder, which includes: a bounding box loss function. Classification loss function Segmentation loss function and depth prediction loss function Composition; among which, the bounding box loss function CIoU Loss is used as the loss function; classification loss function Focal Loss was used as the loss function; standard cross-entropy loss function was used for obstacle instance segmentation and drivable region segmentation. As a segmentation loss function; depth prediction loss function L1 Loss is used to calculate the difference between the predicted and actual values; gradient loss is used to calculate the weights of each loss function, and the total loss is calculated by weighting: Gradnorm adaptively balances the losses between different tasks. After calculation, the total loss is minimized, indicating convergence. Thus, the three tasks eventually converge to obtain vehicle and pedestrian bounding boxes, segment obstacle edges, perform semantic segmentation of feasible regions, and predict environmental boundaries. The three tasks of obstacle instance segmentation, drivable region segmentation, and depth prediction are integrated into a real-world scenario, presenting different regions of obstacle modules and drivable region boundaries in a single image.

[0013] This embodiment also discloses an apparatus, comprising: an acquisition module for acquiring raw point cloud data from multiple radar sensors and image data from a camera, performing spatiotemporal alignment of the raw point cloud data and image data, and generating a depth map of the image from the spatiotemporally aligned data; an extraction module for inputting the depth map into a wavelet DSIRes2Net module to extract multi-scale information and form output features; a fusion module for feeding the output features into a feature fusion module to integrate features to obtain four feature maps of different sizes, one of which retains the original channel dimension as a base mask, and the other three feature maps are designed with bounding boxes, categories, and masks respectively, and then the three branches are stitched together in the channel dimension during model output; and an output module for feeding the stitched features into three task decoders, which respectively perform obstacle instance segmentation, drivable area segmentation, and depth prediction through different encoders, and use Gradnorm adaptive balancing of the loss between different tasks to ensure that the multi-task branches can converge normally, so that the image result presents different regions of obstacle modules and drivable area boundaries.

[0014] This embodiment also discloses a vehicle, which includes: a processor, a memory, and a computer program stored in the memory and capable of running on the processor. When the computer program is executed by the processor, it implements the perception method based on multi-radar and camera fusion as described above.

[0015] The one or more technical solutions provided in this application embodiment have at least the following technical effects or advantages: 1. Feature-level fusion of point cloud information from multiple radars and image information from cameras eliminates the one-sidedness of single information.

[0016] 2. The DSIRes2Net residual module is used to extract multi-scale environmental information. Adding wavelet convolution can capture richer texture and edge information, and more accurately obtain the surrounding environment information of the unmanned mining truck.

[0017] 3. By employing multiple decoders to converge different tasks, multi-task recognition is achieved, reducing computation speed and improving the environmental perception efficiency of unmanned mining trucks. Attached Figure Description

[0018] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0019] Figure 1 is a schematic diagram of the process of the present invention.

[0020] Figure 2 is a schematic diagram of the wavelet DSIRes2Net module in this invention.

[0021] Figure 3 is a schematic diagram of the CWT module in this invention. Detailed Implementation

[0022] The embodiments of the technical solution of the present invention will now be described in detail with reference to the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and are therefore merely examples and should not be used to limit the scope of protection of the present invention. It should be noted that, unless otherwise stated, the technical or scientific terms used in this application should have the ordinary meaning understood by those skilled in the art to which this invention pertains.

[0023] In the description of this application, it should be understood that the terms "upper", "lower", "front", "rear", "left", "right", "top", "bottom", "inner", "outer", etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limiting the present invention.

[0024] Furthermore, the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly defined.

[0025] In this application, unless otherwise expressly specified and limited, the terms "installation," "connection," "joining," and "fixing," etc., should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0026] Example 1, as shown in Figure 1, provides a perception method based on multi-radar and camera fusion, which includes the following steps: Step S1, acquiring raw point cloud data from multiple radar sensors and image data from cameras, performing spatiotemporal alignment on the raw point cloud data and image data, and generating a depth map of the image from the spatiotemporally aligned data. Since different sensors have different sampling frequencies, transmission delays, and internal processing times, it is difficult to guarantee the simultaneity of the data. Therefore, the data acquired by multiple radars and cameras are time-aligned.

[0027] Specifically, the process includes step S101: periodically acquiring and updating the latest data from multiple radar sensors and cameras at the same time point, using a timestamp synchronization algorithm to ensure that the point cloud data and image data are time-consistent; step S102: converting the measurement coordinates of the radar and vision system into vehicle coordinates in space through rotation and translation matrices, thereby achieving spatiotemporally aligned data, and generating a depth map of the image from the aligned data.

[0028] In this embodiment, the sensor includes multiple radars and cameras.

[0029] Specifically, the multiple radars in step S1 include at least two millimeter-wave radars and one lidar. The multiple radars are fixed by corresponding fixed brackets and fixed at certain positions and angles on the left and right sides of the cab and the center grille of the mining truck. The camera is installed at the center grille of the mining truck.

[0030] Specifically, step S101 includes: initializing the data and timestamps of the radar and camera, acquiring the data and timestamps of the lidar, millimeter-wave radar and camera respectively, and keeping the data when the timestamps of the three are aligned, and waiting for the timestamps of other data to be aligned when the timestamps are not aligned. For example, if the radar timestamp is earlier than the camera timestamp, then wait for the camera data; if the radar timestamp is later than the camera timestamp, then wait for the radar data.

[0031] Step S102 specifically includes: obtaining the transformation matrix through extrinsic parameter calibration, initializing the matching radius, and, based on traversing the millimeter-wave radar target, traversing the lidar target and camera target to see if they are within the matching radius. If they are within the matching radius, the matching result is saved in a list, and finally, the time data is consistent; the relationship between the measurement coordinates and the vehicle coordinates is established as follows: The aligned 3D coordinates (x, y, z) are converted into 2D image coordinates (x, y), and the Z coordinate is extracted as the depth value; where the radar and camera measurement coordinates are... The vehicle coordinates are Let the rotation matrix be R and the translation matrix be T. The rotation matrix R is expressed as: The translation matrix T is expressed as: The specific values ​​in the rotation and translation matrices are obtained by extrinsic parameter calibration.

[0032] The purpose of step S2 is to extract features: input the depth map into the wavelet DSIRes2Net module to extract multi-scale information and form output features; specifically, as shown in Figure 2, step S2 includes: S201, inputting the depth map as the first feature into the input... The standard convolution expands the features to n times the original number of channels, and the size is... The original features, after feature upscaling, yield a size of The second feature; S202, divide the upgraded second feature into n feature subsets evenly, and divide the feature subsets from bottom to top into Each feature subset has the same feature space.

[0033] Among them, the first feature subset Output directly without convolution operation, other feature subsets After adding it to the previous feature subset in turn, proceed with... Depthwise separable convolution and denoted as The corresponding output feature subset is obtained. Correspondingly, the output feature subsets from bottom to top are respectively This subset of output features is used as a third feature to extract multi-scale information: .

[0034] Furthermore, step S2 also includes: S203, since semantic segmentation and depth prediction tasks place greater emphasis on detailed and edge information, in order to capture richer texture and edge information, each output feature subset... A CWT convolution module is introduced, as shown in Figure 3, which uses a set of low-pass filters. Calculate the average value of neighboring pixels to extract low-frequency information, using a set of high-pass filters. Calculate interpolation between adjacent pixels to extract high-frequency information: ; Perform a 2D Haar wavelet transform, combine the results in two dimensions to form four filters, and then perform depthwise convolution with a stride of 2. The output is... : Since the filter kernel is an orthonormal basis, it is then obtained through inverse wavelet transform (IWT) transpose convolution: Then, wavelet transform (WT) is applied to the low-frequency and high-frequency information obtained from convolution, followed by depthwise convolution with small kernels on different frequency maps. Finally, IWT is used to output the global feature Y. W is a The weight tensor of the depthwise convolution kernel outputs a subset of global features, which together form the global feature set as the fourth feature; then... The standard convolution reduces the dimensionality of the fourth feature, restoring it to the original number of channels; by adding a skip connection between the input and output, that is, between the first and fifth features, the fifth feature is obtained by combining the first and fourth features and then passed to the output feature.

[0035] The purpose of step S3 is to fuse features: the output features are fed into the feature fusion module to integrate features and obtain four feature maps of different sizes. One feature map retains the original channel dimension as the base mask. The size of each mask is 125×125. The other three feature maps are designed with three branches: bounding box (Box), class (Cls), and mask (Mask). Then, when the model is output, the three branches are spliced ​​together in the channel dimension.

[0036] The purpose of step S4 is to input the obtained stitched features into different decoders for the three tasks and adaptively balance the losses of the three tasks: obstacle instance segmentation, drivable area segmentation and depth prediction are performed by different encoders respectively, and Gradnorm is used to adaptively balance the losses between different tasks so that the multi-task branches can converge normally, so that the image results show different regions of obstacle modules and drivable area boundaries.

[0037] Specifically, obstacle instance segmentation, drivable region segmentation, and depth prediction each correspond to different decoders, which include: bounding box loss function. Classification loss function Segmentation loss function and depth prediction loss function Composition; among which, the bounding box loss function CIoU Loss is used as the loss function; classification loss function Focal Loss was used as the loss function; segmentation loss function The standard cross-entropy loss function is used; the standard cross-entropy loss function is used for obstacle instance segmentation and drivable region segmentation. As a segmentation loss function; depth prediction loss function The difference between the predicted value and the actual value is calculated using L1 Loss.

[0038] Since directly adding the above loss functions is unlikely to converge, gradient loss is used to calculate the weights of each loss function, thereby weighting the total loss. Specifically, this is expressed as follows: Gradnorm adaptively balances the losses between different tasks. After calculation, the total loss is minimized, indicating convergence. Thus, the three tasks eventually converge to obtain vehicle and pedestrian bounding boxes, segment obstacle edges, perform semantic segmentation of feasible regions, and predict environmental boundaries. The three tasks of obstacle instance segmentation, drivable region segmentation, and depth prediction are integrated into a real-world scenario, presenting different regions of obstacle modules and drivable region boundaries in a single image.

[0039] Example 2 This example also discloses an apparatus, comprising: an acquisition module for acquiring raw point cloud data from multiple radar sensors and image data from a camera, performing spatiotemporal alignment of the raw point cloud data and image data, and generating a depth map of the image from the spatiotemporally aligned data; an extraction module for inputting the depth map into a wavelet DSIRes2Net module to extract multi-scale information and form output features; a fusion module for feeding the output features into a feature fusion module to integrate features to obtain four feature maps of different sizes, one of which retains the original channel dimension as a base mask, and the other three feature maps are designed with bounding boxes, categories, and masks respectively, and then the three branches are stitched together in the channel dimension during model output; and an output module for feeding the stitched features into three task decoders, which respectively perform obstacle instance segmentation, drivable area segmentation, and depth prediction through different encoders, and use Gradnorm adaptive balancing of the loss between different tasks to ensure that the multi-task branches can converge normally, so that the image result presents different regions of obstacle modules and drivable area boundaries.

[0040] Example 3 This example also discloses a vehicle, which includes: a processor, a memory, and a computer program stored in the memory and capable of running on the processor. When the computer program is executed by the processor, it implements the perception method based on multi-radar and camera fusion as described above.

[0041] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A perception method based on the fusion of multiple radars and cameras, characterized in that, include: S1. Acquire raw point cloud data from multiple radar sensors and image data from cameras. Spatiotemporally align the raw point cloud data and image data to generate a depth map of the image. S2. Input the depth map into the wavelet DSIRes2Net module to extract multi-scale information and form output features. S3. Feed the output features into the feature fusion module for feature integration to obtain four feature maps of different sizes. One feature map retains the original channel dimension as the base mask, while the other three feature maps are designed with bounding boxes, categories, and masks respectively. These three branches are then concatenated along the channel dimension during model output. S4. Feed the concatenated features into three task decoders. Obstacle instance segmentation, drivable area segmentation, and depth prediction are performed using different encoders. Gradnorm adaptively balances the losses between different tasks to ensure normal convergence of the multi-task branches, resulting in an image that presents different regions of obstacle modules and drivable area boundaries.

2. The perception method based on multi-radar and camera fusion according to claim 1, characterized in that, Step S1 specifically involves: S101, periodically acquiring and updating the latest data from multiple radar sensors and cameras at the same time points, using a timestamp synchronization algorithm to ensure that the point cloud data and image data are time-consistent; S102, spatially converting the measurement coordinates of the radar and vision systems into vehicle coordinates through rotation and translation matrices, thereby achieving time and space alignment, and generating a depth map of the image from the aligned data.

3. The perception method based on multi-radar and camera fusion according to claim 1, characterized in that, Step S101 specifically includes: initializing the data and timestamps of the radar and camera, acquiring the data and timestamps of the lidar, millimeter-wave radar and camera respectively, and keeping the data when the timestamps of the three are aligned, and waiting for the timestamps of other data to be aligned when the timestamps are not aligned.

4. The perception method based on multi-radar and camera fusion according to claim 1, characterized in that, Step S102 specifically includes: initializing the matching radius; based on traversing millimeter-wave radar targets, traversing lidar targets and camera targets to see if they are within the matching radius; if they are within the matching radius, saving the matching results in a list, ultimately obtaining consistent time data; establishing the relationship between the measurement coordinates and the vehicle coordinates as follows: The aligned 3D coordinates are converted to 2D coordinates, and the Z-coordinate is extracted from the 3D coordinates as the depth value to form a depth map; among them, the radar and camera measurement coordinates are... The vehicle coordinates are Let the rotation matrix be R and the translation matrix be T. The rotation matrix R is expressed as: The translation matrix T is expressed as: The specific values ​​in the rotation and translation matrices are obtained by extrinsic parameter calibration.

5. The perception method based on multi-radar and camera fusion according to claim 1, characterized in that, Step S2 includes: S201, inputting the depth map as the first feature into the input... The standard convolution expands the features to n times the original number of channels, and the size is... The original features, after feature upscaling, yield a size of The second feature; S202, divide the upgraded second feature into n feature subsets evenly, and divide the feature subsets from bottom to top into The first feature subset Output directly without convolution operation, other feature subsets After adding it to the previous feature subset in turn, proceed with... Depthwise separable convolution and denoted as The corresponding output feature subset is obtained. Correspondingly, the output feature subsets from bottom to top are respectively This subset of output features is used as a third feature to extract multi-scale information: 。 6. The perception method based on multi-radar and camera fusion according to claim 5, characterized in that, Step S2 further includes: S203, dividing each output feature subset Introducing a CWT convolution module, using a set of low-pass filters Calculate the average value of neighboring pixels to extract low-frequency information, using a set of high-pass filters. Calculate interpolation between adjacent pixels to extract high-frequency information: ； Perform a 2D Haar wavelet transform, combine the results in two dimensions to form four filters, and then perform depthwise convolution with a stride of 2. The output is... ： Then, through inverse wavelet transform (IWT) transpose convolution, we obtain: Then, wavelet transform (WT) is applied to the low-frequency and high-frequency information obtained from convolution, followed by depthwise convolution with small kernels on different frequency maps. Finally, IWT is used to output the global feature Y. W is a The weight tensor of the depthwise convolution kernel outputs a subset of global features, which together form the global feature set as the fourth feature; then... The standard convolution reduces the dimensionality of the fourth feature, restoring it to its original number of channels; by adding skip connections between the input and output, the first and fourth features are combined to obtain the fifth feature, which is then passed to the output feature.

7. The perception method based on multi-radar and camera fusion according to claim 1, characterized in that, Step S4 includes three tasks: obstacle instance segmentation, drivable region segmentation, and depth prediction, each corresponding to a different decoder, which includes: bounding box loss function. Classification loss function Segmentation loss function and depth prediction loss function Composition; among which, the bounding box loss function CIoU Loss is used as the loss function; classification loss function Focal Loss was used as the loss function; standard cross-entropy loss function was used for obstacle instance segmentation and drivable region segmentation. As a segmentation loss function; depth prediction loss function L1 Loss is used to calculate the difference between the predicted and actual values; gradient loss is used to calculate the weights of each loss function, and the total loss is calculated by weighting: Gradnorm adaptively balances the losses between different tasks. After calculation, the total loss is minimized, indicating convergence. Thus, the three tasks eventually converge to obtain vehicle and pedestrian bounding boxes, segment obstacle edges, perform semantic segmentation of feasible regions, and predict environmental boundaries. The three tasks of obstacle instance segmentation, drivable region segmentation, and depth prediction are integrated into a real-world scenario, presenting different regions of obstacle modules and drivable region boundaries in a single image.

8. An apparatus, characterized in that, include: The acquisition module is used to acquire raw point cloud data from multiple radar sensors and image data from cameras, perform spatiotemporal alignment on the raw point cloud data and image data, and generate a depth map of the image from the spatiotemporally aligned data. The extraction module is used to input the depth map into the wavelet DSIRes2Net module, extract multi-scale information and form output features; The fusion module feeds the output features into the feature fusion module to integrate them into four feature maps of different sizes. One feature map retains the original channel dimension as the base mask, while the other three feature maps are designed with bounding boxes, categories, and masks, respectively. These three branches are then concatenated along the channel dimension during model output. The output module feeds the concatenated features into three task decoders, which perform obstacle instance segmentation, drivable region segmentation, and depth prediction through different encoders. Gradnorm adaptively balances the losses between different tasks to ensure that the multi-task branches converge normally, resulting in an image that presents different regions of obstacle modules and drivable region boundaries.

9. A vehicle, characterized in that, include: A processor, a memory, and a computer program stored in the memory and capable of running on the processor, wherein the computer program, when executed by the processor, implements the perception method based on multi-radar and camera fusion as described in any one of claims 1-7.

Images