Multi-task sensing method and device for automatic driving

Abstract

The invention discloses a multi-task sensing method and device for automatic driving. Acquiring image data and point cloud data at a plurality of continuous moments; respectively processing the image data and the point cloud data to obtain an initial first aerial view feature and an initial second aerial view feature; projecting the initial first aerial view features at different moments into the initial first aerial view feature at the target moment to obtain a target first aerial view feature at the target moment, and projecting the initial second aerial view features at different moments into the initial second aerial view feature at the target moment to obtain a target second aerial view feature at the target moment; splicing the first aerial view features and the second aerial view features at the corresponding moments to obtain a feature set containing a plurality of spliced features; and respectively inputting the feature set into a plurality of task heads to obtain corresponding task sensing results. And the accuracy of environment perception is effectively improved.

Description

Technical Field

[0001] Embodiments of the present invention relate to the field of autonomous driving technology, and in particular, to a multi-task perception method and device for autonomous driving. Background Art

[0002] Autonomous driving is leading the automotive industry into a new era. Autonomous driving vehicles mainly include environmental perception systems, path planning systems, and decision-making control systems. It gives vehicles vision through advanced environmental perception systems and uses artificial intelligence algorithms to understand the environment, predict behavior, and plan driving, thereby ensuring the safety and efficient operation of the vehicle.

[0003] Environmental perception technology is a key technology in autonomous driving, but the existing autonomous driving perception methods have the problem of low environmental perception accuracy. Summary of the invention

[0004] The present invention provides a multi-task perception method and device for autonomous driving to solve the problem of low environmental perception accuracy in existing autonomous driving perception methods.

[0005] According to one aspect of the present invention, a multi-task perception method for autonomous driving is provided, characterized by comprising:

[0006] Acquire perception data at a plurality of consecutive moments, wherein the perception data includes image data acquired by a vehicle-mounted camera and point cloud data acquired by a vehicle-mounted radar;

[0007] Processing the image data to obtain an initial first bird's-eye view feature, and processing the point cloud data to obtain an initial second bird's-eye view feature;

[0008] Projecting the initial first bird's-eye view features at different times into the initial first bird's-eye view features at the target time to obtain the target first bird's-eye view features at the target time, and projecting the initial second bird's-eye view features at different times into the initial second bird's-eye view features at the target time to obtain the target second bird's-eye view features at the target time;

[0009] Splicing the first bird's-eye view feature and the second bird's-eye view feature at the corresponding moment to obtain a feature set including a plurality of spliced ​​features, wherein the first bird's-eye view feature includes an initial first bird's-eye view feature and a target first bird's-eye view feature, and the second bird's-eye view feature includes an initial second bird's-eye view feature and a target second bird's-eye view feature;

[0010] The feature sets are respectively input into multiple task heads to obtain corresponding task perception results.

[0011] According to another aspect of the present invention, a multi-task perception device for autonomous driving is provided, characterized in that it includes:

[0012] A data acquisition module, used to acquire perception data at multiple consecutive moments, wherein the perception data includes image data acquired by a vehicle-mounted camera and point cloud data acquired by a vehicle-mounted radar;

[0013] A data processing module, used to process the image data to obtain an initial first bird's-eye view feature, and to process the point cloud data to obtain an initial second bird's-eye view feature;

[0014] a target feature determination module, configured to project the initial first bird's-eye view features at different times into the initial first bird's-eye view features at the target time to obtain the target first bird's-eye view features at the target time, and to project the initial second bird's-eye view features at different times into the initial second bird's-eye view features at the target time to obtain the target second bird's-eye view features at the target time;

[0015] A feature stitching module, configured to stitch the first bird's-eye view feature and the second bird's-eye view feature at the corresponding moment to obtain a feature set including a plurality of stitched features, wherein the first bird's-eye view feature includes an initial first bird's-eye view feature and a target first bird's-eye view feature, and the second bird's-eye view feature includes an initial second bird's-eye view feature and a target second bird's-eye view feature;

[0016] The result determination module is used to input the feature set into multiple task heads respectively to obtain corresponding task perception results.

[0017] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0018] at least one processor; and

[0019] a memory communicatively connected to the at least one processor; wherein,

[0020] The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the multi-task perception method for autonomous driving described in any embodiment of the present invention.

[0021] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the multi-task perception method for autonomous driving described in any embodiment of the present invention when executed.

[0022] The technical solution provided by the embodiment of the present invention obtains perception data at multiple consecutive moments, the perception data includes image data and point cloud data, and processes these perception data to obtain initial first bird's-eye view features and initial second bird's-eye view features, and then projects the initial first bird's-eye view features at different moments to the target moment to obtain the target first bird's-eye view features at the target moment, so that the target first bird's-eye view features include the temporal correlation information of the image, and projects the initial second bird's-eye view features at different moments to the target moment to obtain the target second bird's-eye view features at the target moment, so that the target second bird's-eye view features include the temporal correlation information of the point cloud, and then splices the first bird's-eye view features and the second bird's-eye view features at the corresponding moments, thereby realizing the fusion of two different modal data, image and point cloud, to obtain a feature set for multiple consecutive moments, and then performing multi-task perception, effectively improving the accuracy of environmental perception.

[0023] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0025] Figure 1 is a flowchart of a multi-task perception method for autonomous driving provided by Embodiment 1 of the present invention;

[0026] Figure 2 is a flow chart of a multi-task perception method for autonomous driving provided by Embodiment 2 of the present invention;

[0027] Figure 3 is a processing diagram of a multi-task perception method for autonomous driving provided by an embodiment of the present invention;

[0028] Figure 4 is a schematic diagram of the structure of a multi-task perception device for autonomous driving provided by Embodiment 3 of the present invention;

[0029] Figure 5 It is a structural schematic diagram of an electronic device provided in Embodiment 4 of the present invention. DETAILED DESCRIPTION

[0030] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0031] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0032] Embodiment 1

[0033] Figure 1 This is a flowchart of a multi-task perception method for autonomous driving provided by the first embodiment of the present invention. This embodiment is applicable to the case of multi-task perception for autonomous driving. The method can be executed by a multi-task perception device for autonomous driving. The multi-task perception device for autonomous driving can be implemented in the form of hardware and / or software. The multi-task perception device for autonomous driving can be configured in an electronic device. Figure 1 As shown, the method includes:

[0034] S110, acquiring perception data at a plurality of consecutive moments, wherein the perception data includes image data acquired by a vehicle-mounted camera and point cloud data acquired by a vehicle-mounted radar.

[0035] In this embodiment, a plurality of consecutive moments can be understood as a plurality of consecutive sampling moments, wherein the sampling moment can be understood as a time point at which data is collected by a data acquisition device, and a plurality of consecutive moments can be understood as a series of time points at which the data acquisition device continuously performs sampling operations at preset time intervals within a period of time. Perception data can be understood as original information about the surrounding environment of the autonomous driving vehicle collected by the data acquisition device. In the present invention, perception data includes image data acquired by a vehicle-mounted camera and point cloud data acquired by a vehicle-mounted radar. Among them, the radar may include a 3D millimeter radar, a 4D millimeter radar, and a laser radar, etc., which are not limited in this embodiment.

[0036] Optionally, after acquiring the perception data, the perception data may be preprocessed, such as data cleaning, to obtain perception data with higher quality.

[0037] S120: Process the image data to obtain an initial first bird's-eye view feature, and process the point cloud data to obtain an initial second bird's-eye view feature.

[0038] In this embodiment, the Bird's-Eye-View (BEV) feature is a data representation that provides a bird's-eye view from a high altitude to describe the characteristics of the vehicle's surroundings. The initial first bird's-eye view feature can be represented as an image BEV feature, and the initial second bird's-eye view feature can be represented as a point cloud BEV feature.

[0039] Specifically, in order to achieve the fusion of image data and point cloud data, it is necessary to represent the image data and point cloud data using the same data representation form, that is, different deep learning models can be used to extract features from image data and point cloud data respectively, and then the extracted features are mapped to the BEV plane to obtain the initial first bird's-eye view features and the initial second bird's-eye view features.

[0040] S130, projecting the initial first bird's-eye view features at different times into the initial first bird's-eye view features at the target time to obtain the target first bird's-eye view features at the target time, and projecting the initial second bird's-eye view features at different times into the initial second bird's-eye view features at the target time to obtain the target second bird's-eye view features at the target time.

[0041] In this embodiment, the target moment may be the current moment among the plurality of consecutive moments. Other moments among the plurality of consecutive moments may be used as historical moments.

[0042] Specifically, by using the first preset projection technology, the initial first bird's-eye view features corresponding to each historical moment are projected into the initial first bird's-eye view features of the target moment, and the target first bird's-eye view features of the target moment can be obtained. At this time, the target first bird's-eye view features contain the temporal correlation information of the image; by using the second preset projection technology, the initial second bird's-eye view features of each historical moment are projected into the initial second bird's-eye view features of the target moment, and the target second bird's-eye view features of the target moment can be obtained. At this time, the target second bird's-eye view features contain the time correlation information of the point cloud. In this embodiment, the first preset projection technology and the second preset projection technology are not limited. For example, the first preset projection technology may include optical flow method, feature point tracking, or time series model in deep learning, etc.; the second preset projection technology may include a motion model that considers the object.

[0043] Exemplarily, image data and point cloud data at n moments are obtained, where n moments are continuous. After processing the image data, the initial first bird's-eye view features at n moments are obtained; after processing the point cloud data, the initial second bird's-eye view features at n moments are obtained. Furthermore, the initial first bird's-eye view features at n-1 moments are all projected onto the initial first bird's-eye view features at the nth moment, and the target first bird's-eye view features can be obtained; the initial second bird's-eye view features at n-1 moments are all projected onto the initial second bird's-eye view features at the nth moment, and the target second bird's-eye view features can be obtained. Wherein, the nth moment is the target moment.

[0044] S140. Concatenate the first bird's-eye view feature and the second bird's-eye view feature at the corresponding moment to obtain a feature set including a plurality of concatenated features, wherein the first bird's-eye view feature includes an initial first bird's-eye view feature and a target first bird's-eye view feature, and the second bird's-eye view feature includes an initial second bird's-eye view feature and a target second bird's-eye view feature.

[0045] In this embodiment, the splicing feature is a multimodal BEV feature.

[0046] Specifically, considering that the ability to characterize the surrounding environment using a single modal data is limited, different modal data are fused to improve the ability to characterize the surrounding environment, that is: for each historical moment, the initial first bird's-eye view feature and the initial second bird's-eye view feature corresponding to the current historical moment are spliced ​​to obtain the initial multimodal BEV feature of the current historical moment. For example, the initial first bird's-eye view feature and the initial second bird's-eye view feature at the n-1 moment are spliced ​​to obtain the initial multimodal BEV feature at the n-1 moment; the target first bird's-eye view feature and the target second bird's-eye view feature corresponding to the target moment are spliced ​​to obtain the target multimodal BEV feature, and then the feature set can be determined by multiple initial multimodal BEV features and target multimodal BEV features, so that while considering all features, the features of each moment are also considered, so that the environment of the current vehicle can be more comprehensively characterized. It is worth noting that in this embodiment, the splicing method is not limited. For example, channel splicing can be used to directly stack the feature map in the channel dimension, or the attention mechanism can be used to achieve it by dynamically learning the weight distribution between features.

[0047] S150. Input the feature set into multiple task heads respectively to obtain corresponding task perception results.

[0048] In this embodiment, the task head is used to convert the obtained BEV features into the output of a specific task, for example, the detection result of the target detection task. The multiple task heads may include an image classification task head, a key point detection task head, a posture estimation task head, a target detection task head, a lane line segmentation task head, and a drivable area segmentation task head.

[0049] Specifically, by inputting the feature sets into the task heads customized for each task, the perception results of each task can be obtained in parallel, thereby achieving a comprehensive understanding and decision-making of the environment.

[0050] In some current technologies, a separate task model is used for each individual task, and in the process of training these models, the complexity of the system is increased and the operating efficiency of the system is reduced. In the embodiment of the present invention, a method of jointly training multiple related tasks is adopted, so that the model can learn more general representations, which helps to improve the generalization ability of the model. Processing multiple tasks at the same time effectively improves the generalization ability of the model, reduces task overhead, and effectively saves computing resources; the method is flexible and has high robustness and stability in practical applications.

[0051] The technical solution provided in the first embodiment of the present invention obtains perception data at multiple consecutive moments, the perception data includes image data and point cloud data, and processes these perception data to obtain initial first bird's-eye view features and initial second bird's-eye view features, and then projects the initial first bird's-eye view features at different moments to the target moment to obtain the target first bird's-eye view features at the target moment, so that the target first bird's-eye view features include the temporal correlation information of the image, and projects the initial second bird's-eye view features at different moments to the target moment to obtain the target second bird's-eye view features at the target moment, so that the target second bird's-eye view features include the temporal correlation information of the point cloud, and then splices the first bird's-eye view features and the second bird's-eye view features at the corresponding moments, thereby realizing the fusion of two different modal data, image and point cloud, to obtain a feature set including multiple spliced ​​features, and then performs multi-task perception, effectively improving the accuracy of environmental perception.

[0052] In some embodiments, the multiple task heads include a target detection task head, a lane segmentation task head, and a drivable area segmentation task head; the feature set is input into the multiple task heads respectively to obtain the corresponding task perception results, including: inputting the feature set into the target detection task head and outputting the target detection result; inputting the feature set into the lane segmentation task head and outputting the lane segmentation result; inputting the feature set into the drivable area segmentation task head and outputting the drivable area segmentation result. Through the above technical solution, the target detection task, lane segmentation task, and drivable area segmentation task are integrated to achieve a more comprehensive perception of the vehicle environment, laying the foundation for ensuring the safety of autonomous driving.

[0053] In this embodiment, the target detection task head is used to identify and locate one or more target objects in the image and determine their positions and categories. The lane line segmentation task head is used to identify and segment the lane lines on the road, providing the autonomous driving vehicle with accurate information required for navigation and control. The drivable area segmentation task head is used to identify and segment the drivable area in the current driving scenario.

[0054] In some embodiments, the target detection task head includes a first bird's-eye view encoder and a preset target detection model, the lane line segmentation task head includes a second bird's-eye view encoder and a preset lane line segmentation model, and the drivable area segmentation task head includes a third bird's-eye view encoder and a preset drivable area segmentation model.

[0055] In this embodiment, the first bird's eye view encoder, the second bird's eye view encoder and the third bird's eye view encoder can all be understood as pre-set BEV editors for implementing the feature set conversion into a feature set that is more relevant to the corresponding task. The architectures of the first bird's eye view encoder, the second bird's eye view encoder and the third bird's eye view encoder can be the same, but the corresponding weight parameters are different.

[0056] Specifically, the first bird's-eye view encoder is set according to the target detection task, and the center-based 3D object detection and tracking (CenterPoint) model is used as the preset target detection model; the second bird's-eye view encoder is set according to the lane line segmentation task, and the BEVerse model is used as the preset lane line segmentation model; the third bird's-eye view encoder is set according to the drivable area segmentation task, and the Lidar Multitask model based on the Lidar multitask road perception network is selected as the preset drivable area segmentation model. Through the above technical solutions, the autonomous driving system can perceive the surrounding environment more accurately, providing a strong guarantee for safe driving.

[0057] In some embodiments, the radar is a 4D millimeter wave radar.

[0058] Specifically, 4D millimeter-wave radar is an advanced sensor that can provide information in four dimensions: distance, speed, horizontal angle, and pitch angle of a target through multiple-input multiple-output (MIMO) antenna technology. Considering that 4D millimeter-wave radar can provide robust perception capabilities in extreme environments, this embodiment uses 4D millimeter-wave radar to obtain point cloud data of the environment, laying the foundation for further improving the accuracy of multi-task perception.

[0059] Embodiment 2

[0060] Figure 2 It is a flowchart of a multi-task perception method for autonomous driving provided in Embodiment 2 of the present invention. This embodiment is optimized and expanded on the basis of the above-mentioned optional embodiments.

[0061] Optionally, the image data is processed to obtain an initial first bird's-eye view feature, including: inputting the image data into an image feature extraction network to obtain image semantic features; inputting the image semantic features into a monocular depth estimation network to obtain a cone feature map; and using a bird's-eye view pooler to convert the cone feature map into an initial first bird's-eye view feature. Through the above technical solution, the initial first bird's-eye view feature containing rich semantic and depth information can be effectively generated, thereby providing accurate environmental perception capabilities for autonomous driving vehicles.

[0062] Optionally, the point cloud data is processed to obtain the initial second bird's-eye view features, including: using the external parameter data and internal parameter data of the radar, as well as the internal parameter data and external parameter data of the camera, projecting the point cloud data onto an image plane to obtain a point cloud projection map; inputting the point cloud projection map into a point cloud detection model to obtain the initial second bird's-eye view features. The above technical solution can significantly improve the perception accuracy and reliability of the surrounding environment of the autonomous driving system, and provide strong data support for subsequent path planning and decision-making.

[0063] like Figure 2 As shown, the method includes:

[0064] S210, acquiring perception data at a plurality of consecutive moments, wherein the perception data includes image data acquired by a vehicle-mounted camera and point cloud data acquired by a vehicle-mounted radar.

[0065] S220: Input the image data into an image feature extraction network to obtain image semantic features.

[0066] In this embodiment, the image feature extraction network can be understood as a pre-set neural network for extracting semantic features of image data.

[0067] Specifically, the image data is input into an image feature extraction network, and the network is used to extract image semantic features in the image data, wherein the image semantic features belong to 2D features.

[0068] S230: Input the image semantic features into a monocular depth estimation network to obtain a viewing cone feature map.

[0069] In this embodiment, the monocular depth estimation network is used to obtain the depth value of each pixel in the image. The cone feature map can be understood as a three-dimensional data structure, which is used to represent the mapping relationship from a two-dimensional image to a three-dimensional space and contains rich depth information and image features.

[0070] Specifically, the image semantic features are input into a monocular depth estimation network to obtain a depth map, and then the depth map and the image semantic features are combined to obtain a cone feature map.

[0071] S240, using a bird's-eye view pooler to convert the cone feature map into an initial first bird's-eye view feature.

[0072] In this embodiment, the bird's-eye view pooler, which may also be referred to as a bird's-eye view pooling module, is used to convert a three-dimensional feature map into a feature representation from a bird's-eye view perspective, for example, BEVPoolv2.

[0073] Specifically, the cone feature map is converted into the initial first bird's-eye view feature through a bird's-eye view pooler, so as to map the 2D image semantic features into BEV features.

[0074] S250, using the external parameter data and internal parameter data of the radar, as well as the internal parameter data and external parameter data of the camera, project the point cloud data onto an image plane to obtain a point cloud projection image.

[0075] In this embodiment, the external parameter data of the radar describes the position and attitude of the radar in the world coordinate system; these parameters may include a first rotation matrix and a first translation vector. The internal parameter data of the radar contains the physical characteristics of the radar, such as laser emission angle, beam width, measurement range, etc. The internal parameter data of the camera include focal length, principal point coordinates, pixel size, and possible distortion coefficients. The external parameter data of the camera describes the position and attitude of the camera in the world coordinate system; these parameters may include a second rotation matrix and a second translation vector. The image plane can be understood as a two-dimensional plane of camera imaging, which is formed by projection of points in the three-dimensional world behind the camera lens.

[0076] Specifically, according to the external parameter data and internal parameter data of the radar, as well as the internal parameter data and external parameter data of the camera, the point cloud data is projected onto the image plane to obtain a point cloud projection map.

[0077] S260: Input the point cloud projection image into a point cloud detection model to obtain an initial second bird's-eye view feature.

[0078] In this embodiment, the point cloud detection model is used to divide the three-dimensional point cloud into clusters of pillars, and then operate on these pillars to extract features. For example, the point cloud detection model can be a PointPillars model.

[0079] Specifically, the point cloud detection model is used to voxelize the point cloud projection image into Pillars on the XY plane, and the point cloud features in each Pillar are expanded and learned. A fixed feature vector is obtained through pooling, and finally arranged into the initial second bird's-eye view features.

[0080] Exemplarily, the point cloud feature extraction structure of the PointPillars model is used to voxelize the point cloud projection map. The process is: a fixed-size grid is divided on the XY plane of the point cloud projection map, and each grid unit extends in the Z-axis direction to form Pillars, thereby realizing the voxelization of the point cloud projection map. Furthermore, for each non-empty Pillar, the feature dimension of each point inside it is expanded to 9 dimensions to include more information. Subsequently, these extended point cloud features are learned and the maximum pooling operation is applied to abstract each Pillar into a feature vector. Finally, the feature vectors of all Pillars are arranged on the bird's-eye view (BEV) plane to form a point cloud BEV feature.

[0081] S270, projecting the initial first bird's-eye view features at different times into the initial first bird's-eye view features at the target time to obtain the target first bird's-eye view features at the target time, and projecting the initial second bird's-eye view features at different times into the initial second bird's-eye view features at the target time to obtain the target second bird's-eye view features at the target time.

[0082] S280. Concatenate the first bird's-eye view feature and the second bird's-eye view feature at the corresponding moment to obtain a feature set including a plurality of concatenated features, wherein the first bird's-eye view feature includes an initial first bird's-eye view feature and a target first bird's-eye view feature, and the second bird's-eye view feature includes an initial second bird's-eye view feature and a target second bird's-eye view feature.

[0083] S290. Input the feature set into multiple task heads respectively to obtain corresponding task perception results.

[0084] The technical solution provided by the second embodiment of the present invention realizes the conversion of image data into the initial first bird's-eye view features, the conversion of the point cloud projection map into the initial second bird's-eye view features, and then the fusion of the two modal data for multi-task perception, which effectively improves the accuracy of the multi-task results.

[0085] In some embodiments, the image feature extraction network includes ResNet50 and a feature pyramid network.

[0086] In this embodiment, ResNet50 is a deep residual network, mainly composed of multiple residual blocks, each of which contains multiple convolutional layers, batch normalization layers and ReLU activation functions. These residual blocks add the input directly to the output through skip connections, forming the core of residual learning. Feature pyramid FPN is a network structure for multi-scale target detection, which fuses feature maps of different levels to improve the model's detection ability for targets of different scales. This structure enables each layer of feature maps to have strong semantic information and rich detail information, which helps to improve the detection performance of the model.

[0087] Specifically, the combination of ResNet50 and FPN can fully utilize the powerful feature extraction capability of ResNet50 and the multi-scale feature fusion capability of FPN to extract the semantic features of image data, so that the image feature extraction network has a higher ability to understand and express the semantic information in the image data.

[0088] For example, Figure 3 is a processing diagram of a multi-task perception method for autonomous driving provided by an embodiment of the present invention, such as Figure 3 As shown in the figure: obtain the real image data and point cloud data at n moments; use ResNet50 and FPN to extract features from the image data to obtain image semantic features; convert the image semantic features into image BEV features; project the point cloud data to the image plane according to the internal and external parameter data calibrated by the radar and camera to obtain a point cloud projection map; use the point cloud detection model to obtain the point cloud BEV features; fuse the image BEV features and point cloud BEV features corresponding to n moments to obtain a feature set; input the feature set into multiple task heads respectively: perform 3D target detection through the first bird's-eye view encoder and the 3D target detection model, perform lane line segmentation through the second bird's-eye view encoder and the lane line segmentation model, and realize the segmentation of the drivable area through the third bird's-eye view encoder and the drivable area segmentation model.

[0089] Embodiment 3

[0090] Figure 4 Schematic diagram of the structure of a multi-task perception device for autonomous driving provided by Embodiment 3 of the present invention. Figure 4 As shown, the device comprises:

[0091] A data acquisition module 31 is used to acquire perception data at a plurality of consecutive moments, wherein the perception data includes image data acquired by a vehicle-mounted camera and point cloud data acquired by a vehicle-mounted radar;

[0092] A data processing module 32, used to process the image data to obtain an initial first bird's-eye view feature, and to process the point cloud data to obtain an initial second bird's-eye view feature;

[0093] A target feature determination module 33 is used to project the initial first bird's-eye view features at different times into the initial first bird's-eye view features at the target time to obtain the target first bird's-eye view features at the target time, and to project the initial second bird's-eye view features at different times into the initial second bird's-eye view features at the target time to obtain the target second bird's-eye view features at the target time;

[0094] A feature stitching module 34 is used to stitch the first bird's-eye view feature and the second bird's-eye view feature at the corresponding moment to obtain a feature set including a plurality of stitched features, wherein the first bird's-eye view feature includes an initial first bird's-eye view feature and a target first bird's-eye view feature, and the second bird's-eye view feature includes an initial second bird's-eye view feature and a target second bird's-eye view feature;

[0095] The result determination module 35 is used to input the feature set into multiple task heads respectively to obtain corresponding task perception results.

[0096] The technical solution provided in this third embodiment effectively improves the accuracy of environmental perception.

[0097] Optionally, the data processing module 32 includes:

[0098] A semantic feature extraction unit, used for inputting the image data into an image feature extraction network to obtain image semantic features;

[0099] A feature map determining unit, used for inputting the image semantic features into a monocular depth estimation network to obtain a visual cone feature map;

[0100] The first feature determination unit is used to convert the cone feature map into an initial first bird's-eye view feature by using a bird's-eye view pooler.

[0101] Optionally, the data processing module 32 includes:

[0102] A projection map determining unit, configured to project the point cloud data onto an image plane using the external parameter data and the internal parameter data of the radar, and the internal parameter data and the external parameter data of the camera, to obtain a point cloud projection map;

[0103] The first feature determination unit is used to input the point cloud projection image into the point cloud detection model to obtain an initial second bird's-eye view feature.

[0104] Optionally, the multiple task heads include a target detection task head, a lane line segmentation task head and a drivable area segmentation task head;

[0105] Optionally, the result determination module 35 includes:

[0106] A first result unit, used to input the feature set into the target detection task head and output the target detection result;

[0107] A second result unit, used to input the feature set into the lane line segmentation task head and output the lane line segmentation task result;

[0108] The third result unit is used to input the feature set into the drivable area segmentation task head and output the drivable area segmentation result.

[0109] Optionally, the target detection task head includes a first bird's-eye view encoder and a preset target detection model, the lane line segmentation task head includes a second bird's-eye view encoder and a preset lane line segmentation model, and the drivable area segmentation task head includes a third bird's-eye view encoder and a preset drivable area segmentation model.

[0110] Optionally, the image feature extraction network includes ResNet50 and a feature pyramid network.

[0111] Optionally, the radar is a 4D millimeter wave radar.

[0112] The multi-task perception device for autonomous driving provided in an embodiment of the present invention can execute the multi-task perception method for autonomous driving provided in any embodiment of the present invention, and has functional modules and beneficial effects corresponding to the execution method.

[0113] Embodiment 4

[0114] Figure 5 : is a structural schematic diagram of an electronic device provided in Embodiment 4 of the present invention. The electronic device may be an electronic device intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0115] like Figure 5As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0116] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0117] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 performs the various methods and processes described above, such as a multi-task perception method for autonomous driving.

[0118] In some embodiments, the multi-task perception method for autonomous driving may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the multi-task perception method for autonomous driving described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to perform the multi-task perception method for autonomous driving in any other appropriate manner (e.g., by means of firmware).

[0119] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0120] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0121] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0122] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0123] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0124] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0125] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0126] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A multi-task perception method for autonomous driving, It is characterized in that include: Acquire perception data at a plurality of consecutive moments, wherein the perception data includes image data acquired by a vehicle-mounted camera and point cloud data acquired by a vehicle-mounted radar; Processing the image data to obtain an initial first bird's-eye view feature, and processing the point cloud data to obtain an initial second bird's-eye view feature; Projecting the initial first bird's-eye view features at different times into the initial first bird's-eye view features at the target time to obtain the target first bird's-eye view features at the target time, and projecting the initial second bird's-eye view features at different times into the initial second bird's-eye view features at the target time to obtain the target second bird's-eye view features at the target time; Splicing the first bird's-eye view feature and the second bird's-eye view feature at the corresponding moment to obtain a feature set including a plurality of spliced ​​features, wherein the first bird's-eye view feature includes an initial first bird's-eye view feature and a target first bird's-eye view feature, and the second bird's-eye view feature includes an initial second bird's-eye view feature and a target second bird's-eye view feature; The feature sets are respectively input into multiple task heads to obtain corresponding task perception results.

2. The method according to claim 1, It is characterized in that The image data is processed to obtain an initial first bird's-eye view feature, including: Inputting the image data into an image feature extraction network to obtain image semantic features; Inputting the image semantic features into a monocular depth estimation network to obtain a cone feature map; The cone feature map is converted into an initial first bird's-eye view feature using a bird's-eye view pooler.

3. The method according to claim 1, It is characterized in that The point cloud data is processed to obtain initial second bird's-eye view features, including: Using the external parameter data and the internal parameter data of the radar, and the internal parameter data and the external parameter data of the camera, the point cloud data is projected onto an image plane to obtain a point cloud projection image; The point cloud projection image is input into the point cloud detection model to obtain an initial second bird's-eye view feature.

4. The method according to claim 1, It is characterized in that The multiple task heads include a target detection task head, a lane line segmentation task head and a drivable area segmentation task head; The feature set is input into a plurality of task heads respectively to obtain corresponding task perception results, including: Input the feature set into the target detection task head and output the target detection result; Input the feature set into the lane line segmentation task head, and output the lane line segmentation task result; The feature set is input into the drivable area segmentation task head, and the drivable area segmentation result is output.

5. The method according to claim 4, It is characterized in that The target detection task head includes a first bird's-eye view encoder and a preset target detection model, the lane line segmentation task head includes a second bird's-eye view encoder and a preset lane line segmentation model, and the drivable area segmentation task head includes a third bird's-eye view encoder and a preset drivable area segmentation model.

6. The method according to claim 2, It is characterized in that The image feature extraction network includes ResNet50 and a feature pyramid network.

7. The method according to claim 1, It is characterized in that The radar is a 4D millimeter wave radar.

8. A multi-task perception device for autonomous driving, It is characterized in that include: A data acquisition module, used to acquire perception data at multiple consecutive moments, wherein the perception data includes image data acquired by a vehicle-mounted camera and point cloud data acquired by a vehicle-mounted radar; A data processing module, used to process the image data to obtain an initial first bird's-eye view feature, and to process the point cloud data to obtain an initial second bird's-eye view feature; a target feature determination module, configured to project the initial first bird's-eye view features at different times into the initial first bird's-eye view features at the target time to obtain the target first bird's-eye view features at the target time, and to project the initial second bird's-eye view features at different times into the initial second bird's-eye view features at the target time to obtain the target second bird's-eye view features at the target time; A feature stitching module, configured to stitch the first bird's-eye view feature and the second bird's-eye view feature at the corresponding moment to obtain a feature set including a plurality of stitched features, wherein the first bird's-eye view feature includes an initial first bird's-eye view feature and a target first bird's-eye view feature, and the second bird's-eye view feature includes an initial second bird's-eye view feature and a target second bird's-eye view feature; The result determination module is used to input the feature set into multiple task heads respectively to obtain corresponding task perception results.

9. An electronic device, It is characterized in that The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the multi-task perception method for autonomous driving as described in any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, It is characterized in that When the program is executed by a processor, the multi-task perception method for autonomous driving as described in any one of claims 1 to 7 is implemented.

Images