Task processing method, electronic device, storage medium and program product

Abstract

The invention relates to a task processing method, an electronic device, a storage medium and a program product. The electronic device obtains image feature information and point cloud feature information of a detection scene; performing aerial view feature conversion processing on the image feature information and the point cloud feature information to obtain first aerial view feature information corresponding to the image feature information and second aerial view feature information corresponding to the point cloud feature information; performing fusion processing on the first aerial view feature information and the second aerial view feature information to obtain fused aerial view feature information; and performing task processing according to the fused aerial view feature information to obtain a task processing result corresponding to the detection scene, thereby effectively improving the precision and effect of the task processing result.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a task processing method, an electronic device, a storage medium and a program product. Background Art

[0002] In technologies such as autonomous driving, it is usually necessary to perform related tasks such as target detection. In current target detection tasks, the commonly used method is to use lidar and visual sensors for target detection respectively, and then fuse the obtained results. Although this method is relatively simple and direct, there are inconsistent or incomplete information between the lidar and the visual sensor, which will lead to poor accuracy and effect of the related task processing results. Summary of the invention

[0003] The embodiments of the present application provide a task processing method, an electronic device, a storage medium, and a program product, which can effectively improve the accuracy and effect of task processing results.

[0004] In a first aspect, an embodiment of the present application provides a task processing method, the method comprising:

[0005] Obtain image feature information and point cloud feature information of the detection scene;

[0006] Performing bird's-eye view feature conversion processing on the image feature information and the point cloud feature information respectively to obtain first bird's-eye view feature information corresponding to the image feature information and second bird's-eye view feature information corresponding to the point cloud feature information;

[0007] Fusing the first bird's-eye view feature information and the second bird's-eye view feature information to obtain fused bird's-eye view feature information;

[0008] Task processing is performed based on the fused bird's-eye view feature information to obtain the task processing results corresponding to the detection scene.

[0009] In this embodiment, when the electronic device performs relevant task processing on the current detection scene, it can first obtain image feature information and point cloud feature information of the detection scene, and then convert both the image feature information and the point cloud feature information into bird's-eye view feature information, i.e., first bird's-eye view feature information and second bird's-eye view feature information, and then fuse the first bird's-eye view feature information and the second bird's-eye view feature information, thereby using the fused bird's-eye view feature information to perform task processing, which can effectively improve the robustness of the task processing method and improve the accuracy and effect of the task processing results.

[0010] Furthermore, in some embodiments, acquiring image feature information and point cloud feature information of the detection scene includes:

[0011] Acquire image data and point cloud data of the detection scene; wherein the image data represents image data obtained by photographing the detection scene from different perspectives by a camera; and the point cloud data represents point cloud data obtained by scanning the detection scene by a radar;

[0012] Perform feature extraction processing on the image data according to the image encoder to obtain image feature information;

[0013] The point cloud data is processed for feature extraction according to the point cloud encoder to obtain point cloud feature information.

[0014] In this embodiment, when acquiring image feature information and point cloud feature information of the detection scene, the electronic device can first acquire image data obtained by photographing the detection scene from different perspectives using a camera, and point cloud data obtained by scanning the detection scene using a radar, and then perform feature extraction processing on the image data using an image encoder to obtain image feature information, and perform feature extraction processing on the point cloud data using a point cloud encoder to obtain point cloud feature information, thereby acquiring features of the detection scene in different dimensions to improve the accuracy of subsequent task processing.

[0015] Furthermore, in some embodiments, the image feature information and the point cloud feature information are respectively subjected to bird's-eye view feature conversion processing to obtain first bird's-eye view feature information corresponding to the image feature information and second bird's-eye view feature information corresponding to the point cloud feature information, including:

[0016] Performing depth prediction processing on the image feature information to obtain depth information corresponding to the image feature information;

[0017] Determine first bird's-eye view feature information based on the depth information and the image feature information;

[0018] Performing spatial division processing on the point cloud feature information to obtain the divided point cloud feature data;

[0019] The second bird's-eye view feature information is determined based on the divided point cloud feature data.

[0020] In this embodiment, the electronic device can perform depth prediction on the image feature information to obtain depth information, and then use the depth information and the image feature information to determine the first bird's-eye view feature information, so that the first bird's-eye view feature information can include features of the dimension corresponding to the depth information, so as to improve the accuracy of subsequent task processing based on the first bird's-eye view feature information; at the same time, the point cloud feature information can be spatially divided, and then the second bird's-eye view feature information can be determined using the divided point cloud feature data, which can reduce the amount of calculation and complexity.

[0021] Further, in some embodiments, the depth information includes a depth parameter corresponding to each pixel in the image feature information; determining the first bird's-eye view feature information based on the depth information and the image feature information includes:

[0022] Adjust each pixel feature in the image feature information according to the depth parameter corresponding to each pixel to obtain an adjusted pixel feature;

[0023] Constructing a three-dimensional feature point cloud according to the adjusted pixel features, and performing discretization processing on the feature points of the three-dimensional feature point cloud according to a preset quantization step size to obtain discrete feature points;

[0024] The first bird's-eye view feature information is determined based on the discrete feature points.

[0025] In this embodiment, the electronic device can adjust each pixel feature in the image feature information according to the depth parameter corresponding to each pixel in the depth information to obtain the adjusted pixel feature, so as to obtain a feature with more depth information; and then construct a three-dimensional feature point cloud according to the adjusted pixel feature to better express the spatial structure of the scene; and then use a preset quantization step size to perform discrete feature point processing on the three-dimensional feature point cloud to obtain discrete feature points, thereby mapping the three-dimensional feature point cloud onto a two-dimensional bird's-eye view plane, which is convenient for subsequent operations on the feature points to improve the conversion efficiency of converting the point cloud features into the first bird's-eye view feature information.

[0026] Further, in some embodiments, determining the first bird's-eye view feature information based on the discrete feature points includes:

[0027] Perform pooling on discrete feature points to obtain aggregated feature information;

[0028] Compressing the height information of the discrete feature points to obtain first height feature information;

[0029] The first bird's-eye view feature information is determined according to the first height feature information and the aggregated feature information.

[0030] In this embodiment, the electronic device can summarize and aggregate discrete feature points by performing pooling processing on discrete feature points to obtain aggregated feature information, and compress the height information to obtain first height feature information, thereby ensuring that the final first bird's-eye view feature information also includes features of the height dimension, thereby improving the accuracy of subsequent task processing.

[0031] Further, in some embodiments, determining the second bird's-eye view feature information based on the divided point cloud feature data includes:

[0032] Perform plane projection processing on the divided point cloud feature data to obtain two-dimensional plane feature information;

[0033] Compressing the height information of the divided point cloud feature data to obtain second height feature information;

[0034] The second bird's-eye view feature information is determined according to the two-dimensional plane feature information and the second height feature information.

[0035] In this embodiment, the electronic device can perform planar projection on the point cloud feature data to obtain two-dimensional plane feature information representing horizontal direction information, obtain height information in the height dimension, and compress the height information into second height feature information, so that the second bird's-eye view feature information can be jointly constructed based on the two-dimensional plane feature information in the horizontal direction and the second height feature information in the height dimension, thereby improving the processing accuracy of subsequent task processing.

[0036] Further, in some embodiments, the task processing includes object center detection processing and semantic segmentation processing; the task processing result includes object detection result and semantic segmentation result; the task processing is performed according to the fused bird's-eye view feature information to obtain the task processing result corresponding to the detection scene, including:

[0037] The object detection task module is used to detect the center of the object on the fused bird's-eye view feature information to obtain the object detection result; wherein the object detection result represents the result of detecting the object in the detection scene;

[0038] The semantic segmentation task module is used to perform semantic segmentation processing on the fused bird's-eye view feature information to obtain a semantic segmentation result; wherein the semantic segmentation result represents the result of segmenting targets with different semantics in the detection scene.

[0039] In this embodiment, when the electronic device uses the fused bird's-eye view feature information to perform task processing, different tasks can be performed through different task modules, including using the object detection task module to perform object center detection processing on the fused bird's-eye view feature information to complete the object detection task and obtain object detection results for different objects existing in the detection scene; at the same time, the semantic segmentation task module can also be used to perform semantic segmentation on the fused bird's-eye view feature information to obtain semantic segmentation results after segmenting targets of different categories or different semantics in the detection scene, which can not only improve the effect and accuracy of task processing, but also improve the applicability to different tasks.

[0040] In a second aspect, an embodiment of the present application provides an electronic device, the electronic device comprising an acquisition unit, a conversion unit, a fusion unit, and a task processing unit;

[0041] An acquisition unit, used to acquire image feature information and point cloud feature information of a detection scene;

[0042] a conversion unit, configured to perform bird's-eye view feature conversion processing on the image feature information and the point cloud feature information, respectively, to obtain first bird's-eye view feature information corresponding to the image feature information and second bird's-eye view feature information corresponding to the point cloud feature information;

[0043] A fusion unit, used for fusing the first bird's-eye view feature information and the second bird's-eye view feature information to obtain fused bird's-eye view feature information;

[0044] The task processing unit is used to perform task processing according to the fused bird's-eye view feature information to obtain the task processing result corresponding to the detection scene.

[0045] In this embodiment, when the electronic device performs relevant task processing on the current detection scene, it can first obtain image feature information and point cloud feature information of the detection scene, and then convert both the image feature information and the point cloud feature information into bird's-eye view feature information, i.e., first bird's-eye view feature information and second bird's-eye view feature information, and then fuse the first bird's-eye view feature information and the second bird's-eye view feature information, thereby using the fused bird's-eye view feature information to perform task processing, which can effectively improve the robustness of the task processing method and improve the accuracy and effect of the task processing results.

[0046] In a third aspect, an embodiment of the present application provides an electronic device, which includes a processor and a memory storing instructions executable by the processor; when the instructions are executed by the processor, the above-mentioned task processing method is implemented.

[0047] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-mentioned task processing method.

[0048] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program or instructions, which, when executed by a processor, implements the steps in the above-mentioned task processing method. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 Schematic diagram of the implementation process of the task processing method provided in the embodiment of the present application Figure 1 ;

[0050] Figure 2 Schematic diagram of the implementation process of the task processing method provided in the embodiment of the present application Figure 2 ;

[0051] Figure 3 Schematic diagram of the composition structure of the electronic device provided in the embodiment of the present application Figure 1 ;

[0052] Figure 4Schematic diagram of the composition structure of the electronic device provided in the embodiment of the present application Figure 2 . DETAILED DESCRIPTION

[0053] The following will describe the embodiments of the present invention with reference to the accompanying drawings and preferred embodiments. Those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be understood that the preferred embodiments are only for illustrating the present invention, not for limiting the scope of protection of the present invention.

[0054] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0055] In technologies such as autonomous driving, it is usually necessary to perform related tasks such as target detection. In the current target detection tasks, the common method is to use lidar and visual sensors for target detection respectively, and then fuse the results; or the raw data of lidar and visual sensors can be fused, and then processed using a unified target detection algorithm. The advantage of this method is that it can make full use of the information of the two sensors, but it has high requirements for the fusion algorithm. Although this method is relatively simple and direct, there are inconsistent or incomplete information between lidar and visual sensors, which will lead to poor accuracy and effect of the related task processing results.

[0056] In order to solve the above problems, in the embodiments of the present application, a task processing method, an electronic device, a storage medium and a program product are proposed, wherein the electronic device obtains the image feature information and the point cloud feature information of the detection scene; performs bird's-eye view feature conversion processing on the image feature information and the point cloud feature information respectively, and obtains the first bird's-eye view feature information corresponding to the image feature information and the second bird's-eye view feature information corresponding to the point cloud feature information; performs fusion processing on the first bird's-eye view feature information and the second bird's-eye view feature information to obtain fused bird's-eye view feature information; performs task processing according to the fused bird's-eye view feature information, and obtains the task processing result corresponding to the detection scene. Based on the above scheme, the accuracy and processing effect of task processing can be effectively improved.

[0057] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0058] Figure 1Schematic diagram of the implementation process of the task processing method provided in the embodiment of the present application Figure 1 ,like Figure 1 As shown, in an embodiment of the present application, a task processing method of an electronic device may include the following steps:

[0059] Step 101: Obtain image feature information and point cloud feature information of a detection scene.

[0060] In an embodiment of the present application, the electronic device may first obtain image feature information and point cloud feature information of the detection scene.

[0061] It should be noted that in the embodiments of the present application, the electronic device can be any device with communication and storage functions, and the present application does not limit it; for example, the electronic device can be an in-vehicle electronic device, a car machine, a personal computer (PC), a laptop computer and other electronic devices.

[0062] In the embodiments of the present application, the detection scene may be any scene. For example, when the electronic device is an in-vehicle electronic device, the detection scene may be a real-time environment or real-time scene in which the vehicle is located during driving.

[0063] In an embodiment of the present application, the image feature information is feature information obtained based on image data, and the point cloud feature information is feature information obtained based on point cloud data.

[0064] In some embodiments of the present application, when the electronic device acquires image feature information and point cloud feature information of the detection scene, it can acquire image data and point cloud data of the detection scene; wherein the image data represents image data obtained by shooting the detection scene from different perspectives by a camera; the point cloud data represents point cloud data obtained by scanning the detection scene by a radar; and then, feature extraction is performed on the image data according to an image encoder to obtain image feature information; and feature extraction is performed on the point cloud data according to a point cloud encoder to obtain point cloud feature information.

[0065] In an embodiment of the present application, the camera may include a camera with multiple viewing angles, for example, cameras are respectively arranged in the front, rear, left and right directions of the vehicle, so as to capture image data at different viewing angles.

[0066] In the embodiments of the present application, the type of radar is not limited in the present application, for example, it may be a laser radar, and the electronic device may obtain point cloud data obtained by scanning the detection scene through the laser radar.

[0067] Step 102: Perform bird's-eye view feature conversion processing on the image feature information and the point cloud feature information respectively to obtain first bird's-eye view feature information corresponding to the image feature information and second bird's-eye view feature information corresponding to the point cloud feature information.

[0068] In an embodiment of the present application, after acquiring the image feature information and point cloud feature information of the detection scene, the electronic device can perform bird's-eye view feature conversion processing on the image feature information and the point cloud feature information respectively to obtain first bird's-eye view feature information corresponding to the image feature information and second bird's-eye view feature information corresponding to the point cloud feature information.

[0069] In the embodiments of the present application, a bird's eye view (BEV) is used as a unified representation of fusion, and the bird's eye view is friendly to almost all perception tasks; more importantly, the conversion on the BEV perspective can retain both the structural and semantic features of the laser; on the one hand, the conversion of point cloud feature information to bird's eye view features flattens the sparse radar features in the height dimension, so no geometric distortion occurs; on the other hand, the conversion of image feature information to bird's eye view features can retain the complete semantic information of cameras from different perspectives.

[0070] In some embodiments of the present application, when the electronic device performs bird's-eye view feature conversion processing on the image feature information and the point cloud feature information respectively to obtain the first bird's-eye view feature information corresponding to the image feature information and the second bird's-eye view feature information corresponding to the point cloud feature information, it can perform depth prediction processing on the image feature information to obtain the depth information corresponding to the image feature information; then determine the first bird's-eye view feature information based on the depth information and the image feature information; and perform spatial division processing on the point cloud feature information to obtain the divided point cloud feature data; and then determine the second bird's-eye view feature information based on the divided point cloud feature data.

[0071] In some embodiments of the present application, the depth information may include a depth parameter corresponding to each pixel in the image feature information; the depth parameter may be understood as a probability weight of the pixel at different depth positions.

[0072] In some embodiments of the present application, a bird's-eye view feature conversion process can be performed based on the Lift-Splat-Shoot (LSS) and Bird's Eye View Detection (BEVDet) methods; when the electronic device performs a bird's-eye view feature conversion process on the image feature information, the electronic device can first perform a depth prediction process on the image feature information to obtain a discrete depth distribution of each pixel in the image feature information, that is, a depth parameter, and the depth parameter can represent the possibility of the pixel appearing at different depths; through the depth prediction process, the depth structure of the detection scene can be better acquired.

[0073] In some embodiments of the present application, the point cloud feature information is spatially divided, which can be understood as dividing the three-dimensional point cloud feature information into small voxels, that is, small three-dimensional grids, each of which represents a fixed three-dimensional spatial area; through spatial division processing, the amount of calculation can be reduced and direct processing of sparse, unstructured point clouds can be avoided.

[0074] In some embodiments of the present application, when the electronic device determines the first bird's-eye view feature information based on the depth information and the image feature information, it can adjust each pixel feature in the image feature information according to the depth parameter corresponding to each pixel to obtain the adjusted pixel feature; then construct a three-dimensional feature point cloud according to the adjusted pixel features, and discretize the feature points of the three-dimensional feature point cloud according to a preset quantization step size to obtain discrete feature points; thereby determining the first bird's-eye view feature information based on the discrete feature points.

[0075] In some embodiments of the present application, when adjusting each pixel feature in the image feature information according to the depth parameter corresponding to each pixel, the pixel feature is dispersed to D discrete depth points along the camera light according to the depth parameter. During the dispersion process, each feature pixel is weighted adjusted based on its corresponding depth parameter; through this adjustment, the adjusted pixel feature can be made more characteristic of depth information.

[0076] Exemplarily, by constructing a three-dimensional feature point cloud using the adjusted pixel features, a three-dimensional feature point cloud of size N×H×W×D can be obtained, where N is the number of cameras, H is the height of the three-dimensional feature point cloud image, W is the width of the three-dimensional feature point cloud image, and D is the number of discrete depth points; the three-dimensional feature point cloud can then be discretized according to a preset quantization step size. For example, if the preset quantization step size is r, r=0.4m, the position coordinates of each feature point can be discretized according to r in the two-dimensional plane, i.e., in the direction of the x and y axes, so that the discrete feature points can be aligned at fixed intervals. This operation allows the feature points to be better mapped on the two-dimensional BEV plane, which facilitates subsequent pooling operations to determine the first bird's-eye view feature information.

[0077] In some embodiments of the present application, when the electronic device determines the first bird's-eye view feature information based on discrete feature points, it can perform pooling processing on the discrete feature points to obtain aggregated feature information; and then perform compression processing on the height information of the discrete feature points to obtain first height feature information; thereby determining the first bird's-eye view feature information based on the first height feature information and the aggregated feature information.

[0078] In some embodiments of the present application, the pooling process may be maximum pooling or average pooling, which is not specifically limited in the present application; through the pooling process, the feature points can be effectively aggregated; at the same time, the height information of the feature points on the z-axis, that is, the height dimension, can be compressed to obtain the first height feature information, thereby allowing the height information to be used as part of the feature rather than as part of the spatial dimension; finally, a BEV feature map, that is, the first bird's-eye view feature information, can be constructed based on the first height feature information and the aggregated feature information.

[0079] In some embodiments of the present application, when the electronic device determines the second bird's-eye view feature information based on the divided point cloud feature data, it can perform plane projection processing on the divided point cloud feature data to obtain two-dimensional plane feature information; and perform compression processing on the height information of the divided point cloud feature data to obtain second height feature information; thereby determining the second bird's-eye view feature information based on the two-dimensional plane feature information and the second height feature information.

[0080] Exemplarily, when performing plane projection processing on the divided point cloud feature data, the three-dimensional point cloud feature data can be projected onto a two-dimensional plane, i.e., the plane where the x-axis and the y-axis are located, for each BEV perspective, so that the position information in the horizontal direction can be better represented; at the same time, since the divided point cloud feature data will include feature points of different heights, it is also necessary to compress the height information of the divided point cloud feature data, i.e., the height information of the divided point cloud feature data in the z-axis direction into a feature vector to obtain second height feature information, and then determine the second bird's-eye view feature information based on the two-dimensional plane feature information and the second height feature information, thereby enabling the original feature expression in the height direction to be retained when the three-dimensional point cloud feature data is compressed into a two-dimensional perspective, and effectively reducing the computational complexity of the point cloud.

[0081] Step 103: Fusing the first bird's-eye view feature information and the second bird's-eye view feature information to obtain fused bird's-eye view feature information.

[0082] In an embodiment of the present application, the electronic device may perform bird's-eye view feature conversion processing on the image feature information and the point cloud feature information respectively to obtain the first bird's-eye view feature information corresponding to the image feature information and the second bird's-eye view feature information corresponding to the point cloud feature information, and then perform fusion processing on the first bird's-eye view feature information and the second bird's-eye view feature information to obtain fused bird's-eye view feature information.

[0083] In some embodiments of the present application, a BEV encoder based on convolution blocks can be used to fuse the first bird's-eye view feature information and the second bird's-eye view feature information, thereby compensating for the situation where the first bird's-eye view feature information and the second bird's-eye view feature information cannot be aligned in depth due to inaccurate spatial misalignment to a certain extent, thereby improving the accuracy of the fused bird's-eye view feature information.

[0084] In some embodiments of the present application, when the first bird's-eye view feature information and the second bird's-eye view feature information are fused, different fusion methods can be adopted for the static targets and dynamic targets in the first bird's-eye view feature information and the second bird's-eye view feature information, respectively. For static targets, since the position of the static targets does not change significantly over time, the feature fusion is relatively simple, and only feature stacking is required; for dynamic targets, since the position of the dynamic targets will change at different time points, motion compensation can be adopted to achieve feature fusion of the dynamic targets.

[0085] In some embodiments of the present application, when the first bird's-eye view feature information and the second bird's-eye view feature information are fused to obtain the fused bird's-eye view feature information, feature stacking processing can be performed on the static targets in the first bird's-eye view feature information and the second bird's-eye view feature information to obtain the fused features of the static targets; motion compensation processing can be performed on the dynamic targets in the first bird's-eye view feature information and the second bird's-eye view feature information to obtain the fused features of the dynamic targets; and the fused bird's-eye view feature information is determined based on the fused features of the static targets and the fused features of the dynamic targets.

[0086] Exemplarily, when the first bird's-eye view feature information and the second bird's-eye view feature information corresponding to the current frame are fused, the position of the dynamic target in the current frame can be predicted based on the motion trajectory of the dynamic target in the historical frame, and then the predicted position and the posture transformation corresponding to the current frame are compensated to the historical frame, and then the compensated bird's-eye view feature information corresponding to the historical frame and the bird's-eye view feature information corresponding to the current frame are stacked to obtain the fused bird's-eye view feature information corresponding to the current frame.

[0087] Step 104: Perform task processing according to the fused bird's-eye view feature information to obtain a task processing result corresponding to the detection scene.

[0088] In an embodiment of the present application, the electronic device can fuse the first bird's-eye view feature information and the second bird's-eye view feature information to obtain the fused bird's-eye view feature information, and then perform task processing based on the fused bird's-eye view feature information to obtain a task processing result corresponding to the detection scene.

[0089] In some embodiments of the present application, the electronic device can perform corresponding task processing on the fused bird's-eye view feature information based on different task heads, i.e., different task modules or task layers, to obtain different task processing results; this type of multi-task head structure allows the system to efficiently complete multiple 3D perception tasks in the same framework, such as 3D object detection, BEV map segmentation, etc.

[0090] In some embodiments of the present application, task processing may include object center detection processing and semantic segmentation processing; accordingly, the task processing result may include object detection result and semantic segmentation result.

[0091] In some embodiments of the present application, when the electronic device performs task processing based on the fused bird's-eye view feature information and obtains the task processing result corresponding to the detection scene, the electronic device can use the object detection task module to perform object center detection processing on the fused bird's-eye view feature information to obtain the object detection result; wherein the object detection result represents the result of detecting the object in the detection scene; and use the semantic segmentation task module to perform semantic segmentation processing on the fused bird's-eye view feature information to obtain the semantic segmentation result; wherein the semantic segmentation result represents the result of segmenting targets with different semantics in the detection scene.

[0092] For example, for a 3D object detection task, a heatmap head can be used to detect the center point of an object, thereby marking the center position of all targets in the detection scene; in addition, multiple regression heads can be used to estimate the properties of each object, including estimating the size, rotation angle, or movement speed of the object; for the BEV map segmentation task, since different map categories may overlap, for example, a sidewalk may be a subset of a drivable area. In order to deal with this overlapping problem, a multi-task binary classification method can be used to express the segmentation task as multiple independent binary classification tasks, so that multiple categories can be predicted and segmented.

[0093] The embodiment of the present application provides a task processing method, in which the electronic device obtains the image feature information and point cloud feature information of the detection scene; performs bird's-eye view feature conversion processing on the image feature information and the point cloud feature information respectively, and obtains the first bird's-eye view feature information corresponding to the image feature information and the second bird's-eye view feature information corresponding to the point cloud feature information; fuses the first bird's-eye view feature information and the second bird's-eye view feature information to obtain fused bird's-eye view feature information; performs task processing according to the fused bird's-eye view feature information to obtain the task processing result corresponding to the detection scene. It can be seen that in this embodiment, when the electronic device performs relevant task processing on the current detection scene, it can first obtain the image feature information and point cloud feature information of the detection scene, and then convert the image feature information and the point cloud feature information into bird's-eye view feature information, that is, the first bird's-eye view feature information and the second bird's-eye view feature information, and then fuses the first bird's-eye view feature information and the second bird's-eye view feature information, so as to use the fused bird's-eye view feature information to perform task processing, which can effectively improve the robustness of the task processing method and improve the accuracy and effect of the task processing result.

[0094] Based on the above embodiment, in another embodiment of the present application, illustratively, as Figure 2 As shown, when the electronic device performs task processing on a certain detection scene, it can first obtain multi-dimensional red, green, blue (Red, Green, Blue, RGB) image data (step 201), wherein the multi-dimensional RGB image data can be obtained by photographing the detection scene using a multi-view camera; and obtain point cloud data (step 202), wherein the point cloud data can be obtained by scanning the detection scene using a laser radar; then the image data are respectively input into an image encoder to obtain image feature information (step 203), and the point cloud data are input into a point cloud encoder to obtain point cloud feature information (step 204). Step 204); then perform BEV feature conversion on the image feature information to obtain first bird's-eye view feature information (step 205), perform BEV feature conversion on the point cloud feature information to obtain second bird's-eye view feature information (step 206), and then input the first bird's-eye view feature information and the second bird's-eye view feature information into a BEV encoder to obtain fused BEV features (step 207); finally, 3D target detection (step 208) and semantic segmentation (step 209) can be performed based on the fused bird's-eye view feature information to obtain the task results of 3D target detection and the results of semantic segmentation.

[0095] It should be noted that features of different perspectives can be extracted based on different views; for example, for a camera, each camera has a different perspective, including front, back, left, and right perspectives. This view difference will increase the difficulty of feature fusion; in addition, the current related methods will project the lidar point cloud data to a sparse depth of 2.5D, which will cause the problem of geometric damage; and related methods will project the image features obtained by the camera, such as semantic labels, machine learning model features, or virtual points, to lidar features. Due to the different feature densities between camera features and lidar features, less than 5% of the camera features can be correctly matched with the lidar features, which will affect the performance in semantic-oriented tasks such as BEV map segmentation. The present application converts image features into BEV features and point cloud features into BEV features, and then fuses the two BEV features, thereby reducing the difficulty of fusion, reducing the risk of information loss, and improving the applicability to different types of tasks; and using BEV features as a unified representation of fusion can retain the structural features of the laser and the semantic features of the camera. On the one hand, the projection of the laser radar to BEV flattens the sparse radar features along the height dimension, so that no geometric distortion occurs. On the other hand, the conversion of camera image features to BEV features can retain the complete semantic information of the camera from different perspectives.

[0096] In some embodiments of the present application, it is very important to convert image data into BEV features because the association between depth and the pixel features corresponding to each camera is essentially fuzzy. According to LSS and BEVDet, the discrete depth distribution of each pixel can be predicted first, and then each pixel feature is dispersed to D discrete depth points along the camera light according to the depth parameter. During the dispersion process, each feature pixel rescales the relevant features according to its corresponding depth parameter to generate a three-dimensional feature point cloud of size N×H×W×D, and then the three-dimensional feature point cloud is discretized according to a preset quantization step r, such as r=0.4m. Finally, the BEV pooling operation is used to aggregate the features within each BEV grid and compress along the z-axis to obtain a highly corresponding feature vector.

[0097] In some embodiments of the present application, after the corresponding features of all sensors, namely cameras and lidars, are converted into BEV features, the two BEV features can be fused and stacked; in the same space, the BEV features corresponding to the lidar and the BEV features corresponding to the camera will still have inaccurate spatial misalignment to a certain extent. For this reason, the present application applies a BEV encoder based on convolution blocks to compensate for this local misalignment, thereby improving the fusion effect.

[0098] In some embodiments of the present application, a multi-task head can be applied to the fused BEV feature map. This method is applicable to most 3D perception tasks, such as 3D object detection and BEV map segmentation. For 3D object detection tasks, a task head of a center heat map of a specific class can be used to predict the center position of all objects, and some regression heads can be used to estimate the size, rotation and speed of objects. For BEV map segmentation tasks, since different map categories may overlap, for example, crosswalks are a subset of the drivable space, a multi-task binary classification method can be used to express the segmentation task as multiple independent binary classification tasks, so that multiple categories can be predicted and segmented.

[0099] It should be noted that this application can improve the perception accuracy while compensating for the shortcomings of the visual and lidar sensors by performing feature fusion on the data obtained by the lidar and visual sensors, such as cameras; secondly, this fusion solution can effectively avoid the failure of the algorithm when a certain sensor fails, that is, when the lidar fails, the algorithm on the visual sensor side can still work effectively, or, when a certain camera fails, the overall algorithm can still work effectively.

[0100] In an embodiment of the present application, a bird's-eye view is used as an input representation. Since the bird's-eye view can present the entire scene in a bird's-eye view and present the environmental information around the vehicle as a two-dimensional plane image, it helps to simplify the target detection task and reduce the impact of vertical occlusion. This representation helps to improve the efficiency and accuracy of target detection. A dual-branch network structure is adopted, including a branch for processing lidar data and another branch for processing camera or camera data. The dual-branch structure can make full use of the information of the lidar and camera and learn the correlation between them, so as to achieve more accurate target detection. In addition, in the feature fusion stage, a multi-scale fusion strategy can be adopted to fuse the bird's-eye view feature information of different scales corresponding to the lidar number and the camera respectively. This multi-scale feature fusion can improve the robustness of target detection, enable the system to better adapt to targets of different scales and sizes, and can provide important technical support for technical fields such as autonomous driving and intelligent security.

[0101] The embodiment of the present application provides a task processing method, in which the electronic device obtains the image feature information and point cloud feature information of the detection scene; performs bird's-eye view feature conversion processing on the image feature information and the point cloud feature information respectively, and obtains the first bird's-eye view feature information corresponding to the image feature information and the second bird's-eye view feature information corresponding to the point cloud feature information; fuses the first bird's-eye view feature information and the second bird's-eye view feature information to obtain fused bird's-eye view feature information; performs task processing according to the fused bird's-eye view feature information to obtain the task processing result corresponding to the detection scene. It can be seen that in this embodiment, when the electronic device performs relevant task processing on the current detection scene, it can first obtain the image feature information and point cloud feature information of the detection scene, and then convert the image feature information and the point cloud feature information into bird's-eye view feature information, that is, the first bird's-eye view feature information and the second bird's-eye view feature information, and then fuses the first bird's-eye view feature information and the second bird's-eye view feature information, so as to use the fused bird's-eye view feature information to perform task processing, which can effectively improve the robustness of the task processing method and improve the accuracy and effect of the task processing result.

[0102] Based on the above embodiment, in another embodiment of the present application, Figure 3 Schematic diagram of the composition structure of the electronic device provided in the embodiment of the present application Figure 1 ,like Figure 3 As shown, the electronic device 1 provided in the embodiment of the present application may include an acquisition unit 11, a conversion unit 12, a fusion unit 13 and a task processing unit 14.

[0103] An acquisition unit 11 is used to acquire image feature information and point cloud feature information of a detection scene;

[0104] A conversion unit 12 is used to perform bird's-eye view feature conversion processing on the image feature information and the point cloud feature information respectively, to obtain first bird's-eye view feature information corresponding to the image feature information and second bird's-eye view feature information corresponding to the point cloud feature information;

[0105] A fusion unit 13 is used to fuse the first bird's-eye view feature information and the second bird's-eye view feature information to obtain fused bird's-eye view feature information;

[0106] The task processing unit 14 is used to perform task processing according to the fused bird's-eye view feature information to obtain a task processing result corresponding to the detection scene.

[0107] In some embodiments of the present application, the acquisition unit 11 is also used to acquire image data and point cloud data of the detection scene; and perform feature extraction processing on the image data according to the image encoder to obtain image feature information; and perform feature extraction processing on the point cloud data according to the point cloud encoder to obtain point cloud feature information; wherein the image data represents image data obtained by shooting the detection scene from different perspectives with a camera; and the point cloud data represents point cloud data obtained by scanning the detection scene with a radar.

[0108] In some embodiments of the present application, the conversion unit 12 is also used to perform depth prediction processing on the image feature information to obtain depth information corresponding to the image feature information; and determine the first bird's-eye view feature information based on the depth information and the image feature information; and perform spatial division processing on the point cloud feature information to obtain divided point cloud feature data; and determine the second bird's-eye view feature information based on the divided point cloud feature data.

[0109] In some embodiments of the present application, the depth information includes a depth parameter corresponding to each pixel in the image feature information; the conversion unit 12 is further used to adjust each pixel feature in the image feature information according to the depth parameter corresponding to each pixel to obtain an adjusted pixel feature; and to construct a three-dimensional feature point cloud according to the adjusted pixel features, and to discretize the feature points of the three-dimensional feature point cloud according to a preset quantization step size to obtain discrete feature points; and to determine the first bird's-eye view feature information based on the discrete feature points.

[0110] In some embodiments of the present application, the conversion unit 12 is also used to perform pooling processing on discrete feature points to obtain aggregated feature information; and to perform compression processing on the height information of discrete feature points to obtain first height feature information; and to determine the first bird's-eye view feature information based on the first height feature information and the aggregated feature information.

[0111] In some embodiments of the present application, the conversion unit 12 is also used to perform plane projection processing on the divided point cloud feature data to obtain two-dimensional plane feature information; and to compress the height information of the divided point cloud feature data to obtain second height feature information; and to determine the second bird's-eye view feature information based on the two-dimensional plane feature information and the second height feature information.

[0112] In some embodiments of the present application, task processing includes object center detection processing and semantic segmentation processing; task processing results include object detection results and semantic segmentation results; the task processing unit 14 is also used to use the object detection task module to perform object center detection processing on the fused bird's-eye view feature information to obtain object detection results; wherein the object detection result represents the result of detecting the object in the detection scene; and use the semantic segmentation task module to perform semantic segmentation processing on the fused bird's-eye view feature information to obtain a semantic segmentation result; wherein the semantic segmentation result represents the result of segmenting targets with different semantics in the detection scene.

[0113] In some embodiments of the present application, the fusion unit 13 is also used to perform feature stacking processing on static targets in the first bird's-eye view feature information and the second bird's-eye view feature information to obtain fusion features of the static targets; and to perform motion compensation processing on dynamic targets in the first bird's-eye view feature information and the second bird's-eye view feature information to obtain fusion features of the dynamic targets; and to determine the fused bird's-eye view feature information based on the fusion features of the static targets and the fusion features of the dynamic targets.

[0114] In the embodiments of the present application, further, Figure 4 Schematic diagram of the composition structure of the electronic device provided in the embodiment of the present application Figure 2 ,like Figure 4 As shown, the electronic device 1 provided in the embodiment of the present application may further include a processor 15 and a memory 16 storing executable instructions of the processor 15 ; further, the electronic device 1 may further include a communication interface 17 and a bus 18 for connecting the processor 15 , the memory 16 and the communication interface 17 .

[0115] In the embodiment of the present application, the processor 15 can be at least one of an application-specific integrated circuit (ASIC), a digital signal processor (DSP), a digital signal processing device (DSPD), a programmable logic device (PLD), a field programmable gate array (FPGA), a central processing unit (CPU), a controller, a microcontroller, and a microprocessor. It can be understood that for different devices, the electronic device used to implement the function of the processor can also be other, and the embodiment of the present application is not specifically limited. The electronic device 10 can also include a memory 16, which can be connected to the processor 15, wherein the memory 16 is used to store executable program code, the program code includes computer operation instructions, and the memory 16 may include a high-speed RAM memory, and may also include a non-volatile memory, for example, at least two disk memories.

[0116] In the embodiment of the present application, the bus 18 is used to connect the communication interface 17, the processor 15 and the memory 16, and the mutual communication between these devices.

[0117] In the embodiment of the present application, the memory 16 is used to store instructions and data.

[0118] Further, in an embodiment of the present application, the processor 15 is used to obtain image feature information and point cloud feature information of the detection scene;

[0119] Performing bird's-eye view feature conversion processing on the image feature information and the point cloud feature information respectively to obtain first bird's-eye view feature information corresponding to the image feature information and second bird's-eye view feature information corresponding to the point cloud feature information;

[0120] Fusing the first bird's-eye view feature information and the second bird's-eye view feature information to obtain fused bird's-eye view feature information;

[0121] Task processing is performed based on the fused bird's-eye view feature information to obtain the task processing results corresponding to the detection scene.

[0122] In practical applications, the memory 16 may be a volatile memory, such as a random access memory (RAM); or a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk (HDD) or a solid-state drive (SSD); or a combination of the above types of memory, and provide instructions and data to the processor 15.

[0123] In addition, each functional module in this embodiment can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above integrated unit can be implemented in the form of hardware or software functional modules.

[0124] If the integrated unit is implemented in the form of a software function module and is not sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this embodiment is essentially or the part that contributes to the prior art or the whole or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) or a processor to perform all or part of the steps of the method of this embodiment. The aforementioned storage medium includes: U disk, mobile hard disk, read only memory (ROM), random access memory (RAM), disk or optical disk, etc. Various media that can store program codes.

[0125] An embodiment of the present application provides an electronic device, which includes an acquisition unit, a conversion unit, a fusion unit and a task processing unit; the acquisition unit is used to acquire image feature information and point cloud feature information of a detection scene; the conversion unit is used to perform bird's-eye view feature conversion processing on the image feature information and the point cloud feature information, respectively, to obtain first bird's-eye view feature information corresponding to the image feature information and second bird's-eye view feature information corresponding to the point cloud feature information; the fusion unit is used to perform fusion processing on the first bird's-eye view feature information and the second bird's-eye view feature information to obtain fused bird's-eye view feature information; the task processing unit is used to perform task processing according to the fused bird's-eye view feature information to obtain a task processing result corresponding to the detection scene. It can be seen that when the electronic device performs relevant task processing on the current detection scene, it can first obtain the image feature information and point cloud feature information of the detection scene, and then convert the image feature information and the point cloud feature information into bird's-eye view feature information, that is, the first bird's-eye view feature information and the second bird's-eye view feature information, and then fuse the first bird's-eye view feature information and the second bird's-eye view feature information, so as to use the fused bird's-eye view feature information to perform task processing, which can effectively improve the robustness of the task processing method and improve the accuracy and effect of the task processing results.

[0126] Specifically, the program instructions corresponding to a task processing method in this embodiment can be stored in a storage medium such as a CD, a hard disk, a USB flash drive, etc. When the program instructions corresponding to a task processing method in the storage medium are read or executed by an electronic device, the following steps are included:

[0127] Obtain image feature information and point cloud feature information of the detection scene;

[0128] Performing bird's-eye view feature conversion processing on the image feature information and the point cloud feature information respectively to obtain first bird's-eye view feature information corresponding to the image feature information and second bird's-eye view feature information corresponding to the point cloud feature information;

[0129] Fusing the first bird's-eye view feature information and the second bird's-eye view feature information to obtain fused bird's-eye view feature information;

[0130] Task processing is performed based on the fused bird's-eye view feature information to obtain the task processing results corresponding to the detection scene.

[0131] An embodiment of the present application provides a computer program product, including a computer program or instructions. When the computer program or instructions are executed by a processor, the steps in the above-mentioned task processing method are implemented.

[0132] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of hardware embodiments, software embodiments, or embodiments in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage and optical storage, etc.) that contain computer-usable program code.

[0133] The present application is described with reference to implementation flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the process in the flowchart. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0134] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which is implemented in the implementation flow diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0135] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for executing the steps in the flowchart. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0136] The above embodiments are only preferred embodiments for fully illustrating the present invention, and the protection scope of the present invention is not limited thereto. Any equivalent substitution or change made by a person skilled in the art based on the present invention is within the protection scope of the present invention.

Claims

1. A task processing method, characterized in that: The method comprises: Obtain image feature information and point cloud feature information of the detection scene; Performing bird's-eye view feature conversion processing on the image feature information and the point cloud feature information respectively to obtain first bird's-eye view feature information corresponding to the image feature information and second bird's-eye view feature information corresponding to the point cloud feature information; fusing the first bird's-eye view feature information and the second bird's-eye view feature information to obtain fused bird's-eye view feature information; Task processing is performed according to the fused bird's-eye view feature information to obtain a task processing result corresponding to the detection scene.

2. The method according to claim 1, characterized in that The step of acquiring the image feature information and the point cloud feature information of the detection scene includes: Acquire image data and point cloud data of the detection scene; wherein the image data represents image data obtained by photographing the detection scene from different perspectives by a camera; and the point cloud data represents point cloud data obtained by scanning the detection scene by a radar; Performing feature extraction processing on the image data according to the image encoder to obtain the image feature information; The point cloud data is subjected to feature extraction processing according to a point cloud encoder to obtain the point cloud feature information.

3. The method according to claim 2, characterized in that The performing bird's-eye view feature conversion processing on the image feature information and the point cloud feature information respectively to obtain first bird's-eye view feature information corresponding to the image feature information and second bird's-eye view feature information corresponding to the point cloud feature information includes: Performing depth prediction processing on the image feature information to obtain depth information corresponding to the image feature information; determining the first bird's-eye view feature information based on the depth information and the image feature information; Performing spatial division processing on the point cloud feature information to obtain divided point cloud feature data; The second bird's-eye view feature information is determined based on the divided point cloud feature data.

4. The method according to claim 3, characterized in that The depth information includes a depth parameter corresponding to each pixel in the image feature information; and determining the first bird's-eye view feature information based on the depth information and the image feature information includes: Adjusting each pixel feature in the image feature information according to the depth parameter corresponding to each pixel to obtain an adjusted pixel feature; Constructing a three-dimensional feature point cloud according to the adjusted pixel features, and performing discretization processing on the feature points of the three-dimensional feature point cloud according to a preset quantization step size to obtain discrete feature points; The first bird's-eye view feature information is determined based on the discrete feature points.

5. The method according to claim 4, characterized in that The determining the first bird's-eye view feature information based on the discrete feature points includes: Performing pooling processing on the discrete feature points to obtain aggregated feature information; Compressing the height information of the discrete feature points to obtain first height feature information; The first bird's-eye view feature information is determined according to the first height feature information and the aggregated feature information.

6. The method according to claim 3, characterized in that The determining the second bird's-eye view feature information based on the divided point cloud feature data includes: Performing plane projection processing on the divided point cloud feature data to obtain two-dimensional plane feature information; Compressing the height information of the divided point cloud feature data to obtain second height feature information; The second bird's-eye view feature information is determined according to the two-dimensional plane feature information and the second height feature information.

7. The method according to any one of claims 1 to 6, characterized in that: The task processing includes object center detection processing and semantic segmentation processing; the task processing result includes object detection result and semantic segmentation result; the task processing is performed according to the fused bird's-eye view feature information to obtain the task processing result corresponding to the detection scene, including: Using the object detection task module to perform detection processing on the object center of the fused bird's-eye view feature information to obtain the object detection result; wherein the object detection result represents the result of detecting the object in the detection scene; The semantic segmentation task module is used to perform the semantic segmentation processing on the fused bird's-eye view feature information to obtain the semantic segmentation result; wherein the semantic segmentation result represents the result of segmenting targets with different semantics in the detection scene.

8. The method according to any one of claims 1 to 6, characterized in that: The fusing the first bird's-eye view feature information and the second bird's-eye view feature information to obtain fused bird's-eye view feature information includes: Performing feature stacking processing on the static targets in the first bird's-eye view feature information and the second bird's-eye view feature information to obtain fused features of the static targets; Performing motion compensation processing on the dynamic targets in the first bird's-eye view feature information and the second bird's-eye view feature information to obtain fusion features of the dynamic targets; The fused bird's-eye view feature information is determined according to the fused features of the static target and the fused features of the dynamic target.

9. An electronic device, characterized in that: The electronic device includes an acquisition unit, a conversion unit, a fusion unit and a task processing unit; The acquisition unit is used to acquire image feature information and point cloud feature information of the detection scene; The conversion unit is used to perform bird's-eye view feature conversion processing on the image feature information and the point cloud feature information respectively, to obtain first bird's-eye view feature information corresponding to the image feature information and second bird's-eye view feature information corresponding to the point cloud feature information; The fusion unit is used to fuse the first bird's-eye view feature information and the second bird's-eye view feature information to obtain fused bird's-eye view feature information; The task processing unit is used to perform task processing according to the fused bird's-eye view feature information to obtain a task processing result corresponding to the detection scene.

10. An electronic device, characterized in that: The electronic device includes a processor and a memory storing instructions executable by the processor; when the instructions are executed by the processor, the method according to any one of claims 1 to 8 is implemented.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

12. A computer program product comprising a computer program or instructions, characterized in that When the computer program or instruction is executed by a processor, the steps of the task processing method described in any one of claims 1 to 8 are implemented.

Images