Bird's eye view based perception processing method and device, electronic device, and medium

Abstract

Embodiments of the present application provide a perception processing method and device based on a bird's-eye view, an electronic device, and a medium. The method comprises: acquiring image data collected in an autonomous driving process, and performing perspective conversion on first feature information in the image data based on camera parameters to obtain second feature information under a bird's-eye view; using a diffusion model to denoise the second feature information to obtain third feature information; and performing a perception task under the bird's-eye view according to the third feature information. Through the embodiments of the present application, the diffusion model is used to denoise the feature information under the bird's-eye view, the projection position deviation of the features in the image data under the bird's-eye view is reduced, the quality of the feature information under the bird's-eye view is improved, and the performance of the perception task under the bird's-eye view is improved.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving technology, and in particular to a perception processing method and apparatus, electronic device, and medium based on a bird's-eye view. Background Technology

[0002] In autonomous driving scenarios, the collected image data will be converted to a bird's-eye view (BEV) perspective, which will then enable perception tasks to be performed from this perspective.

[0003] In existing technologies, the perspective transformation of image data is usually based on camera parameters. However, this perspective transformation depends on the accuracy of the camera parameters and the flatness of the ground. If the camera parameters are not accurate enough or the ground is not flat enough (such as when a vehicle is on an uphill or downhill road), the projection position of the features in the image data will be significantly deviated from the bird's-eye view, thus affecting the perception task performed from the bird's-eye view. Summary of the Invention

[0004] In view of the above problems, a bird's-eye view-based perception processing method, apparatus, electronic device, and medium are proposed to overcome or at least partially solve the above problems, including:

[0005] A perception processing method based on a bird's-eye view, the method comprising:

[0006] The system acquires image data collected during autonomous driving and performs perspective transformation on the first feature information in the image data based on camera parameters to obtain the second feature information from a bird's-eye view.

[0007] A diffusion model is used to denoise the second feature information to obtain the third feature information;

[0008] Based on the third feature information, perform a perception task from a bird's-eye view.

[0009] Optionally, a diffusion model is used to denoise the second feature information to obtain the third feature information, including:

[0010] Generate depth distribution information for image data;

[0011] Using depth distribution information as input to the diffusion model, the second feature information is denoised to obtain the third feature information.

[0012] Optionally, a diffusion model is used to denoise the second feature information to obtain the third feature information, including:

[0013] Using the second feature information as the input condition of the diffusion model, the diffusion model is used to denoise the second feature information to obtain the third feature information.

[0014] Optionally, based on the third feature information, a perception task is performed from a bird's-eye view, including:

[0015] The second and third feature information are fused.

[0016] Based on the fused feature information, a perception task is performed from a bird's-eye view.

[0017] Optionally, the second feature information and the third feature information are fused, including:

[0018] A cross-attention model is used to determine the mapping relationship between the second and third feature information, and the second and third feature information are fused based on the mapping relationship.

[0019] Optionally, it also includes:

[0020] When the second or third feature information is detected as incomplete object feature information, a diffusion model is used to reconstruct the second or third feature information.

[0021] Optionally, the perception task from a bird's-eye view includes one or more of the following:

[0022] Semantic segmentation task, 3D object detection task.

[0023] A perception processing device based on a bird's-eye view, the device comprising:

[0024] The perspective conversion module is used to acquire image data collected during autonomous driving, and based on camera parameters, to convert the first feature information in the image data to obtain the second feature information from a bird's-eye view.

[0025] The feature denoising module is used to denoise the second feature information using a diffusion model to obtain the third feature information;

[0026] The perception task execution module is used to perform perception tasks from a bird's-eye view based on the third feature information.

[0027] An electronic device includes a processor, a memory, and a computer program stored in the memory and capable of running on the processor. When the computer program is executed by the processor, it implements the above-described bird's-eye view-based perception processing method.

[0028] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described bird's-eye view-based perception processing method.

[0029] The embodiments of the present invention have the following advantages:

[0030] In this embodiment of the invention, image data collected during autonomous driving is acquired, and based on camera parameters, the first feature information in the image data is transformed by perspective conversion to obtain the second feature information under a bird's-eye view. A diffusion model is used to denoise the second feature information to obtain the third feature information. Then, based on the third feature information, a perception task under a bird's-eye view is performed. This realizes the use of a diffusion model to denoise the feature information under a bird's-eye view, reduces the projection position deviation of features in the image data under a bird's-eye view, improves the quality of feature information under a bird's-eye view, and thus improves the performance of perception tasks under a bird's-eye view. Attached Figure Description

[0031] To more clearly illustrate the technical solution of the present invention, the accompanying drawings used in the description of the present invention will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0032] Figure 1 This is a flowchart illustrating the steps of a bird's-eye view-based perception method according to an embodiment of the present invention.

[0033] Figure 2a This is a schematic diagram of a system architecture based on a diffusion model provided in an embodiment of the present invention;

[0034] Figure 2b This is a schematic diagram of a cross-attention model provided in an embodiment of the present invention;

[0035] Figure 3 This is a flowchart of another perception method based on a bird's-eye view provided in an embodiment of the present invention;

[0036] Figure 4 This is a flowchart of another perception method based on a bird's-eye view provided in an embodiment of the present invention;

[0037] Figure 5 This is a flowchart of another perception method based on a bird's-eye view provided in an embodiment of the present invention;

[0038] Figure 6 This is a flowchart of another perception method based on a bird's-eye view provided in an embodiment of the present invention;

[0039] Figure 7 This is a structural block diagram of a bird's-eye view-based sensing device provided in an embodiment of the present invention. Detailed Implementation

[0040] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0041] Reference Figure 1 The diagram illustrates a flowchart of a bird's-eye view-based perception method according to an embodiment of the present invention, which may specifically include the following steps:

[0042] Step 101: Acquire image data collected during autonomous driving, and based on camera parameters, perform viewpoint transformation on the first feature information in the image data to obtain the second feature information from a bird's-eye view.

[0043] Cameras can be deployed in mobile platforms such as vehicles to collect image data. During autonomous driving, these cameras can collect image data of the surrounding environment to assist in autonomous driving.

[0044] In practical applications, the image data acquired by the camera is two-dimensional image data. Due to the limitations of features in two-dimensional images, such as the inability of two-dimensional image data to reflect the spatial position of features relative to the surrounding environment, it may have a negative impact on some perception tasks in autonomous driving, such as affecting the accuracy of semantic segmentation tasks and three-dimensional object detection tasks.

[0045] Based on this, features from a two-dimensional image perspective can be transformed to a bird's-eye view perspective by combining camera parameters (such as the camera's intrinsic and extrinsic parameters). Figure 2a The perspective shift process is as follows:

[0046] 1. The backbone network extracts features from the image data, resulting in feature maps of different sizes. These feature maps can then be input into the viewpoint converter. For example, the backbone network can use a ResNet50 structure, which can output five layers of feature maps at different scales.

[0047] 2. In the view transformer, camera parameters can be incorporated to project the input feature map onto a bird's-eye view, and then the projected feature maps are stitched together to form the feature map in the bird's-eye view. For example, the view transformer can employ a multi-scale dense transformer network (MDT), which projects five layers of feature maps of different scales onto the bird's-eye view, resulting in five feature maps with different depth ranges. These feature maps can then be stitched together along the channel direction, such as according to the distance between the feature map and the origin (which can be the camera coordinate system), thus obtaining the feature map in the bird's-eye view.

[0048] In one example, after obtaining features from a two-dimensional image perspective, data augmentation can be performed on the features; similarly, after obtaining features from a bird's-eye view perspective, data augmentation can also be performed on the features.

[0049] Step 102: Using a diffusion model, the second feature information is denoised to obtain the third feature information.

[0050] Since the perspective transformation process is combined with camera parameters, and this perspective transformation method depends on the accuracy of the camera parameters and the flatness of the ground, if the camera parameters are not accurate enough or the ground is not flat enough (such as when the vehicle is on an uphill or downhill road), the projection position of the features in the image data will be significantly deviated from the bird's-eye view, thus affecting the perception task performed from the bird's-eye view.

[0051] Based on this, after combining camera parameters to perform viewpoint transformation, the embodiments of the present invention can use a diffusion model to denoise the second feature information obtained after viewpoint transformation, and obtain the denoised third feature information. The third feature information has better quality than the second feature information, and reduces the projection position deviation of features in two-dimensional image data under bird's-eye view.

[0052] The specific structural implementation of the diffusion model can adopt a structure similar to UNet, such as... Figure 2a The diffusion model includes an encoder and a decoder. The difference is that the diffusion model adds skip connections, which can speed up the convergence of the model. The diffusion model can gradually denoise a Gaussian noise distribution and turn it into noise-free data.

[0053] In one example, input conditions can be set for the diffusion model, such as Figure 2aThis forms a Conditional Diffusion Model. By setting input conditions, the denoising effect of the diffusion model can be improved. The second feature information with noise is input into the diffusion model with set conditions, and the noise is gradually removed through multiple iterations to obtain high-quality third feature information.

[0054] In one embodiment of the present invention, a diffusion model is used to denoise the second feature information to obtain the third feature information, including:

[0055] The depth distribution information of the generated image data is used as the input condition for the diffusion model. The diffusion model is then used to denoise the second feature information to obtain the third feature information.

[0056] During the viewpoint transformation process, depth distribution information of the image data can be generated. The depth distribution information can include the depth information of the features in the image data. Then, the depth distribution information can be used as the input condition of the diffusion model. The diffusion model with the depth distribution information as the input condition is then used to denoise the second feature information to obtain the third feature information.

[0057] In one embodiment of the present invention, a diffusion model is used to denoise the second feature information to obtain the third feature information, including:

[0058] Using the second feature information as the input condition of the diffusion model, the diffusion model is used to denoise the second feature information to obtain the third feature information.

[0059] In practical applications, the second feature information obtained after perspective transformation can be used as the input condition of the diffusion model. Then, the diffusion model with the second feature information as the input condition can be used to denoise the second feature information to obtain the third feature information.

[0060] Step 103: Perform a perception task from a bird's-eye view based on the third feature information.

[0061] After obtaining the third feature information, perception tasks can be performed from a bird's-eye view during autonomous driving, based on the third feature information, to assist autonomous driving, such as using the perception results of the perception tasks to assist in tasks such as road layout estimation and motion path planning in autonomous driving.

[0062] As an example, perception tasks from a bird's-eye view include one or more of the following:

[0063] Semantic segmentation task, 3D object detection task.

[0064] The semantic segmentation task involves analyzing the third feature information from a bird's-eye view to obtain pixel-level semantic classification results from that view, such as what the semantic type of a pixel is (e.g., car, pedestrian, obstacle, etc.).

[0065] The task of 3D object detection is to analyze the third feature information from a bird's-eye view to obtain the 3D spatial information and semantic type of the contained objects. The 3D spatial information can include position, size (length, width, height, etc.), velocity, orientation angle, etc.

[0066] In one example, different task-specific decoder structures can be set for different perception tasks, such as... Figure 2a For semantic segmentation tasks, the ResNet structure with fully connected classification layers can be used, while for 3D object detection tasks, the CenterNet structure can be used.

[0067] In this embodiment of the invention, the algorithm can be flexibly extended to different perception tasks by changing the task decoding head. It can be made into a single-task perception algorithm or a unified multi-task perception algorithm.

[0068] In one embodiment of the present invention, a perception task from a bird's-eye view is performed based on third feature information, including:

[0069] The second and third feature information are fused; based on the fused feature information, a perception task is performed from a bird's-eye view.

[0070] In practical applications, the second feature information (features obtained from a bird's-eye view after perspective transformation) and the third feature information (features obtained after denoising the second feature information through a diffusion model) can be fused. This allows for the combination of the advantages of both to form higher-quality feature information. Then, based on the fused feature information, a perception task can be performed from a bird's-eye view.

[0071] In one embodiment of the present invention, the fusion of the second feature information and the third feature information includes:

[0072] A cross-attention model is used to determine the mapping relationship between the second and third feature information, and the second and third feature information are fused based on the mapping relationship.

[0073] In practical applications, a cross-attention module can be pre-configured, such as... Figure 2aA cross-attention model can be used to establish a mapping relationship between the second feature information before denoising and the third feature information after denoising, such as a pixel-level mapping relationship. Then, based on the mapping relationship, the second feature information and the third feature information can be fused, such as fusing the same pixels.

[0074] In cross-attention models, such as Figure 2b The third feature information output by the diffusion model can be used as a query request to query the second feature information, obtain the corresponding key-value pairs in the second feature information, and then establish a mapping relationship between the two. Based on the mapping relationship, the second feature information and the third feature information can be fused.

[0075] In one example, different weights can be set for the second and third feature information, and then the weights can be used to fuse the second and third feature information.

[0076] In this embodiment of the invention, a more accurate depth mapping network is constructed through a cross-attention mechanism, thereby improving the perception performance of position, velocity, and orientation angle, and improving the quality of 3D target detection.

[0077] In one embodiment of the present invention, it further includes:

[0078] When the second or third feature information is detected as incomplete object feature information, a diffusion model is used to reconstruct the second or third feature information.

[0079] For perception tasks such as 3D object detection, the perception capability decreases when processing incomplete object feature information (feature information of occluded or truncated objects), while diffusion models have a natural generative capability and can reconstruct occluded or truncated objects.

[0080] Based on this, after obtaining the second feature information from the bird's-eye view, it is possible to detect whether the second feature information is incomplete object feature information. If the second feature information is incomplete object feature information, a diffusion model can be used to reconstruct the second feature information to obtain complete second feature information. If the second feature information is complete object feature information, no processing is required.

[0081] In practical applications, after obtaining the third feature information from a bird's-eye view, it is also possible to detect whether the third feature information is incomplete object feature information. If the third feature information is incomplete object feature information, a diffusion model can be used to reconstruct the third feature information to obtain complete third feature information. If the third feature information is complete object feature information, no processing is required.

[0082] In this embodiment of the invention, image data collected during autonomous driving is acquired, and based on camera parameters, the first feature information in the image data is transformed by perspective conversion to obtain the second feature information under a bird's-eye view. A diffusion model is used to denoise the second feature information to obtain the third feature information. Then, based on the third feature information, a perception task under a bird's-eye view is performed. This realizes the use of a diffusion model to denoise the feature information under a bird's-eye view, reduces the projection position deviation of features in the image data under a bird's-eye view, improves the quality of feature information under a bird's-eye view, and thus improves the performance of perception tasks under a bird's-eye view.

[0083] Reference Figure 3 The diagram illustrates a flowchart of another perception method based on a bird's-eye view provided by an embodiment of the present invention, which may specifically include the following steps:

[0084] Step 301: Acquire image data collected during autonomous driving, and based on camera parameters, perform viewpoint transformation on the first feature information in the image data to obtain the second feature information from a bird's-eye view.

[0085] Step 302: Generate depth distribution information of the image data.

[0086] During the perspective transformation process, depth distribution information of the image data can be generated, which can include the depth information of features in the image data.

[0087] Step 303: Using the depth distribution information as the input condition of the diffusion model, the diffusion model is used to denoise the second feature information to obtain the third feature information.

[0088] After obtaining the depth distribution information, the depth distribution information can be used as the input condition of the diffusion model. Then, the diffusion model with the depth distribution information as the input condition is used to denoise the second feature information to obtain the third feature information.

[0089] Step 304: Perform a perception task from a bird's-eye view based on the third feature information.

[0090] Reference Figure 4 The diagram illustrates a flowchart of another perception method based on a bird's-eye view provided by an embodiment of the present invention, which may specifically include the following steps:

[0091] Step 401: Acquire image data collected during autonomous driving, and based on camera parameters, perform viewpoint transformation on the first feature information in the image data to obtain the second feature information from a bird's-eye view.

[0092] Step 402: Using the second feature information as the input condition of the diffusion model, the diffusion model is used to denoise the second feature information to obtain the third feature information.

[0093] In practical applications, the second feature information obtained after perspective transformation can be used as the input condition of the diffusion model. Then, the diffusion model with the second feature information as the input condition can be used to denoise the second feature information to obtain the third feature information.

[0094] Step 403: Perform a perception task from a bird's-eye view based on the third feature information.

[0095] Reference Figure 5 The diagram illustrates a flowchart of another perception method based on a bird's-eye view provided by an embodiment of the present invention, which may specifically include the following steps:

[0096] Step 501: Acquire image data collected during autonomous driving, and based on camera parameters, perform viewpoint transformation on the first feature information in the image data to obtain the second feature information from a bird's-eye view.

[0097] Step 502: Using a diffusion model, the second feature information is denoised to obtain the third feature information.

[0098] Step 503: Fuse the second feature information and the third feature information.

[0099] In practical applications, the second feature information (features obtained from a bird's-eye view after perspective transformation) and the third feature information (features obtained after denoising the second feature information through a diffusion model) can be fused to combine the advantages of both and form higher quality feature information.

[0100] Step 504: Perform a perception task from a bird's-eye view based on the fused feature information.

[0101] After obtaining the fused feature information, perception tasks can be performed from a bird's-eye view.

[0102] Reference Figure 6 The diagram illustrates a flowchart of another perception method based on a bird's-eye view provided by an embodiment of the present invention, which may specifically include the following steps:

[0103] Step 601: Acquire image data collected during autonomous driving, and based on camera parameters, perform viewpoint transformation on the first feature information in the image data to obtain the second feature information from a bird's-eye view.

[0104] Step 602: Generate depth distribution information of image data.

[0105] During the perspective transformation process, depth distribution information of the image data can be generated, which can include the depth information of features in the image data.

[0106] Step 603: Using the depth distribution information as the input condition of the diffusion model, the diffusion model is used to denoise the second feature information to obtain the third feature information.

[0107] After obtaining the depth distribution information, the depth distribution information can be used as the input condition of the diffusion model. Then, the diffusion model with the depth distribution information as the input condition is used to denoise the second feature information to obtain the third feature information.

[0108] Step 604: Using a cross-attention model, determine the mapping relationship between the second feature information and the third feature information, and fuse the second feature information and the third feature information based on the mapping relationship.

[0109] In practical applications, a cross-attention model can be pre-set, such as... Figure 2a A cross-attention model can be used to establish a mapping relationship between the second feature information before denoising and the third feature information after denoising, such as a pixel-level mapping relationship. Then, based on the mapping relationship, the second feature information and the third feature information can be fused, such as fusing the same pixels.

[0110] In cross-attention models, such as Figure 2b The third feature information output by the diffusion model can be used as a query request to query the second feature information, obtain the corresponding key-value pairs in the second feature information, and then establish a mapping relationship between the two. Then, the second feature information and the third feature information can be fused based on the mapping relationship.

[0111] In one example, different weights can be set for the second and third feature information, and then the weights can be used to fuse the second and third feature information.

[0112] In this embodiment of the invention, a more accurate depth mapping network is constructed through a cross-attention mechanism, thereby improving the perception performance of position, velocity, and orientation angle, and improving the quality of 3D target detection.

[0113] Step 605: Perform a perception task from a bird's-eye view based on the fused feature information.

[0114] After obtaining the fused feature information, perception tasks can be performed from a bird's-eye view.

[0115] It should be noted that, for the sake of simplicity, the method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments of the present invention are not limited to the described order of actions, because according to the embodiments of the present invention, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions involved are not necessarily essential to the embodiments of the present invention.

[0116] Reference Figure 7 The diagram shows a structural schematic of a bird's-eye view-based sensing device according to an embodiment of the present invention, which may specifically include the following modules:

[0117] The perspective conversion module 701 is used to acquire image data collected during autonomous driving, and based on camera parameters, to convert the first feature information in the image data to obtain the second feature information from a bird's-eye view.

[0118] The feature denoising module 702 is used to denoise the second feature information using a diffusion model to obtain the third feature information.

[0119] The perception task execution module 703 is used to perform perception tasks from a bird's-eye view based on the third feature information.

[0120] In one embodiment of the present invention, the feature denoising module 702 includes:

[0121] The depth distribution information generation submodule is used to generate depth distribution information for image data.

[0122] The denoising submodule with depth distribution information as a condition is used to denoise the second feature information by using the depth distribution information as the input condition of the diffusion model and adopting the diffusion model to obtain the third feature information.

[0123] In one embodiment of the present invention, the feature denoising module 702 includes:

[0124] The second feature information-conditional denoising submodule is used to denoise the second feature information by using the second feature information as the input condition of the diffusion model and adopting the diffusion model to obtain the third feature information.

[0125] In one embodiment of the present invention, the perception task execution module 703 includes:

[0126] The feature fusion submodule is used to fuse the second and third feature information.

[0127] The task execution submodule based on fused features is used to perform perception tasks from a bird's-eye view based on the fused feature information.

[0128] In one embodiment of the present invention, the feature fusion submodule includes:

[0129] The mapping relationship fusion unit is used to determine the mapping relationship between the second feature information and the third feature information by adopting a cross-attention model, and to fuse the second feature information and the third feature information based on the mapping relationship.

[0130] In one embodiment of the present invention, it further includes:

[0131] The feature reconstruction module is used to reconstruct the second or third feature information by employing a diffusion model when the second or third feature information is detected as incomplete object feature information.

[0132] In one embodiment of the present invention, the perception task from a bird's-eye view includes any one or more of the following:

[0133] Semantic segmentation task, 3D object detection task.

[0134] In this embodiment of the invention, image data collected during autonomous driving is acquired, and based on camera parameters, the first feature information in the image data is transformed by perspective conversion to obtain the second feature information under a bird's-eye view. A diffusion model is used to denoise the second feature information to obtain the third feature information. Then, based on the third feature information, a perception task under a bird's-eye view is performed. This realizes the use of a diffusion model to denoise the feature information under a bird's-eye view, reduces the projection position deviation of features in the image data under a bird's-eye view, improves the quality of feature information under a bird's-eye view, and thus improves the performance of perception tasks under a bird's-eye view.

[0135] An embodiment of the present invention also provides an electronic device, which may include a processor, a memory, and a computer program stored in the memory and capable of running on the processor. When the computer program is executed by the processor, it implements the above-described bird's-eye view-based perception processing method.

[0136] An embodiment of the present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the above-described bird's-eye view-based perception processing method.

[0137] As the device embodiment is basically similar to the method embodiment, the description is relatively simple, and relevant parts can be found in the description of the method embodiment.

[0138] It should be noted that the user-related data involved in the embodiments of the present invention are all processed in accordance with the relevant data protection laws and policies of the country where the user is located, and with the authorization granted by the owner of the relevant device.

[0139] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0140] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, apparatus, or computer program products. Therefore, embodiments of the present invention can take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects. Furthermore, embodiments of the present invention can take the form of computer program products implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0141] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0142] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0143] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal equipment, causing a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0144] Although preferred embodiments of the present invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the present invention.

[0145] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes the aforementioned element.

[0146] The above provides a detailed description of the bird's-eye view-based perception processing method, device, electronic equipment, and medium. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A perception processing method based on a bird's-eye view, characterized in that, The method includes: Image data collected during autonomous driving is acquired, and based on camera parameters, the first feature information in the image data is transformed by perspective to obtain the second feature information from a bird's-eye view. A diffusion model is used to denoise the second feature information to obtain the third feature information; Based on the third feature information, perform a perception task from a bird's-eye view. The method of using a diffusion model to denoise the second feature information to obtain the third feature information includes: Generate depth distribution information of the image data; Using the depth distribution information as the input condition of the diffusion model, the second feature information is denoised using the diffusion model to obtain the third feature information.

2. The method according to claim 1, characterized in that, The diffusion model is used to denoise the second feature information to obtain the third feature information, including: Using the second feature information as the input condition of the diffusion model, the diffusion model is used to denoise the second feature information to obtain the third feature information.

3. The method according to claim 1 or 2, characterized in that, The step of performing a perception task from a bird's-eye view based on the third feature information includes: The second and third feature information are fused together; Based on the fused feature information, a perception task is performed from a bird's-eye view.

4. The method according to claim 3, characterized in that, The fusion of the second feature information and the third feature information includes: A cross-attention model is used to determine the mapping relationship between the second feature information and the third feature information, and the second feature information and the third feature information are fused based on the mapping relationship.

5. The method according to claim 1, characterized in that, Also includes: When the second or third feature information is detected as incomplete object feature information, the diffusion model is used to reconstruct the second or third feature information.

6. The method according to claim 1, characterized in that, Perception tasks from a bird's-eye view include one or more of the following: Semantic segmentation task, 3D object detection task.

7. A perception processing device based on a bird's-eye view, characterized in that, The device includes: The perspective conversion module is used to acquire image data collected during autonomous driving, and based on camera parameters, to convert the first feature information in the image data to obtain the second feature information from a bird's-eye view. The feature denoising module is used to denoise the second feature information using a diffusion model to obtain the third feature information; The perception task execution module is used to perform perception tasks from a bird's-eye view based on the third feature information. The feature denoising module includes: The depth distribution information generation submodule is used to generate depth distribution information for image data; The denoising submodule based on depth distribution information is used to denoise the second feature information by using the depth distribution information as the input condition of the diffusion model and employing the diffusion model to obtain the third feature information.

8. An electronic device, characterized in that, It includes a processor, a memory, and a computer program stored in the memory and capable of running on the processor, wherein the computer program, when executed by the processor, implements the bird's-eye view-based perception processing method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, A computer program is stored on the computer-readable storage medium, which, when executed by a processor, implements the bird's-eye view-based perception processing method as described in any one of claims 1 to 6.

Images