Scene perception method and related apparatuses

Abstract

The application discloses a scene perception method and a related device thereof, which can obtain a more accurate perception result of a target scene. The method of the application comprises the following steps: when scene perception of a target scene is required, image data of the target scene and / or point cloud data of the target scene are obtained first, and the image data and / or the point cloud data are input into a target model. Then, the target model performs feature extraction on the image data and / or the point cloud data first, so as to obtain a first BEV feature of the image data and / or a second BEV feature of the point cloud data. Then, the target model performs feature fusion on the first BEV feature and / or the second BEV feature by using a preset third BEV feature, so as to obtain a feature fusion result. Finally, the target model performs further processing on the feature fusion result, so as to obtain a perception result of the target scene. Thus, the scene perception of the target scene is completed.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence (AI) technology, and in particular to a scene perception method and related equipment. Background Technology

[0002] In the field of autonomous driving, in order for a vehicle's autonomous driving system to make accurate autonomous driving decisions, the system often needs to possess high-precision scene (environment) perception capabilities. Generally, autonomous driving systems can acquire scene information through the vehicle's sensors to obtain scene perception results based on this information.

[0003] In related technologies, vehicle sensors typically include cameras and LiDAR, among others. When a vehicle's autonomous driving system needs to perceive its surroundings, it first acquires image data of the scene using cameras and point cloud data using LiDAR. This data is then input into a neural network model built into the system. The neural network model then extracts features from both the image and point cloud data, and fuses these features to obtain the scene perception result based on the feature fusion.

[0004] In the process described above, the neural network model needs to fuse features from image data and point cloud data to obtain scene perception results using the feature fusion results. Therefore, it is evident that there is a strong coupling between the image data and point cloud data input to the neural network model. In certain special cases, if a portion of the data cannot be obtained, the final scene perception result obtained by the neural network model will be inaccurate. Summary of the Invention

[0005] This application provides a scene perception method and related device, which can still obtain relatively accurate perception results of the target scene even if the target model only obtains image data of the target scene or only obtains point cloud data of the target scene.

[0006] A first aspect of this application provides a scene perception method, which is implemented through a target model and includes:

[0007] When scene perception is required for a target scene, image data of the target scene can be acquired first using a camera and / or point cloud data of the target scene using a LiDAR. After obtaining the image data and / or point cloud data of the target scene, the image data and / or point cloud data of the target scene can be input into the target model.

[0008] After obtaining the image data and / or point cloud data of the target scene, the target model can extract features from the image data and / or point cloud data of the target scene to obtain the first bird's-eye view (BEV) features of the image data and / or the second BEV features of the point cloud data.

[0009] After obtaining the first BEV feature and / or the second BEV feature, the target model can perform feature fusion on the first BEV feature and / or the second BEV feature based on the preset third BEV feature, thereby obtaining the feature fusion result.

[0010] After obtaining the feature fusion result, the target model can further process the feature fusion structure to obtain the perception result of the target scene. This completes the scene perception for the target scene.

[0011] As can be seen from the above method, when scene perception of a target scene is required, image data and / or point cloud data of the target scene can be acquired first, and then input into the target model. Next, the target model can extract features from the image data and / or point cloud data to obtain the first BEV feature of the image data and / or the second BEV feature of the point cloud data. Then, the target model can use a preset third BEV feature to perform feature fusion on the first BEV feature and / or the second BEV feature to obtain the feature fusion result. Finally, the target model can further process the feature fusion result to obtain the perception result of the target scene. Thus, scene perception for the target scene is completed. In the aforementioned process, the target model can fuse the third BEV feature and the first BEV feature of the target scene image data to obtain a feature fusion result; it can also fuse the third BEV feature and the second BEV feature of the target scene point cloud data to obtain a feature fusion result; or it can fuse the third BEV feature, the first BEV feature of the target scene image data, and the second BEV feature of the target scene point cloud data to obtain a feature fusion result, thereby using the feature fusion result to obtain the scene perception result. Therefore, it can be seen that there is no strong coupling between the target scene image data and the target scene point cloud data input to the target model. Even if, due to certain special circumstances, the target model only obtains the target scene image data or only obtains the target scene point cloud data, the final perception result of the target scene obtained by the target model can still have high accuracy.

[0012] In one possible implementation, feature extraction of image data and / or point cloud data to obtain a first BEV feature of the image data and / or a second BEV feature of the point cloud data includes: encoding the image data and / or point cloud data to obtain the first BEV feature of the image data and / or the second BEV feature of the point cloud data. In the aforementioned implementation, if the data input to the target model includes both image data and point cloud data of the target scene, the target model can encode the image data of the target scene to obtain the first BEV feature of the image data, and the target model can also encode the point cloud data of the target scene to obtain the second BEV feature of the point cloud data. If the data input to the target model only includes image data of the target scene, the target model can encode the image data of the target scene to obtain only the first BEV feature of the image data. If the data input to the target model only includes point cloud data of the target scene, in the feature extraction module of the target model, the target model can encode the point cloud data of the target scene to obtain only the second BEV feature of the point cloud data.

[0013] In one possible implementation, feature fusion is performed on the first BEV feature and / or the second BEV feature based on a preset third BEV feature. The feature fusion result includes: processing the first BEV feature and / or the second BEV feature based on a deformable attention mechanism to obtain a fourth BEV feature; adding the third BEV feature and the fourth BEV feature to obtain a fifth BEV feature; processing the fifth BEV feature based on a hybrid expert model to obtain a sixth BEV feature; adding the fifth BEV feature and the sixth BEV feature to obtain a seventh BEV feature; and obtaining the feature fusion result based on the seventh BEV feature. In the aforementioned implementation, if the target model obtains both the first BEV feature and the second BEV feature, the target model can process the first BEV feature and the second BEV feature based on the third BEV feature using a deformable attention mechanism to obtain the fourth BEV feature. If the target model obtains only the first BEV feature, the target model can process the first BEV feature based on the third BEV feature using a deformable attention mechanism to obtain the fourth BEV feature. If the target model obtains only the second BEV feature, it can process the second BEV feature based on the third BEV feature using a deformable attention mechanism to obtain the fourth BEV feature. After obtaining the fourth BEV feature, the cross-modal deformable attention layer can send it to the first summing layer. After obtaining the fourth BEV feature, the target model can add the third and fourth BEV features to obtain the fifth BEV feature. After obtaining the fifth BEV feature, the target model can process it using a hybrid expert model to obtain the sixth BEV feature. After obtaining the sixth BEV feature, the target model can add the fifth and sixth BEV features to obtain the seventh BEV feature. After obtaining the seventh BEV feature, the target model can further process it to obtain the feature fusion result, or it can directly use the seventh BEV feature as the feature fusion result. Therefore, the target model in this application provides a cross-modal attention perception architecture and a hybrid expert architecture. The cross-modal attention perception architecture can ensure that the model can still make accurate scene perception for the target scene even when the input of the model is missing (e.g., only image data of the target scene or only point cloud data of the target scene). The hybrid expert architecture can ensure that the model can complete a variety of perception tasks for the target scene (e.g., object detection and object segmentation).

[0014] In one possible implementation, processing the first BEV feature and / or the second BEV feature based on a pre-set third BEV feature using a deformable attention mechanism to obtain a fourth BEV feature includes: processing the pre-set third BEV feature using a first multilayer perceptron to obtain first position information of a first sub-feature of the first BEV feature and / or second position information of a second sub-feature of the second BEV feature; obtaining the first sub-feature and / or the second sub-feature from the first BEV feature and / or the second BEV feature based on the first position information and / or the second position information; processing the third BEV feature using a second multilayer perceptron to obtain first weight information of the first sub-feature of the first BEV feature and / or second weight information of the second sub-feature of the second BEV feature; and performing a weighted summation of the first sub-feature and / or the second sub-feature based on the first weight information and / or the second weight information to obtain the fourth BEV feature. In the aforementioned implementation, the target model can obtain the fourth BEV feature in multiple ways: (1) If the target model obtains both the first BEV feature and the second BEV feature, the target model can process the third BEV feature based on the first multilayer perceptron to obtain the first position information of the first sub-feature of the first BEV feature and the second position information of the second sub-feature of the second BEV feature. Then, the target model can select the first sub-feature from the first BEV feature and the second sub-feature from the second BEV feature according to the instructions of the first and second position information. At the same time, the target model can also process the third BEV feature based on the second multilayer perceptron to obtain the first weight information of the first sub-feature and the second weight information of the second sub-feature. Finally, the target model can use the first weight information and the second weight information to perform a weighted summation of the first sub-feature and the second sub-feature to obtain the fourth BEV feature. (2) If the target model obtains only the first BEV feature, the target model can process the third BEV feature based on the first multilayer perceptron to obtain the first position information of the first sub-feature of the first BEV feature. Next, the target model can select the first sub-feature from the first BEV features according to the indication of the first position information. At the same time, the target model can also process the third BEV feature based on the second multilayer perceptron to obtain the first weight information of the first sub-feature. Finally, the target model can use the first weight information to perform a weighted summation of the first sub-feature to obtain the fourth BEV feature. (3) If the target model obtains only the second BEV feature, the target model can process the third BEV feature based on the first multilayer perceptron to obtain the second position information of the second sub-feature of the second BEV feature. Next, the target model can select the second sub-feature from the second BEV feature according to the indication of the second position information.Simultaneously, the target model can also process the third BEV feature using a second multilayer perceptron to obtain the second weight information of the second sub-feature. Finally, the target model can use the second weight information to perform a weighted summation of the second sub-feature to obtain the fourth BEV feature. Therefore, this implementation is a specific approach to a cross-modal attention-based perception architecture, ensuring that scene perception for the target scene can still be completed smoothly and accurately even when the model's input is incomplete.

[0015] In one possible implementation, processing the fifth BEV feature using a hybrid expert model to obtain the sixth BEV feature includes: processing the fifth BEV feature using a first expert model to obtain the eighth BEV feature; processing the fifth BEV feature using a second expert model to obtain the ninth BEV feature; and processing the eighth and ninth BEV features using a third expert model to obtain the sixth BEV feature. In this implementation, the target model can process the fifth BEV feature using the first expert model to obtain the eighth BEV feature. Simultaneously, the target model can also process the fifth BEV feature using the second expert model to obtain the ninth BEV feature. Finally, the target model can process the eighth and ninth BEV features using the third expert model to obtain the sixth BEV feature. Therefore, this implementation is a specific implementation of a hybrid expert architecture, which ensures that the model can complete multiple perception tasks for the target scenario.

[0016] In one possible implementation, processing the fifth BEV feature based on a hybrid expert model to obtain the sixth BEV feature includes: dividing the fifth BEV feature into multiple third sub-features; dividing the fifth BEV feature into multiple fourth sub-features; processing the multiple third and fourth sub-features based on multiple fourth expert models to obtain multiple fifth sub-features, wherein processing based on one fourth expert model corresponds to at least one third sub-feature and at least one fourth sub-feature; and adding the multiple fifth sub-features to obtain the sixth BEV feature. In the aforementioned implementation, the target model can divide the fifth BEV feature into multiple third sub-features. Simultaneously, the target model can also divide the fifth BEV feature into multiple fourth sub-features. Next, the target model can process the multiple third and fourth sub-features based on multiple fourth expert models to obtain multiple fifth sub-features. Finally, the target model can add the multiple fifth sub-features to obtain the sixth BEV feature. Therefore, this implementation is another specific implementation of the hybrid expert architecture, which ensures that the model can complete multiple perception tasks for the target scenario.

[0017] In one possible implementation, the point cloud data is data acquired under conditions of interference with the LiDAR, and the image data is data acquired under conditions of interference with the camera. In the aforementioned implementation, the image data of the target scene input to the target model can be data acquired by the camera under interference; similarly, the point cloud data of the target scene input to the target model can also be data acquired by the LiDAR under interference. Therefore, the image data and point cloud data of the target scene input to the target model can be of relatively poor quality (data with some loss), yet the target model can still obtain sufficiently accurate perception results of the target scene based on this data.

[0018] In one possible implementation, based on a preset third BEV feature, feature fusion is performed on the first BEV feature and / or the second BEV feature to obtain the feature fusion result. This further includes: normalizing the third BEV feature to obtain a normalized third BEV feature; normalizing the first BEV feature and / or the second BEV feature to obtain a normalized first BEV feature and / or a normalized second BEV feature; and processing the first BEV feature and / or the second BEV feature based on the preset third BEV feature using a deformable attention mechanism to obtain a fourth BEV feature. This includes: processing the normalized first BEV feature and / or the normalized second BEV feature based on the normalized third BEV feature using a deformable attention mechanism to obtain the fourth BEV feature.

[0019] In one possible implementation, based on a preset third BEV feature, feature fusion is performed on the first BEV feature and / or the second BEV feature to obtain the feature fusion result. This further includes: normalizing the fifth BEV feature to obtain a normalized fifth BEV feature; and processing the fifth BEV feature based on a hybrid expert model to obtain a sixth BEV feature, which includes: processing the normalized fifth BEV feature based on a hybrid expert model to obtain a sixth BEV feature.

[0020] A second aspect of this application provides a model training method, comprising: acquiring image data and / or point cloud data of a target scene; processing the image data and / or point cloud data using a model to be trained to obtain a perception result of the target scene, wherein the model to be trained is used to: extract features from the image data and / or point cloud data to obtain a first BEV feature of the image data and / or a second BEV feature of the point cloud data; perform feature fusion on the first BEV feature and / or the second BEV feature based on a preset third BEV feature to obtain a feature fusion result; acquire the perception result of the target scene based on the feature fusion result; and train the model to be trained based on the perception result of the target scene to obtain a target model.

[0021] The target model trained using the above method possesses scene perception capabilities. Specifically, when scene perception of a target scene is required, image data and / or point cloud data of the target scene can be acquired first, and then input into the target model. Next, the target model can extract features from the image data and / or point cloud data to obtain first BEV features from the image data and / or second BEV features from the point cloud data. Then, the target model can use a pre-set third BEV feature to fuse the first BEV features and / or second BEV features, thereby obtaining a feature fusion result. Finally, the target model can further process the feature fusion result to obtain the scene perception result. This completes the scene perception for the target scene. In the aforementioned process, the target model can fuse the third BEV feature and the first BEV feature of the target scene image data to obtain a feature fusion result; it can also fuse the third BEV feature and the second BEV feature of the target scene point cloud data to obtain a feature fusion result; or it can fuse the third BEV feature, the first BEV feature of the target scene image data, and the second BEV feature of the target scene point cloud data to obtain a feature fusion result, thereby using the feature fusion result to obtain the scene perception result. Therefore, it can be seen that there is no strong coupling between the target scene image data and the target scene point cloud data input to the target model. Even if, due to certain special circumstances, the target model only obtains the target scene image data or only obtains the target scene point cloud data, the final perception result of the target scene obtained by the target model can still have high accuracy.

[0022] In one possible implementation, the model to be trained is used to: encode image data and / or point cloud data to obtain a first BEV feature of the image data and / or a second BEV feature of the point cloud data.

[0023] In one possible implementation, the model to be trained is used to: process the first BEV feature and / or the second BEV feature based on a pre-set third BEV feature using a deformable attention mechanism to obtain a fourth BEV feature; add the third BEV feature and the fourth BEV feature to obtain a fifth BEV feature; process the fifth BEV feature based on a hybrid expert model to obtain a sixth BEV feature; add the fifth BEV feature and the sixth BEV feature to obtain a seventh BEV feature; and obtain the feature fusion result based on the seventh BEV feature.

[0024] In one possible implementation, the model to be trained is used to: process a pre-set third BEV feature using a first multilayer perceptron to obtain first position information of a first sub-feature of the first BEV feature and / or second position information of a second sub-feature of the second BEV feature; based on the first position information and / or the second position information, obtain a first sub-feature and / or a second sub-feature from the first BEV feature and / or the second BEV feature; process the third BEV feature using a second multilayer perceptron to obtain first weight information of the first sub-feature of the first BEV feature and / or second weight information of the second sub-feature of the second BEV feature; and based on the first weight information and / or the second weight information, perform a weighted summation of the first sub-feature and / or the second sub-feature to obtain a fourth BEV feature.

[0025] In one possible implementation, the model to be trained is used to: process the fifth BEV feature based on a first expert model to obtain the eighth BEV feature; process the fifth BEV feature based on a second expert model to obtain the ninth BEV feature; and process the eighth and ninth BEV features based on a third expert model to obtain the sixth BEV feature.

[0026] In one possible implementation, the model to be trained is used to: divide the fifth BEV feature into multiple third sub-features; divide the fifth BEV feature into multiple fourth sub-features; process the multiple third sub-features and multiple fourth sub-features based on multiple fourth expert models to obtain multiple fifth sub-features, wherein the processing based on one fourth expert model corresponds to at least one third sub-feature and at least one fourth sub-feature; and add the multiple fifth sub-features to obtain the sixth BEV feature.

[0027] In one possible implementation, the point cloud data is data acquired when the LiDAR is interfered with, and the image data is data acquired when the camera is interfered with.

[0028] A third aspect of this application provides a scene perception device comprising a target model. The device includes: a first acquisition module for acquiring image data and / or point cloud data of the target scene; a feature extraction module for extracting features from the image data and / or point cloud data to obtain a first BEV feature of the image data and / or a second BEV feature of the point cloud data; a feature fusion module for fusing the first BEV feature and / or the second BEV feature based on a preset third BEV feature to obtain a feature fusion result; and a second acquisition module for acquiring a perception result of the target scene based on the feature fusion result.

[0029] As can be seen from the above device, when scene perception of a target scene is required, image data and / or point cloud data of the target scene can be acquired first, and then input into the target model. Next, the target model can extract features from the image data and / or point cloud data to obtain first BEV features of the image data and / or second BEV features of the point cloud data. Then, the target model can use a preset third BEV feature to perform feature fusion on the first BEV features and / or second BEV features to obtain a feature fusion result. Finally, the target model can further process the feature fusion result to obtain the perception result of the target scene. Thus, scene perception of the target scene is completed. In the aforementioned process, the target model can fuse the third BEV feature and the first BEV feature of the target scene image data to obtain a feature fusion result; it can also fuse the third BEV feature and the second BEV feature of the target scene point cloud data to obtain a feature fusion result; or it can fuse the third BEV feature, the first BEV feature of the target scene image data, and the second BEV feature of the target scene point cloud data to obtain a feature fusion result, thereby using the feature fusion result to obtain the scene perception result. Therefore, it can be seen that there is no strong coupling between the target scene image data and the target scene point cloud data input to the target model. Even if, due to certain special circumstances, the target model only obtains the target scene image data or only obtains the target scene point cloud data, the final perception result of the target scene obtained by the target model can still have high accuracy.

[0030] In one possible implementation, a feature extraction module is used to encode image data and / or point cloud data to obtain a first BEV feature of the image data and / or a second BEV feature of the point cloud data.

[0031] In one possible implementation, the feature fusion module is configured to: process the first BEV feature and / or the second BEV feature based on a pre-set third BEV feature using a deformable attention mechanism to obtain a fourth BEV feature; add the third BEV feature and the fourth BEV feature to obtain a fifth BEV feature; process the fifth BEV feature based on a hybrid expert model to obtain a sixth BEV feature; add the fifth BEV feature and the sixth BEV feature to obtain a seventh BEV feature; and obtain the feature fusion result based on the seventh BEV feature.

[0032] In one possible implementation, the feature fusion module is configured to: process a preset third BEV feature using a first multilayer perceptron to obtain first position information of a first sub-feature of the first BEV feature and / or second position information of a second sub-feature of the second BEV feature; based on the first position information and / or the second position information, obtain a first sub-feature and / or a second sub-feature from the first BEV feature and / or the second BEV feature; process the third BEV feature using a second multilayer perceptron to obtain first weight information of the first sub-feature of the first BEV feature and / or second weight information of the second sub-feature of the second BEV feature; and based on the first weight information and / or the second weight information, perform a weighted summation of the first sub-feature and / or the second sub-feature to obtain a fourth BEV feature.

[0033] In one possible implementation, the feature fusion module is used to: process the fifth BEV feature based on a first expert model to obtain the eighth BEV feature; process the fifth BEV feature based on a second expert model to obtain the ninth BEV feature; and process the eighth and ninth BEV features based on a third expert model to obtain the sixth BEV feature.

[0034] In one possible implementation, the feature fusion module is used to: divide the fifth BEV feature into multiple third sub-features; divide the fifth BEV feature into multiple fourth sub-features; process the multiple third sub-features and multiple fourth sub-features based on multiple fourth expert models to obtain multiple fifth sub-features, wherein the processing based on one fourth expert model corresponds to at least one third sub-feature and at least one fourth sub-feature; and add the multiple fifth sub-features to obtain the sixth BEV feature.

[0035] In one possible implementation, the point cloud data is data acquired when the LiDAR is interfered with, and the image data is data acquired when the camera is interfered with.

[0036] A fourth aspect of this application provides a model training apparatus, comprising: an acquisition module for acquiring image data and / or point cloud data of a target scene; a processing module for processing the image data and / or point cloud data using a model to be trained to obtain a perception result of the target scene, wherein the model to be trained is configured to: extract features from the image data and / or point cloud data to obtain a first BEV feature of the image data and / or a second BEV feature of the point cloud data; fuse the first BEV feature and / or the second BEV feature based on a preset third BEV feature to obtain a feature fusion result; acquire the perception result of the target scene based on the feature fusion result; and a training module for training the model to be trained based on the perception result of the target scene to obtain a target model.

[0037] The target model trained by the aforementioned device possesses scene perception capabilities. Specifically, when scene perception of a target scene is required, image data and / or point cloud data of the target scene can be acquired first, and then input into the target model. Next, the target model can extract features from the image data and / or point cloud data to obtain first BEV features from the image data and / or second BEV features from the point cloud data. Then, the target model can use a preset third BEV feature to perform feature fusion on the first BEV features and / or second BEV features to obtain a feature fusion result. Finally, the target model can further process the feature fusion result to obtain the perception result of the target scene. This completes the scene perception for the target scene. In the aforementioned process, the target model can fuse the third BEV feature and the first BEV feature of the target scene image data to obtain a feature fusion result; it can also fuse the third BEV feature and the second BEV feature of the target scene point cloud data to obtain a feature fusion result; or it can fuse the third BEV feature, the first BEV feature of the target scene image data, and the second BEV feature of the target scene point cloud data to obtain a feature fusion result, thereby using the feature fusion result to obtain the scene perception result. Therefore, it can be seen that there is no strong coupling between the target scene image data and the target scene point cloud data input to the target model. Even if, due to certain special circumstances, the target model only obtains the target scene image data or only obtains the target scene point cloud data, the final perception result of the target scene obtained by the target model can still have high accuracy.

[0038] In one possible implementation, the model to be trained is used to encode image data and / or point cloud data to obtain a first BEV feature of the image data and / or a second BEV feature of the point cloud data.

[0039] In one possible implementation, the model to be trained is used to: process the first BEV feature and / or the second BEV feature based on a pre-set third BEV feature using a deformable attention mechanism to obtain a fourth BEV feature; add the third BEV feature and the fourth BEV feature to obtain a fifth BEV feature; process the fifth BEV feature based on a hybrid expert model to obtain a sixth BEV feature; add the fifth BEV feature and the sixth BEV feature to obtain a seventh BEV feature; and obtain the feature fusion result based on the seventh BEV feature.

[0040] In one possible implementation, the model to be trained is used to: process a pre-set third BEV feature using a first multilayer perceptron to obtain first position information of a first sub-feature of the first BEV feature and / or second position information of a second sub-feature of the second BEV feature; based on the first position information and / or the second position information, obtain a first sub-feature and / or a second sub-feature from the first BEV feature and / or the second BEV feature; process the third BEV feature using a second multilayer perceptron to obtain first weight information of the first sub-feature of the first BEV feature and / or second weight information of the second sub-feature of the second BEV feature; and based on the first weight information and / or the second weight information, perform a weighted summation of the first sub-feature and / or the second sub-feature to obtain a fourth BEV feature.

[0041] In one possible implementation, the model to be trained is used to: process the fifth BEV feature based on a first expert model to obtain the eighth BEV feature; process the fifth BEV feature based on a second expert model to obtain the ninth BEV feature; and process the eighth and ninth BEV features based on a third expert model to obtain the sixth BEV feature.

[0042] In one possible implementation, the model to be trained is used to: divide the fifth BEV feature into multiple third sub-features; divide the fifth BEV feature into multiple fourth sub-features; process the multiple third sub-features and multiple fourth sub-features based on multiple fourth expert models to obtain multiple fifth sub-features, wherein the processing based on one fourth expert model corresponds to at least one third sub-feature and at least one fourth sub-feature; and add the multiple fifth sub-features to obtain the sixth BEV feature.

[0043] In one possible implementation, the point cloud data is data acquired when the LiDAR is interfered with, and the image data is data acquired when the camera is interfered with.

[0044] A fifth aspect of this application provides a scene perception device, which includes a memory and a processor; the memory stores code, and the processor is configured to execute the code. When the code is executed, the scene perception device performs the method described in the first aspect or any possible implementation thereof.

[0045] A sixth aspect of this application provides a model training apparatus, which includes a memory and a processor; the memory stores code, and the processor is configured to execute the code. When the code is executed, the model training apparatus performs the method described in the second aspect or any possible implementation thereof.

[0046] A seventh aspect of this application provides a circuit system including a processing circuit configured to perform the method described in the first aspect, any possible implementation of the first aspect, the second aspect, or any possible implementation of the second aspect.

[0047] An eighth aspect of this application provides a chip system including a processor for calling a computer program or computer instructions stored in a memory to cause the processor to perform the method as described in the first aspect, any possible implementation of the first aspect, the second aspect, or any possible implementation of the second aspect.

[0048] In one possible implementation, the processor is coupled to the memory via an interface.

[0049] In one possible implementation, the chip system also includes a memory that stores computer programs or computer instructions.

[0050] A ninth aspect of this application provides a computer storage medium storing a computer program that, when executed by a computer, causes the computer to perform the method as described in the first aspect, any possible implementation of the first aspect, the second aspect, or any possible implementation of the second aspect.

[0051] A tenth aspect of this application provides a computer program product storing instructions that, when executed by a computer, cause the computer to perform the method as described in the first aspect, any possible implementation of the first aspect, the second aspect, or any possible implementation of the second aspect.

[0052] In this embodiment, when scene perception of a target scene is required, image data and / or point cloud data of the target scene can be acquired first, and then input into the target model. Next, the target model can extract features from the image data and / or point cloud data to obtain a first BEV feature of the image data and / or a second BEV feature of the point cloud data. Then, the target model can use a preset third BEV feature to perform feature fusion on the first BEV feature and / or the second BEV feature to obtain a feature fusion result. Finally, the target model can further process the feature fusion result to obtain the perception result of the target scene. Thus, scene perception of the target scene is completed. In the aforementioned process, the target model can fuse the third BEV feature and the first BEV feature of the target scene image data to obtain a feature fusion result; it can also fuse the third BEV feature and the second BEV feature of the target scene point cloud data to obtain a feature fusion result; or it can fuse the third BEV feature, the first BEV feature of the target scene image data, and the second BEV feature of the target scene point cloud data to obtain a feature fusion result, thereby using the feature fusion result to obtain the scene perception result. Therefore, it can be seen that there is no strong coupling between the target scene image data and the target scene point cloud data input to the target model. Even if, due to certain special circumstances, the target model only obtains the target scene image data or only obtains the target scene point cloud data, the final perception result of the target scene obtained by the target model can still have high accuracy. Attached Figure Description

[0053] Figure 1 A structural diagram illustrating the main framework of artificial intelligence;

[0054] Figure 2a A schematic diagram of the structure of the scene perception system provided in the embodiments of this application;

[0055] Figure 2b This is another structural schematic diagram of the scene perception system provided in the embodiments of this application;

[0056] Figure 2c A schematic diagram of a scene perception device provided in an embodiment of this application;

[0057] Figure 3 A schematic diagram of the system 100 architecture provided in the embodiments of this application;

[0058] Figure 4 A schematic diagram of the structure of the target model provided in the embodiments of this application;

[0059] Figure 5 A flowchart illustrating the scene perception method provided in this application embodiment;

[0060] Figure 6 A schematic diagram of the feature extraction module provided in an embodiment of this application;

[0061] Figure 7 A schematic diagram of the feature fusion module provided in an embodiment of this application;

[0062] Figure 8 A schematic diagram of the structure of the first fusion unit provided in the embodiments of this application;

[0063] Figure 9 A schematic diagram of the structure of the second fusion unit provided in the embodiments of this application;

[0064] Figure 10 A schematic diagram of the cross-modal deformable attention layer of the first fusion unit provided in the embodiments of this application;

[0065] Figure 11 A schematic diagram of the hybrid expert layer of the first fusion unit provided in the embodiments of this application;

[0066] Figure 12 Another schematic diagram of the hybrid expert layer of the first fusion unit provided in the embodiments of this application;

[0067] Figure 13 Another structural schematic diagram of the first fusion unit provided in the embodiments of this application;

[0068] Figure 14 A schematic flowchart of the model training method provided in the embodiments of this application;

[0069] Figure 15 A schematic diagram of the scene perception device provided in the embodiments of this application;

[0070] Figure 16 A schematic diagram of the structure of the model training apparatus provided in the embodiments of this application;

[0071] Figure 17 A schematic diagram of the structure of the execution device provided in the embodiments of this application;

[0072] Figure 18 A schematic diagram of the structure of the training device provided in the embodiments of this application;

[0073] Figure 19 This is a schematic diagram of the structure of a chip provided in an embodiment of this application. Detailed Implementation

[0074] This application provides a scene perception method and related device, which can still obtain relatively accurate perception results of the target scene even if the target model only obtains image data of the target scene or only obtains point cloud data of the target scene.

[0075] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms are interchangeable where appropriate; this is merely a way of distinguishing objects with the same attributes in the embodiments of this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of elements is not necessarily limited to those elements, but may include other elements not explicitly listed or inherent to those processes, methods, products, or apparatuses.

[0076] In the field of autonomous driving, in order for a vehicle's autonomous driving system to make accurate autonomous driving decisions, the system often needs to possess high-precision scene (environment) perception capabilities. Generally, autonomous driving systems can acquire scene information through the vehicle's sensors to obtain scene perception results based on this information.

[0077] In related technologies, vehicle sensors typically include cameras and LiDAR, among others. When a vehicle's autonomous driving system needs to perceive its surroundings, it first acquires image data of the scene using cameras and point cloud data using LiDAR. This data is then input into a neural network model built into the system. The neural network model then extracts features from both the image and point cloud data, and fuses these features to obtain the scene perception result based on the feature fusion.

[0078] In the above process, the neural network model needs to fuse features from image data and point cloud data to obtain scene perception results using the feature fusion results. Therefore, it is evident that there is a strong coupling between the image data and point cloud data input to the neural network model. In certain special cases (e.g., camera or LiDAR malfunction), if a portion of the data cannot be acquired (e.g., the camera fails to acquire image data or the LiDAR fails to acquire point cloud data), the final scene perception result obtained by the neural network model will be inaccurate.

[0079] Furthermore, if the quality of the acquired data is low (for example, the field of view of the LiDAR is limited, making it impossible to acquire point cloud data from all directions), that is, the acquired data has a certain loss, this will also lead to the inaccurate scene perception results obtained by the neural network model.

[0080] To address the aforementioned problems, this application provides a scene perception method that can be implemented in conjunction with artificial intelligence (AI) technology. AI technology is a discipline that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence. AI technology achieves optimal results by perceiving the environment, acquiring knowledge, and using that knowledge. In other words, artificial intelligence is a branch of computer science that attempts to understand the essence of intelligence and produce new intelligent machines that can react in a way similar to human intelligence. Using artificial intelligence for data processing is a common application of AI.

[0081] First, the overall workflow of the artificial intelligence system is described; please refer to [link / reference]. Figure 1 , Figure 1 This is a structural diagram illustrating the main framework of artificial intelligence. The framework is explained below from two dimensions: the "Intelligent Information Chain" (horizontal axis) and the "IT Value Chain" (vertical axis). The "Intelligent Information Chain" reflects a series of processes from data acquisition to processing. For example, it could be the general process of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, and intelligent execution and output. In this process, data undergoes a condensation process of "data—information—knowledge—wisdom." The "IT Value Chain" reflects the value that artificial intelligence brings to the information technology industry, from the underlying infrastructure of human intelligence and information (provided and processed by technology) to the industrial ecosystem of the system.

[0082] (1) Infrastructure

[0083] Infrastructure provides computing power to support artificial intelligence systems, enabling communication with the external world and providing support through a basic platform. This communication occurs through sensors; computing power is provided by intelligent chips (hardware acceleration chips such as CPUs, NPUs, GPUs, ASICs, and FPGAs); and the basic platform includes distributed computing frameworks and related platform guarantees and support, which may include cloud storage and computing, interconnected networks, etc. For example, sensors communicate with the outside world to acquire data, and this data is provided to intelligent chips in the distributed computing system provided by the basic platform for computation.

[0084] (2) Data

[0085] The data at the next layer of infrastructure is used to represent the data sources in the field of artificial intelligence. The data involves graphics, images, voice, text, as well as IoT data from traditional devices, including business data from existing systems and sensor data such as force, displacement, liquid level, temperature, and humidity.

[0086] (3) Data processing

[0087] Data processing typically includes methods such as data training, machine learning, deep learning, search, reasoning, and decision-making.

[0088] Among them, machine learning and deep learning can perform intelligent information modeling, extraction, preprocessing, and training on data by symbolizing and formalizing it.

[0089] Reasoning refers to the process in which, in a computer or intelligent system, the machine thinks and solves problems by simulating human intelligent reasoning, based on reasoning control strategies and using formalized information. Typical functions include search and matching.

[0090] Decision-making refers to the process of making decisions based on intelligent information after reasoning, and it typically provides functions such as classification, sorting, and prediction.

[0091] (4) General ability

[0092] After the data processing mentioned above, the results of the data processing can be used to form some general capabilities, such as algorithms or a general system, for example, translation, text analysis, computer vision processing, speech recognition, image recognition, etc.

[0093] (5) Smart products and industry applications

[0094] Intelligent products and industry applications refer to products and applications of artificial intelligence systems in various fields. They are the encapsulation of overall artificial intelligence solutions, productizing intelligent information decision-making and realizing practical applications. Their application areas mainly include: intelligent terminals, intelligent transportation, intelligent healthcare, autonomous driving, smart cities, etc.

[0095] The following sections will introduce several application scenarios for this application.

[0096] Figure 2a This is a schematic diagram of a scene perception system provided in an embodiment of this application. The scene perception system includes a user device and a data processing device. The user device includes smart terminals such as a mobile phone used by the user and an onboard computer in a vehicle driven by the user. The user device is the initiator of scene perception; as the initiator of scene perception requests, requests are typically initiated by the user through the user device.

[0097] The aforementioned data processing equipment can be devices or servers with data processing capabilities, such as cloud servers, network servers, application servers, and management servers. The data processing equipment receives scene perception requests from smart terminals through an interactive interface, and then performs scene perception through machine learning, deep learning, search, reasoning, and decision-making methods via a storage device and a data processing processor. The storage device in the data processing equipment can be a general term, including local storage and a database storing historical data. The database can be located on the data processing equipment or on other network servers.

[0098] exist Figure 2a In the scene perception system shown, the user device can acquire relevant data of a target scene specified by the user, and then send a request to the data processing device. This causes the data processing device to perform scene perception processing on the relevant data from the user device, thereby obtaining a perception result for the target scene. For example, when the user triggers the user device, the user device can acquire relevant data of the target scene (e.g., image data and / or point cloud data of the target scene) through sensors such as cameras and / or LiDAR. The user device can then send a scene perception request to the data processing device, causing the data processing device to perform a series of processes on the relevant data of the target scene based on the request, thereby obtaining a perception result for the target scene. This could be, for example, detection results for objects in the target scene (e.g., pedestrians, vehicles, and lane lines) or object segmentation results.

[0099] exist Figure 2a In this context, the data processing device can execute the scene perception method of the embodiments of this application.

[0100] Figure 2b This is another structural schematic diagram of the scene perception system provided in the embodiments of this application. Figure 2b In this context, the user equipment (UE) directly functions as a data processing device. This UE can directly acquire input from the user and process it directly through its own hardware. The specific process is similar to... Figure 2a Similar to the description above, it will not be repeated here.

[0101] exist Figure 2b In the scene perception system shown, when a user triggers the user device, the user device can acquire relevant data of the target scene. Then, the user device can perform a series of processing on the relevant data of the target scene to obtain the perception results of the target scene, such as object detection results or object segmentation results in the target scene.

[0102] exist Figure 2b In this context, the user equipment itself can execute the scene perception method of the embodiments of this application.

[0103] Figure 2c This is a schematic diagram of a scene perception device provided in an embodiment of this application.

[0104] The above Figure 2a and Figure 2b The user equipment in the context can specifically be Figure 2c Local device 301 or local device 302 in the system. Figure 2a The data processing equipment in the middle can specifically be Figure 2c The execution device 210 in the process includes a data storage system 250 that can store the data to be processed by the execution device 210. The data storage system 250 can be integrated into the execution device 210 or set up in the cloud or on other network servers.

[0105] Figure 2a and Figure 2b The processor in the system can perform data training / machine learning / deep learning using neural network models or other models (e.g., support vector machine-based models), and use the data to train or learn the model to perform scene perception applications on images, thereby obtaining the corresponding processing results.

[0106] Figure 3 A schematic diagram of the system 100 architecture provided in this application embodiment, in Figure 3 In this embodiment, the execution device 110 is configured with an input / output (I / O) interface 112 for data interaction with external devices. Users can input data to the I / O interface 112 through the client device 140. The input data may include various scheduled tasks, callable resources, and other parameters.

[0107] During the preprocessing of input data by the execution device 110, or during the calculation module 111 of the execution device 110 performing calculations and other related processing (such as implementing the neural network function in this application), the execution device 110 may call data, code, etc. in the data storage system 150 for corresponding processing, or store the data, instructions, etc. obtained from the corresponding processing into the data storage system 150.

[0108] Finally, I / O interface 112 returns the processing result to client device 140, thereby providing it to the user.

[0109] It is worth noting that the training device 120 can generate corresponding target models / rules based on different training data for different objectives or tasks. These target models / rules can then be used to achieve the aforementioned objectives or complete the aforementioned tasks, thereby providing the user with the required results. The training data can be stored in the database 130 and originates from training samples collected by the data acquisition device 160.

[0110] exist Figure 3 In the scenario shown, the user can manually provide input data, which can be done through the interface provided by I / O interface 112. Alternatively, the client device 140 can automatically send input data to I / O interface 112. If user authorization is required for the client device 140 to automatically send input data, the user can set the corresponding permissions in the client device 140. The user can view the output results of the execution device 110 on the client device 140, which can be presented in various forms such as display, sound, or animation. The client device 140 can also act as a data acquisition terminal, collecting the input data and output results of the input I / O interface 112 as new sample data and storing them in the database 130. Alternatively, data can be collected directly from the I / O interface 112 without going through the client device 140, using the input data and output results of the input I / O interface 112 as new sample data and storing them in the database 130.

[0111] It is worth noting that, Figure 3 This is merely a schematic diagram of a system architecture provided in an embodiment of this application. The positional relationships between the devices, components, modules, etc., shown in the diagram do not constitute any limitation. For example, in Figure 3 In this context, the data storage system 150 is an external memory relative to the execution device 110. However, in other cases, the data storage system 150 can also be placed within the execution device 110. For example... Figure 3 As shown, a neural network can be trained using training device 120.

[0112] This application also provides a chip including a neural network processor (NPU). This chip can be configured as follows: Figure 3 The execution device 110 shown is used to perform the calculations of the calculation module 111. This chip can also be located in, for example... Figure 3 The training device 120 shown is used to complete the training work of the training device 120 and output the target model / rules.

[0113] The Neural Processing Unit (NPU) is a coprocessor mounted on the main central processing unit (CPU) (host CPU), where tasks are assigned by the CPU. The core of the NPU is the computation circuitry, which is controlled by a controller to retrieve data from memory (weight memory or input memory) and perform calculations.

[0114] In some implementations, the arithmetic circuitry includes multiple process engines (PEs). In some implementations, the arithmetic circuitry is a two-dimensional pulsating array. The arithmetic circuitry can also be a one-dimensional pulsating array or other electronic circuitry capable of performing mathematical operations such as multiplication and addition. In some implementations, the arithmetic circuitry is a general-purpose matrix processor.

[0115] For example, suppose we have an input matrix A, a weight matrix B, and an output matrix C. The arithmetic circuit retrieves the corresponding data of matrix B from the weight memory and caches it in each PE (Process Equipment) of the arithmetic circuit. The arithmetic circuit retrieves the data of matrix A from the input memory and performs matrix operations with matrix B. The partial or final result of the obtained matrix is ​​stored in the accumulator.

[0116] Vector computation units can further process the output of computational circuits, such as vector multiplication, vector addition, exponentiation, logarithmic operations, size comparisons, etc. For example, vector computation units can be used for computation in non-convolutional / non-FC layers of neural networks, such as pooling, batch normalization, and local response normalization.

[0117] In some implementations, the vector computation unit can store the processed output vector into a unified buffer. For example, the vector computation unit can apply a nonlinear function to the output of the arithmetic circuit, such as a vector of accumulated values, to generate activation values. In some implementations, the vector computation unit generates normalized values, merged values, or both. In some implementations, the processed output vector can be used as activation input to the arithmetic circuit, for example, for use in subsequent layers of a neural network.

[0118] The unified memory is used to store input data and output data.

[0119] The weight data is directly transferred from the external memory to the input memory and / or unified memory, stored in the weight memory, and stored in the unified memory to the external memory through the direct memory access controller (DMAC).

[0120] The bus interface unit (BIU) is used to enable interaction between the main CPU, DMAC, and instruction fetch memory via a bus.

[0121] The instruction fetch buffer, connected to the controller, is used to store the instructions used by the controller.

[0122] The controller is used to invoke instructions cached in the memory to control the operation of the computing accelerator.

[0123] Generally, the unified memory, input memory, weight memory, and instruction fetch memory are all on-chip memories, while external memory is memory outside the NPU. This external memory can be double data rate synchronous dynamic random access memory (DDRSDRAM), high bandwidth memory (HBM), or other readable and writable memories.

[0124] Since the embodiments of this application involve a large number of neural network applications, for ease of understanding, the relevant terms and concepts such as neural networks involved in the embodiments of this application will be introduced below.

[0125] (1) Neural Network

[0126] A neural network can be composed of neural units, which can be operational units that take xs and an intercept of 1 as inputs, and whose output can be:

[0127] (1)

[0128] Where s = 1, 2, ..., n, where n is a natural number greater than 1, Ws is the weight of xs, and b is the bias of the neural unit. f is the activation function of the neural unit, used to introduce nonlinear characteristics into the neural network to convert the input signal in the neural unit into the output signal. The output signal of this activation function can be used as the input of the next convolutional layer. The activation function can be the sigmoid function. A neural network is a network formed by connecting many of the above-mentioned individual neural units together, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the features of the local receptive field, which can be a region composed of several neural units.

[0129] The function of each layer in a neural network can be expressed mathematically. To describe it: From a physical perspective, the work of each layer in a neural network can be understood as transforming the input space (the set of input vectors) to the output space (i.e., from the row space to the column space of a matrix) through five operations on the input space. These five operations include: 1. Dimensionality increase / decrease; 2. Magnification / scaling; 3. Rotation; 4. Translation; 5. "Bending". Operations 1, 2, and 3 are... Completed, operation 4 is performed by Completed, operation 5 is then performed by To achieve this. The term "space" is used here because the object being classified is not a single thing, but a category of things; space refers to the collection of all individuals within this category. This is a weight vector, where each value represents the weight of a neuron in that layer of the neural network. The weights of each layer determine the spatial transformation from the input space to the output space described above. It controls how the space is transformed. The goal of training a neural network is to ultimately obtain the weight matrix of all layers of the trained neural network (composed of vectors from many layers). (The resulting weight matrix). Therefore, the training process of a neural network is essentially about learning how to transform the control space, more specifically, learning the weight matrix.

[0130] Because we want the output of the neural network to be as close as possible to the actual predicted value, we can compare the current network's prediction with the desired target value, and then update the weight vector of each layer of the neural network based on the difference between the two (of course, there is usually an initialization process before the first update, that is, pre-configuring the parameters of each layer in the neural network). For example, if the network's prediction is too high, the weight vector is adjusted to make it predict lower, and this adjustment is continued until the neural network can predict the actual target value. Therefore, it is necessary to predefine "how to compare the difference between the predicted value and the target value," which is the loss function or objective function. These are important equations used to measure the difference between the predicted value and the target value. Taking the loss function as an example, the higher the output value (loss) of the loss function, the greater the difference, so training the neural network becomes the process of minimizing this loss as much as possible.

[0131] (2) Backpropagation algorithm

[0132] Neural networks can employ backpropagation (BP) to correct the parameters of the initial neural network model during training, thereby reducing the reconstruction error loss. Specifically, forward propagation of the input signal to the output generates error loss; this error loss information is then propagated back to update the parameters of the initial neural network model, leading to convergence of the error loss. The backpropagation algorithm is an error-loss-driven backpropagation process aimed at obtaining the optimal parameters of the neural network model, such as the weight matrix.

[0133] (3) Bird's-eye view

[0134] Bird's-eye view (BEV) images are a natural and direct candidate view that can serve as a unified representation. Compared to the front view or perspective view, which are widely studied in the field of 2D vision, BEV representation has several inherent advantages. First, BEV does not suffer from the occlusion and scale problems prevalent in 2D tasks. In the field of autonomous driving, problems such as identifying vehicles with occlusions or intersecting traffic can be better solved using BEV. Furthermore, representing objects or road elements in the form of BEV will facilitate the development and deployment of subsequent modules (such as planning and control).

[0135] The method provided in this application is described below from the perspectives of neural network training and neural network application.

[0136] The model training method provided in this application involves data sequence processing and can be applied to data training, machine learning, deep learning, and other methods. It performs symbolic and formal intelligent information modeling, extraction, preprocessing, and training on training data (e.g., image data and / or point cloud data of the target scene in the model training method provided in this application), ultimately obtaining a trained neural network (e.g., the target model in the model training method provided in this application). Furthermore, the scene perception method provided in this application can utilize the trained neural network to input input data (e.g., image data and / or point cloud data of the target scene in the scene perception method provided in this application) into the trained neural network to obtain output data (e.g., the perception result of the target scene in the scene perception method provided in this application). It should be noted that the model training method and scene perception method provided in this application are inventions based on the same concept and can be understood as two parts of a system or two stages of a whole process: such as the model training stage and the model application stage.

[0137] The scene perception method provided in this application can be applied not only to object detection and segmentation in the field of autonomous driving, but also to robot route planning in the field of logistics and transportation, and is not limited thereto. The scene perception method provided in this application can be implemented using a target model. Figure 4 A schematic diagram of the structure of the target model provided in the embodiments of this application, such as Figure 4 As shown, the target model includes a feature extraction module, a feature fusion module, and a prediction module. The input of the feature extraction module serves as the input of the entire target model. The output of the feature extraction module is connected to the input of the feature fusion module, and the output of the feature fusion module is connected to the output of the prediction module. The output of the prediction module serves as the output of the entire target model. To understand the workflow of the target model, the following section combines... Figure 5 The workflow of the target model will be introduced. Figure 5 A flowchart illustrating the scene perception method provided in this application embodiment is shown below. Figure 5 As shown, the method includes:

[0138] 501. Acquire image data and / or point cloud data of the target scene.

[0139] In this embodiment, when scene perception is required for a target scene, image data of the target scene can be collected by a camera and / or point cloud data of the target scene can be collected by a lidar.

[0140] Specifically, the following situations typically arise when collecting data:

[0141] (1) The user indicates that the image data of the target scene and the point cloud data of the target scene need to be acquired at the same time, and the camera and the lidar are in working condition. Therefore, the image data of the target scene can be acquired by the camera and the point cloud data of the target scene can be acquired by the lidar.

[0142] (2) The user indicates that the image data and point cloud data of the target scene need to be acquired at the same time, but only the camera is working and the lidar is in a fault state. Therefore, the image data of the target scene can be acquired by the camera, but the point cloud data of the target scene cannot be acquired by the lidar.

[0143] (3) The user indicates that both the image data and the point cloud data of the target scene need to be acquired at the same time, but only the lidar is working and the camera is in a faulty state. Therefore, the point cloud data of the target scene can only be acquired through the lidar, and the image data of the target scene cannot be acquired through the camera.

[0144] (4) The user indicates that only image data of the target scene needs to be acquired and the camera is in working condition. Therefore, only image data of the target scene is acquired through the camera, and point cloud data of the target scene is not acquired through the lidar.

[0145] (5) The user indicates that only the point cloud data of the target scene needs to be acquired, and the LiDAR is in working condition. Therefore, only the point cloud data of the target scene is acquired through the LiDAR, and the image data of the target scene is not acquired through the camera, etc.

[0146] Furthermore, the image data of the target scene acquired in the above-mentioned situations can be data captured by the camera without interference (in which case the acquired image data is of normal quality) or data captured by the camera under interference (in which case the acquired image data is of poor quality). The situations where the camera is interfered with include the following:

[0147] (1) View noise (VN): Camera view noise can be caused by a variety of factors, including sensor noise, electrical interference, atmospheric conditions, or compression artifacts. This embodiment simulates camera view noise by replacing a portion of the data acquired by the camera with randomly generated noise. In this way, the data acquired by the camera in the presence of view noise can be obtained, that is, image data carrying noise.

[0148] (2) View Drop (VD): View drop in a camera can be caused by various factors, including incorrect camera positioning or camera hardware failure. Similar to view noise, this embodiment sets a portion of the data collected by the camera to zero. In this way, the data collected by the camera in the presence of view drop can be obtained, which is image data with defects (not comprehensive).

[0149] (3) Obstacle occlusion (OO): This refers to a phenomenon that occurs when an object in the target scene is partially occluded by an obstacle or occluder. To simulate this phenomenon, this embodiment uses alpha mixing on the data collected by a predefined mask camera. In this way, the data collected by the camera in the presence of obstacle occlusion can be obtained, that is, the image data with occlusion, etc.

[0150] Furthermore, the point cloud data of the target scene acquired in the above situations can be either data collected by the LiDAR without interference (in which case the collected point cloud data is of normal quality) or data collected by the LiDAR under interference (in which case the collected point cloud data is of poor quality). The situations where the LiDAR is interfered with include the following:

[0151] (1) Limited Field of View (LF): Generally, point cloud data is collected from a 360-degree viewpoint. However, due to hardware failures of the lidar, point cloud data at certain angles may be lost, resulting in a collection angle of less than 360 degrees. In this embodiment, point cloud data with missing angles can be simulated, including 360, 240, 180, 120 degrees, etc. In this way, the data collected by the lidar under limited field of view can be obtained, that is, the point cloud data with missing angles.

[0152] (2) Missing Objects (MO): In the real world, differences in factors such as the color and surface material of objects may reduce the number of reflection points detected by the lidar. In this embodiment, the point cloud data of missing objects is simulated by deleting point cloud data of objects with a certain probability at a specific ratio. The selected ratio is set to 0.0, 0.1, 0.3, 0.5, 0.7, 1.0, etc. In this way, the data collected by the lidar in the presence of missing objects can be obtained, that is, the point cloud data of the partially missing objects.

[0153] (3) Beam Reduction (BR): Beam reduction is a phenomenon that occurs in sensing systems when the available power supply or processing power of the lidar is limited. This embodiment simulates point cloud data with fewer beams by reducing the number of lidar beams to 1, 4, 8, 16, and 32 for data acquisition evaluation. This allows us to obtain the data acquired by the lidar under beam reduction conditions, i.e., point cloud data with fewer beams, etc.

[0154] After obtaining the image data and / or point cloud data of the target scene, the image data and / or point cloud data of the target scene can be input into the target model.

[0155] 502. Perform feature extraction on image data and / or point cloud data to obtain the first BEV feature of the image data and / or the second BEV feature of the point cloud data.

[0156] After obtaining the image data and / or point cloud data of the target scene, the target model can perform feature extraction on the image data and / or point cloud data of the target scene to obtain the first BEV feature of the image data and / or the second BEV feature of the point cloud data.

[0157] Specifically, the feature extraction module of the target model may include a first encoder and a second encoder. The target model can then acquire the first BEV features of the image data and / or the second BEV features of the point cloud data in the following ways:

[0158] If the input data to the target model includes both image data and point cloud data of the target scene, in the feature extraction module of the target model, the first encoder can encode the image data of the target scene to obtain the first BEV feature of the image data and send the first RVE feature to the feature fusion module. Similarly, the second encoder can encode the point cloud data of the target scene to obtain the second BEV feature of the point cloud data and send the second RVE feature to the feature fusion module. If the input data to the target model only includes image data of the target scene, in the feature extraction module of the target model, the first encoder can encode the image data of the target scene to obtain only the first BEV feature of the image data and send the first RVE feature to the feature fusion module. If the input data to the target model only includes point cloud data of the target scene, in the feature extraction module of the target model, the second encoder can encode the point cloud data of the target scene to obtain only the second BEV feature of the point cloud data and send the second RVE feature to the feature fusion module.

[0159] For example, such as Figure 6 As shown ( Figure 6 (This is a schematic diagram of a feature extraction module provided in an embodiment of this application). After the image captured by the camera and the point cloud captured by the LiDAR are input into the feature extraction module of the target model, the first encoder in the feature extraction module can encode the image to obtain the BEV feature B1 of the image. At the same time, the second encoder can encode the point cloud to obtain the BEV feature B2 of the point cloud. Then, the first encoder can send B1 to the feature fusion module, and the second encoder can send B2 to the feature fusion module.

[0160] The first encoder first generates multiple 2D features of the image, then uses camera intrinsic projection to map these 2D features into a discrete 3D space to obtain multiple corresponding 3D features, and then compresses these 3D features into the BEV features B1 of the image. The second encoder uses voxelization and sparse 3D convolution to process the point cloud, thereby generating the BEV features B2 of the point cloud.

[0161] 503. Based on the preset third BEV feature, perform feature fusion on the first BEV feature and / or the second BEV feature to obtain the feature fusion result.

[0162] After obtaining the first BEV feature and / or the second BEV feature, the target model can perform feature fusion on the first BEV feature and / or the second BEV feature based on the preset third BEV feature, thereby obtaining the feature fusion result.

[0163] Specifically, the feature fusion module of the target model can include multiple first fusion units and multiple second fusion units connected in series. The target model can then obtain the feature fusion result in the following way:

[0164] (1) For the first fusion unit, the input of the unit includes a preset third BEV feature, as well as the first BEV feature and / or the second BEV feature. For the second fusion unit, the input of the unit includes the output of the first fusion unit, as well as the first BEV feature and / or the second BEV feature, ..., and for the last fusion unit, the input of the unit includes the output of the penultimate fusion unit, as well as the first BEV feature and / or the second BEV feature. Then, the output of the last fusion unit can be sent to the first fusion unit.

[0165] (2) For the first second fusion unit, the input of this unit includes the output of the last first fusion unit. For the second second fusion unit, the input of this unit includes the output of the first second fusion unit, ..., and for the last second fusion unit, the input of this unit includes the output of the penultimate second fusion unit. Therefore, the output of the last second fusion unit can be used as the feature fusion result and sent to the prediction module.

[0166] As in the example above, such as Figure 7 As shown ( Figure 7 This is a schematic diagram of the feature fusion module provided in an embodiment of this application, and Figure 7 Is Figure 6 Based on the above (derived from the above), let the feature fusion module contain three first fusion units and two second fusion units connected in series. Then, the inputs of the first first fusion unit are B1, B2, and the preset BEV feature B3, and the output of the first first fusion unit is B7. The inputs of the second first fusion unit are B7, B2, and B3, and the output of the second first fusion unit is B... 11 The input to the third first fusion unit is B. 11 B2 and B3, and the output of the third first fusion unit is B. 15The input to the first second fusion unit is B. 15 And the output of the first second fusion unit is B 19 The input to the second fusion unit is B. 19 And the output of the second fusion unit is B. 23 That is, the feature fusion result, so the second fusion unit can convert B 23 Send to the prediction module.

[0167] More specifically, in the feature fusion module, each first fusion unit includes a cross-modal deformable attention layer, a first summation layer, a mixture of experts (MoE) layer, and a second summation layer. The first fusion unit can also obtain its corresponding output in the following ways (since all the first fusion units have the same structure, i.e., their workflows are similar, the following description uses the workflow of the first first fusion unit as an example):

[0168] (1.1) In the first fusion unit, if the features input to the first fusion unit include both first BEV features and second BEV features, the cross-modal deformable attention layer can process the first and second BEV features based on the third BEV feature using a deformable attention mechanism to obtain the fourth BEV feature. If the features input to the first fusion unit include only the first BEV feature, the cross-modal deformable attention layer can process the first BEV feature based on the third BEV feature using a deformable attention mechanism to obtain the fourth BEV feature. If the features input to the first fusion unit include only the second BEV feature, the cross-modal deformable attention layer can process the second BEV feature based on the third BEV feature using a deformable attention mechanism to obtain the fourth BEV feature. After obtaining the fourth BEV feature, the cross-modal deformable attention layer can send the fourth BEV feature to the first summing layer.

[0169] (1.2) After obtaining the fourth BEV feature, the first addition layer can add the third and fourth BEV features to obtain the fifth BEV feature. After obtaining the fifth BEV feature, the first addition layer can send the fifth BEV feature to the hybrid expert layer.

[0170] (1.3) After obtaining the fifth BEV feature, the hybrid expert layer can process the fifth BEV feature based on the hybrid expert model to obtain the sixth BEV feature. After obtaining the sixth BEV feature, the hybrid expert layer can send the sixth BEV feature to the second summing layer.

[0171] (1.4) After obtaining the sixth BEV feature, the second addition layer can add the fifth BEV feature and the sixth BEV feature to obtain the seventh BEV feature. After obtaining the seventh BEV feature, the second addition layer can send the seventh BEV feature as the output of the first fusion unit to the second fusion unit.

[0172] (1.5) Since the structure of the remaining first fusion units is the same as that of the first first fusion unit, the workflow of the remaining first fusion units can refer to the workflow of the first first fusion unit mentioned above, and will not be repeated here. Then, the last first fusion unit can send its output to the first second fusion unit.

[0173] As in the example above, such as Figure 8 As shown ( Figure 8 A schematic diagram of the structure of the first fusion unit provided in an embodiment of this application, and Figure 8 Is Figure 7 Based on the above (derived from the above), the first fusion unit comprises a cross-modal deformable attention layer, a first additive layer, a hybrid expert layer, and a second additive layer (the second and third fusion units are similar, and will not be described further here). Since the inputs of the first fusion unit are B1, B2, and B3, the cross-modal deformable attention layer can process B1 and B2 based on B3 using a deformable attention mechanism to obtain BEV feature B4, and send B4 to the first additive layer. The first additive layer can add B3 and B4 to obtain BEV feature B5, and send B5 to the hybrid expert layer. The hybrid expert layer can process B5 based on a hybrid expert model to obtain BEV feature B6, and send B6 to the second additive layer. The second additive layer can add B5 and B6 to obtain BEV feature B7, and send B7 to the second fusion unit.

[0174] Therefore, B7, which is input to the second first fusion unit, is the same as B3, which is input to the first first fusion unit. Thus, the second first fusion unit can ultimately obtain the BEV feature B. 11 and B 11 Send to the third first fusion unit. Input B to the third first fusion unit. 11 Just like B3, which is input to the first fusion unit, the third fusion unit can ultimately obtain the BEV feature B. 15 and B 15 Send to the first second fusion unit.

[0175] More specifically, in the feature fusion module, each second fusion unit includes a self-attention layer, a first summation layer, a hybrid expert layer, and a second summation layer. The second fusion unit can also obtain its corresponding output in the following ways (since all second fusion units have the same structure, i.e., their workflows are similar, the following description uses the workflow of the first second fusion unit as an example):

[0176] (2.1) In the first second fusion unit, the input of the first second fusion unit is the output of the last first fusion unit. The difference between the second fusion unit and the first fusion unit is that the second fusion unit contains a self-attention layer, while the first fusion unit contains a cross-modal deformable attention layer. Therefore, in the first second fusion unit, the self-attention layer can first process the output of the last first fusion unit based on the self-attention mechanism, and then give the processing result to the first additive layer. After being processed by the first additive layer, the hybrid expert layer and the second additive layer in sequence (this process can be referred to in (1.2) to (1.4) above, and will not be repeated here), the output of the first second fusion unit can be finally obtained and sent to the second second fusion unit.

[0177] (2.2) Since the structure of the remaining second fusion units is the same as that of the first second fusion unit, the workflow of the remaining second fusion units can refer to the workflow of the first second fusion unit, and will not be repeated here. Then, the output of the last second fusion unit can be used as the feature fusion result and sent to the prediction module.

[0178] As in the example above, such as Figure 9 As shown ( Figure 9 This is a schematic diagram of the structure of the second fusion unit provided in an embodiment of this application, and Figure 9 Is Figure 8 Based on the above, the first second fusion unit (obtained by drawing on the basis of B) contains a self-attention layer, a first additive layer, a hybrid expert layer, and a second additive layer (the second second fusion unit is similar, and will not be described further here). Since the input of the first second fusion unit is B... 15 Therefore, the self-attention layer can affect B. 15 By performing self-attention-based processing, the BEV feature B is obtained. 16 and B 16 Send to the first additive layer. The first additive layer can then convert B... 15 And B 16 Add them together to obtain the BEV feature B. 17 and B 17 Send to the hybrid expert layer. The hybrid expert layer can process B. 17The BEV feature B is obtained by performing processing based on a hybrid expert model. 18 and B 18 Send to the second additive layer. The second additive layer can then process B. 17 And B 18 Add them together to obtain the BEV feature B. 19 and B 19 Send to the second fusion unit.

[0179] So, the input B to the second fusion unit 19 As input to the first second fusion unit B 15 Therefore, the second fusion unit can ultimately obtain the BEV feature B. 23 and B 23 The result of feature fusion is sent to the prediction module.

[0180] More specifically, in each first fusion unit, the cross-modal deformable attention layer may include a first multilayer perceptron layer, a second multilayer perceptron layer, a selection layer, and a computation layer. Therefore, the first fusion unit can acquire the fourth BEV feature in several ways:

[0181] (1.1.1) In the cross-modal deformable attention layer of the first fusion unit, if the features input to the cross-modal deformable attention layer include both first BEV features and second BEV features, the first multilayer perceptron layer can process the third BEV feature based on the first multilayer perceptron to obtain the first position information of the first sub-feature of the first BEV feature and the second position information of the second sub-feature of the second BEV feature. Next, the selection layer can select one or more first sub-features from the first BEV features and one or more second sub-features from the second BEV features according to the indications of the first and second position information. Simultaneously, the second multilayer perceptron layer can also process the third BEV feature based on the second multilayer perceptron to obtain the first weight information of the first sub-feature and the second weight information of the second sub-feature. Finally, the computation layer can use the first and second weight information to perform a weighted sum of the first and second sub-features to obtain the fourth BEV feature.

[0182] (1.1.2) In the cross-modal deformable attention layer of the first fusion unit, if the only feature input to the cross-modal deformable attention layer is the first BEV feature, the first multilayer perceptron layer can process the third BEV feature based on the first multilayer perceptron to obtain the first position information of the first sub-feature of the first BEV feature. Then, the selection layer can select the first sub-feature from the first BEV feature according to the indication of the first position information. Simultaneously, the second multilayer perceptron layer can also process the third BEV feature based on the second multilayer perceptron to obtain the first weight information of the first sub-feature. Finally, the computation layer can use the first weight information to perform a weighted summation of the first sub-feature to obtain the fourth BEV feature.

[0183] (1.1.3) In the cross-modal deformable attention layer of the first fusion unit, if the only feature input to the cross-modal deformable attention layer is the second BEV feature, the first multilayer perceptron layer can process the third BEV feature based on the first multilayer perceptron to obtain the second position information of the second sub-feature of the second BEV feature. Then, the selection layer can select the second sub-feature from the second BEV feature according to the indication of the second position information. Simultaneously, the second multilayer perceptron layer can also process the third BEV feature based on the second multilayer perceptron to obtain the second weight information of the second sub-feature. Finally, the computation layer can use the second weight information to perform a weighted summation of the second sub-feature to obtain the fourth BEV feature.

[0184] As in the example above, such as Figure 10 As shown ( Figure 10 This is a schematic diagram of the cross-modal deformable attention layer of the first fusion unit provided in the embodiments of this application, and Figure 10 Is Figure 9 Based on the above (derived from the first fusion unit), the cross-modal deformable attention layer of the first fusion unit includes a first multilayer perceptron layer, a second multilayer perceptron layer, a selection layer, and a computation layer (the cross-modal deformable attention layers of the remaining first fusion units are similar, and will not be described again here). In the cross-modal deformable attention layer of the first fusion unit, since the input of the cross-modal deformable attention layer is B1, B2, and B3, the first multilayer perceptron layer can calculate B3 to obtain the position information of the sub-feature b1 of B1 and the position information of the sub-feature b2 of B2, and provide them to the selection layer. The selection layer can select b1 from B1 and b2 from B2 based on this position information, and provide them to the computation layer. At the same time, the second multilayer perceptron layer can calculate B3 to obtain the weight information of b1 and the weight information of b2, and provide them to the computation layer. Then, the computation layer can perform a weighted summation of b1 and b2 based on this weight information to obtain B4.

[0185] More specifically, in each first fusion unit, the hybrid expert layer can have various structures. For example, the hybrid expert layer may include a first expert model layer, a second expert model layer, and a third expert model layer; or it may include a first router layer, a second router layer, multiple fourth expert model layers, and an additive layer. It should be noted that the expert model layers involved here can be linear layers or convolutional layers, etc. Therefore, the first fusion unit can obtain the sixth BEV feature in the following ways:

[0186] (1.3.1) In the hybrid expert layer of the first fusion unit, the first expert model layer can process the fifth BEV feature based on the first expert model (e.g., linear computation or convolution processing, etc.) to obtain the eighth BEV feature, which is then sent to the third expert model layer. Simultaneously, the second expert model layer can process the fifth BEV feature based on the second expert model (e.g., linear computation or convolution processing, etc.) to obtain the ninth BEV feature, which is also sent to the third expert model layer. Finally, the third expert model layer can process the eighth and ninth BEV features based on the third expert model (e.g., linear computation or convolution processing, etc.) to obtain the sixth BEV feature.

[0187] As in the example above, such as Figure 11 As shown ( Figure 11 This is a schematic diagram of the hybrid expert layer of the first fusion unit provided in the embodiments of this application, and Figure 11 Is Figure 10 Based on the above (derived from the first fusion unit), the hybrid expert layer of the first fusion unit includes a first expert model layer, a second expert model layer, and a third expert model layer (the hybrid expert layers of the remaining first fusion units are similar, and will not be elaborated here). In the hybrid expert layer of the first fusion unit, the first expert model layer can process B5 based on the first expert model (expert) and send the processing result to the third expert model layer. Similarly, the second expert model layer can also process B5 based on the second expert model and send the processing result to the third expert model layer. Then, the third expert model layer can process the two processing results based on the third expert model to obtain B6.

[0188] (1.3.2) In the hybrid expert layer of the first fusion unit, the first router layer can divide the fifth BEV feature into multiple third sub-features and send each third sub-feature to a certain fourth expert model layer. Simultaneously, the second router layer can divide the fifth BEV feature into multiple fourth sub-features and send each fourth sub-feature to a certain fourth expert model layer. Since the number of fourth expert model layers is usually less than the number of third and fourth sub-features, a fourth expert model layer can receive at least one third sub-feature and at least one fourth sub-feature. Then, multiple fourth expert model layers can process the multiple third and fourth sub-features based on multiple fourth expert models to obtain multiple fifth sub-features, which are then sent to the addition layer. The number of fifth sub-features is usually the same as the number of fourth expert model layers. Finally, the addition layer can add the multiple fifth sub-features to obtain the sixth BEV feature.

[0189] As in the example above, such as Figure 12 As shown ( Figure 12 This is another schematic diagram of the hybrid expert layer of the first fusion unit provided in the embodiments of this application, and Figure 12 Is Figure 10 Based on the above (derived from the above), the hybrid expert layer of the first fusion unit includes a first router layer, a second router layer, multiple fourth expert model layers, and an addition layer (the hybrid expert layers of the remaining first fusion units are similar, and will not be elaborated here). In the hybrid expert layer of the first fusion unit, the first router layer can divide B5 into sub-features B 5，1 B 5，m Wait (m≥2), and put b 5，1 , ..., b 5，m Similarly, after being sent to multiple fourth-level expert model layers, the first router layer can also divide B5 into sub-features b. 5，m+1 , ..., b 5，m+k Wait (k≥2), and put b 5，m+1 , ..., b 5，m+k The data is sent to multiple fourth-expert model layers. These layers can then process the sub-features based on their respective fourth-expert models, resulting in sub-feature b. 6，1 , ..., b 6，n and put b 6，1 , ..., b 6，n Send to the addition layer. Then, the addition layer can process b. 6，1 , ..., b 6，n Add them together to get B6.

[0190] More specifically, each first fusion unit may further include a first normalization layer and a second normalization layer (similarly, each second fusion unit may also further include a first normalization layer and a second normalization layer). For example, in the first first fusion unit, the first normalization layer may normalize the third BEV feature to obtain a normalized third BEV feature, and normalize the first BEV feature and / or the second BEV feature to obtain a normalized first BEV feature and / or a normalized second BEV feature. Then, the cross-modal deformable attention layer may, based on the normalized third BEV feature, process the normalized first BEV feature and / or the normalized second BEV feature using a deformable attention mechanism to obtain a fourth BEV feature.

[0191] Furthermore, the second normalization layer can normalize the fifth BEV feature, thus obtaining the normalized fifth BEV feature. Then, the hybrid expert layer can process the normalized fifth BEV feature based on a hybrid expert model to obtain the sixth BEV feature.

[0192] As in the example above, such as Figure 13 As shown ( Figure 13 This is another structural schematic diagram of the first fusion unit provided in the embodiments of this application, and Figure 13 Is Figure 7 The first fusion unit (derived from the above) includes a first normalization layer, a cross-modal deformable attention layer, a first addition layer, a second normalization layer, a hybrid expert layer, and a second addition layer (the second and third fusion units are similar, and will not be described further here). Since the inputs of the first fusion unit are B1, B2, and B3, the first normalization layer can first normalize B1, B2, and B3 to obtain the normalized BEV features B`1, B`2, and B`3. The cross-modal deformable attention layer can process B`1 and B`2 based on B`3 using a deformable attention mechanism to obtain BEV feature B4, and send B4 to the first addition layer. The first addition layer can add B3 and B4 to obtain BEV feature B5, and send B5 to the second normalization layer. The second normalization layer can normalize B5 to obtain the normalized BEV feature B`5. The hybrid expert layer processes B'5 based on a hybrid expert model to obtain BEV feature B6, and sends B6 to the second addition layer. The second addition layer adds B5 and B6 to obtain BEV feature B7, and sends B7 to the second first fusion unit.

[0193] It should be understood that in this embodiment, the feature fusion module is only illustrated by including multiple first fusion units and multiple second fusion units. In practical applications, the feature fusion module may also include only one first fusion unit. In this case, the output of the first fusion unit can be directly used as the feature fusion result.

[0194] It should also be understood that in a hybrid expert model, the number of parallel expert model layers preceding the third expert model layer is the same as the number of prediction heads in the prediction module. This embodiment only uses two prediction heads (e.g., a detection head and a segmentation head) for illustrative purposes; therefore, there are two parallel expert model layers preceding the third expert model layer (i.e., the first expert model layer and the second expert model layer). However, in practical applications, there can be more or fewer of these parallel expert model layers, depending on the number of prediction heads.

[0195] It should also be understood that in a hybrid expert model, the number of parallel router layers is the same as the number of prediction heads in the prediction module. This embodiment only illustrates two prediction heads (e.g., a detection head and a segmentation head), so there are two parallel router layers (i.e., a first router layer and a second router layer). However, in practical applications, there may be more or fewer of these parallel router layers, depending on the number of prediction heads.

[0196] 504. Based on the feature fusion results, obtain the perception results of the target scene.

[0197] After obtaining the feature fusion result, the target model can further process the feature fusion structure to obtain the perception result of the target scene. This completes the scene perception for the target scene.

[0198] Specifically, the target model can obtain the perception results of the target scene in the following ways:

[0199] After obtaining the feature fusion results, each prediction head in the prediction module (such as the detection head and the segmentation head) can perform a series of processing on the feature fusion results to obtain various prediction results of the target scene (such as the object detection results and the object segmentation results of the target scene). These prediction results are the perception results of the target scene.

[0200] Furthermore, the target models provided in the embodiments of this application (e.g., MetaBEV-T and MetaBEV-C in the first column of Table 1) can be compared with models provided by related technologies whose input is only images (e.g., other models besides MetaBEV-T and MetaBEV-C in the first column, such as M...). 2The target models provided in this application's embodiments (e.g., MetaBEV-T and MetaBEV-C in the second column) can be compared with models provided by related technologies whose input is only point clouds (e.g., other models besides MetaBEV-T and MetaBEV-C in the second column, such as PointPillars and CenterPoint). The target models provided in this application's embodiments (e.g., MetaBEV-T and MetaBEV-MTL in the third column) can also be compared with models provided by related technologies whose input is both images and point clouds (e.g., other models besides MetaBEV-T and MetaBEV-MTL in the third column, such as PointPainting and MVP). The comparison results are shown in Table 1.

[0201] Table 1

[0202]

[0203] Furthermore, the target model provided in the embodiments of this application (e.g., MetaBEV in Table 2) can be compared with models provided by other related technologies (the input of these models can be a combination of images and point clouds, or only images, or only point clouds, for example, other models besides MetaBEV in Table 2, such as TransFusion, etc.), and the comparison results are shown in Table 2:

[0204] Table 2

[0205]

[0206] Furthermore, in cases where the camera or lidar is interfered with, the target model provided in the embodiments of this application (e.g., MetaBEV in Tables 3 to 7) can be compared with a model provided by another part of the related technologies (e.g., BEVFusion in Tables 3 to 7), and the comparison results are shown in Tables 3 to 7:

[0207] Table 3

[0208]

[0209] Table 4

[0210]

[0211] Table 5

[0212]

[0213] Table 6

[0214]

[0215] Table 7

[0216]

[0217] As can be seen from Tables 1 to 7, the performance of the target model provided in the embodiments of this application is superior to the performance of models provided by multiple related technologies.

[0218] In this embodiment, when scene perception of a target scene is required, image data and / or point cloud data of the target scene can be acquired first, and then input into the target model. Next, the target model can extract features from the image data and / or point cloud data to obtain a first BEV feature of the image data and / or a second BEV feature of the point cloud data. Then, the target model can use a preset third BEV feature to perform feature fusion on the first BEV feature and / or the second BEV feature to obtain a feature fusion result. Finally, the target model can further process the feature fusion result to obtain the perception result of the target scene. Thus, scene perception of the target scene is completed. In the aforementioned process, the target model can fuse the third BEV feature and the first BEV feature of the target scene image data to obtain a feature fusion result; it can also fuse the third BEV feature and the second BEV feature of the target scene point cloud data to obtain a feature fusion result; or it can fuse the third BEV feature, the first BEV feature of the target scene image data, and the second BEV feature of the target scene point cloud data to obtain a feature fusion result, thereby using the feature fusion result to obtain the scene perception result. Therefore, it can be seen that there is no strong coupling between the target scene image data and the target scene point cloud data input to the target model. Even if, due to certain special circumstances, the target model only obtains the target scene image data or only obtains the target scene point cloud data, the final perception result of the target scene obtained by the target model can still have high accuracy.

[0219] Furthermore, in this embodiment, the image data of the target scene input to the target model can be data collected by the camera under interference. Similarly, the point cloud data of the target scene input to the target model can also be data collected by the lidar under interference. It can be seen that the image data and point cloud data of the target scene input to the target model can be data of poor quality (data with some loss). The target model can still obtain a perception result of the target scene with sufficient accuracy based on these data.

[0220] The above is a detailed description of the scene perception method provided in the embodiments of this application. The following will introduce the model training method provided in the embodiments of this application. Figure 14 A schematic flowchart of the model training method provided in the embodiments of this application is shown below. Figure 14 As shown, the method includes:

[0221] 1401. Acquire image data and / or point cloud data of the target scene.

[0222] In this embodiment, when the model to be trained needs to be trained, a batch of training data can be obtained first. This batch of training data includes image data and / or point cloud data of the target scene. For the image data and / or point cloud data of the target scene, the actual perception result of the target scene is known.

[0223] In one possible implementation, the point cloud data is data acquired when the LiDAR is interfered with, and the image data is data acquired when the camera is interfered with.

[0224] For an explanation of step 1401, please refer to [link / reference]. Figure 5 The relevant descriptions of step 501 in the illustrated embodiment will not be repeated here.

[0225] 1402. The image data and / or point cloud data are processed by the model to be trained to obtain the perception result of the target scene. The model to be trained is used to: extract features from the image data and / or point cloud data to obtain the first BEV feature of the image data and / or the second BEV feature of the point cloud data; perform feature fusion on the first BEV feature and / or the second BEV feature based on the preset third BEV feature to obtain the feature fusion result; and obtain the perception result of the target scene based on the feature fusion result.

[0226] After obtaining the image data and / or point cloud data of the target scene, the image data and / or point cloud data of the target scene can be input into the model to be trained. Therefore, the model to be trained can first extract features from the image data and / or point cloud data of the target scene to obtain the first BEV feature of the image data and / or the second BEV feature of the point cloud data. Next, the model to be trained can perform feature fusion on the first BEV feature and / or the second BEV feature based on a preset third BEV feature to obtain the feature fusion result. Finally, the model to be trained can obtain the (predicted) perception result of the target scene based on the feature fusion result.

[0227] In one possible implementation, the model to be trained is used to: encode image data and / or point cloud data to obtain a first BEV feature of the image data and / or a second BEV feature of the point cloud data.

[0228] In one possible implementation, the model to be trained is used to: process the first BEV feature and / or the second BEV feature based on a pre-set third BEV feature using a deformable attention mechanism to obtain a fourth BEV feature; add the third BEV feature and the fourth BEV feature to obtain a fifth BEV feature; process the fifth BEV feature based on a hybrid expert model to obtain a sixth BEV feature; add the fifth BEV feature and the sixth BEV feature to obtain a seventh BEV feature; and obtain the feature fusion result based on the seventh BEV feature.

[0229] In one possible implementation, the model to be trained is used to: process a pre-set third BEV feature using a first multilayer perceptron to obtain first position information of a first sub-feature of the first BEV feature and / or second position information of a second sub-feature of the second BEV feature; based on the first position information and / or the second position information, obtain a first sub-feature and / or a second sub-feature from the first BEV feature and / or the second BEV feature; process the third BEV feature using a second multilayer perceptron to obtain first weight information of the first sub-feature of the first BEV feature and / or second weight information of the second sub-feature of the second BEV feature; and based on the first weight information and / or the second weight information, perform a weighted summation of the first sub-feature and / or the second sub-feature to obtain a fourth BEV feature.

[0230] In one possible implementation, the model to be trained is used to: process the fifth BEV feature based on a first expert model to obtain the eighth BEV feature; process the fifth BEV feature based on a second expert model to obtain the ninth BEV feature; and process the eighth and ninth BEV features based on a third expert model to obtain the sixth BEV feature.

[0231] In one possible implementation, the model to be trained is used to: divide the fifth BEV feature into multiple third sub-features; divide the fifth BEV feature into multiple fourth sub-features; process the multiple third sub-features and multiple fourth sub-features based on multiple fourth expert models to obtain multiple fifth sub-features, wherein the processing based on one fourth expert model corresponds to at least one third sub-feature and at least one fourth sub-feature; and add the multiple fifth sub-features to obtain the sixth BEV feature.

[0232] 1403. Based on the perception results of the target scene, train the model to be trained to obtain the target model.

[0233] After obtaining the perceived results of the target scene, since the actual perceived results of the target scene are known, a pre-defined loss function can be used to calculate both the perceived results and the actual perceived results of the target scene, thus obtaining the target loss. The target loss indicates the difference between the perceived results and the actual perceived results of the target scene. Next, based on the target loss, the parameters of the model to be trained can be updated, resulting in the updated model to be trained. Then, the updated model can be trained using the next batch of training data until the model training conditions are met (e.g., the target loss converges, etc.), thus obtaining the target model. Figure 5 The target model in the illustrated embodiment.

[0234] The target model trained in this embodiment of the application has scene perception capabilities. Specifically, when scene perception of a target scene is required, image data and / or point cloud data of the target scene can be acquired first, and then input into the target model. Next, the target model can extract features from the image data and / or point cloud data to obtain a first BEV feature of the image data and / or a second BEV feature of the point cloud data. Then, the target model can use a preset third BEV feature to perform feature fusion on the first BEV feature and / or the second BEV feature to obtain a feature fusion result. Finally, the target model can further process the feature fusion result to obtain the perception result of the target scene. Thus, scene perception for the target scene is completed. In the aforementioned process, the target model can fuse the third BEV feature and the first BEV feature of the target scene image data to obtain a feature fusion result; it can also fuse the third BEV feature and the second BEV feature of the target scene point cloud data to obtain a feature fusion result; or it can fuse the third BEV feature, the first BEV feature of the target scene image data, and the second BEV feature of the target scene point cloud data to obtain a feature fusion result, thereby using the feature fusion result to obtain the scene perception result. Therefore, it can be seen that there is no strong coupling between the target scene image data and the target scene point cloud data input to the target model. Even if, due to certain special circumstances, the target model only obtains the target scene image data or only obtains the target scene point cloud data, the final perception result of the target scene obtained by the target model can still have high accuracy.

[0235] The above is a detailed description of the scene perception method and model training method provided in the embodiments of this application. The scene perception device and model training device provided in the embodiments of this application will be described below. Figure 15 A schematic diagram of the scene perception device provided in the embodiments of this application is shown below. Figure 15 As shown, the device includes:

[0236] The first acquisition module 1501 is used to acquire image data and / or point cloud data of the target scene;

[0237] Feature extraction module 1502 is used to extract features from image data and / or point cloud data to obtain first BEV features of image data and / or second BEV features of point cloud data.

[0238] The feature fusion module 1503 is used to perform feature fusion on the first BEV feature and / or the second BEV feature based on the preset third BEV feature to obtain the feature fusion result;

[0239] The second acquisition module 1504 is used to acquire the perception results of the target scene based on the feature fusion results.

[0240] In this embodiment, when scene perception of a target scene is required, image data and / or point cloud data of the target scene can be acquired first, and then input into the target model. Next, the target model can extract features from the image data and / or point cloud data to obtain a first BEV feature of the image data and / or a second BEV feature of the point cloud data. Then, the target model can use a preset third BEV feature to perform feature fusion on the first BEV feature and / or the second BEV feature to obtain a feature fusion result. Finally, the target model can further process the feature fusion result to obtain the perception result of the target scene. Thus, scene perception of the target scene is completed. In the aforementioned process, the target model can fuse the third BEV feature and the first BEV feature of the target scene image data to obtain a feature fusion result; it can also fuse the third BEV feature and the second BEV feature of the target scene point cloud data to obtain a feature fusion result; or it can fuse the third BEV feature, the first BEV feature of the target scene image data, and the second BEV feature of the target scene point cloud data to obtain a feature fusion result, thereby using the feature fusion result to obtain the scene perception result. Therefore, it can be seen that there is no strong coupling between the target scene image data and the target scene point cloud data input to the target model. Even if, due to certain special circumstances, the target model only obtains the target scene image data or only obtains the target scene point cloud data, the final perception result of the target scene obtained by the target model can still have high accuracy.

[0241] In one possible implementation, the feature extraction module 1502 is used to encode image data and / or point cloud data to obtain a first BEV feature of the image data and / or a second BEV feature of the point cloud data.

[0242] In one possible implementation, the feature fusion module 1503 is used to: process the first BEV feature and / or the second BEV feature based on a pre-set third BEV feature using a deformable attention mechanism to obtain a fourth BEV feature; add the third BEV feature and the fourth BEV feature to obtain a fifth BEV feature; process the fifth BEV feature based on a hybrid expert model to obtain a sixth BEV feature; add the fifth BEV feature and the sixth BEV feature to obtain a seventh BEV feature; and obtain the feature fusion result based on the seventh BEV feature.

[0243] In one possible implementation, the feature fusion module 1503 is configured to: process a preset third BEV feature based on a first multilayer perceptron to obtain first position information of a first sub-feature of the first BEV feature and / or second position information of a second sub-feature of the second BEV feature; based on the first position information and / or the second position information, obtain a first sub-feature and / or a second sub-feature from the first BEV feature and / or the second BEV feature; process the third BEV feature based on a second multilayer perceptron to obtain first weight information of the first sub-feature of the first BEV feature and / or second weight information of the second sub-feature of the second BEV feature; and based on the first weight information and / or the second weight information, perform a weighted summation of the first sub-feature and / or the second sub-feature to obtain a fourth BEV feature.

[0244] In one possible implementation, the feature fusion module 1503 is used to: process the fifth BEV feature based on a first expert model to obtain the eighth BEV feature; process the fifth BEV feature based on a second expert model to obtain the ninth BEV feature; and process the eighth and ninth BEV features based on a third expert model to obtain the sixth BEV feature.

[0245] In one possible implementation, the feature fusion module 1503 is used to: divide the fifth BEV feature into multiple third sub-features; divide the fifth BEV feature into multiple fourth sub-features; process the multiple third sub-features and multiple fourth sub-features based on multiple fourth expert models to obtain multiple fifth sub-features, wherein the processing based on one fourth expert model corresponds to at least one third sub-feature and at least one fourth sub-feature; and add the multiple fifth sub-features to obtain the sixth BEV feature.

[0246] In one possible implementation, the point cloud data is data acquired when the LiDAR is interfered with, and the image data is data acquired when the camera is interfered with.

[0247] Figure 16 A schematic diagram of the model training apparatus provided in the embodiments of this application is shown below. Figure 16 As shown, the device includes:

[0248] The acquisition module 1601 is used to acquire image data and / or point cloud data of the target scene;

[0249] The processing module 1602 is used to process image data and / or point cloud data through a model to be trained to obtain the perception result of the target scene. The model to be trained is used to: extract features from the image data and / or point cloud data to obtain a first BEV feature of the image data and / or a second BEV feature of the point cloud data; perform feature fusion on the first BEV feature and / or the second BEV feature based on a preset third BEV feature to obtain a feature fusion result; and obtain the perception result of the target scene based on the feature fusion result.

[0250] Training module 1603 is used to train the model to be trained based on the perception results of the target scene, so as to obtain the target model.

[0251] The target model trained in this embodiment of the application has scene perception capabilities. Specifically, when scene perception of a target scene is required, image data and / or point cloud data of the target scene can be acquired first, and then input into the target model. Next, the target model can extract features from the image data and / or point cloud data to obtain a first BEV feature of the image data and / or a second BEV feature of the point cloud data. Then, the target model can use a preset third BEV feature to perform feature fusion on the first BEV feature and / or the second BEV feature to obtain a feature fusion result. Finally, the target model can further process the feature fusion result to obtain the perception result of the target scene. Thus, scene perception for the target scene is completed. In the aforementioned process, the target model can fuse the third BEV feature and the first BEV feature of the target scene image data to obtain a feature fusion result; it can also fuse the third BEV feature and the second BEV feature of the target scene point cloud data to obtain a feature fusion result; or it can fuse the third BEV feature, the first BEV feature of the target scene image data, and the second BEV feature of the target scene point cloud data to obtain a feature fusion result, thereby using the feature fusion result to obtain the scene perception result. Therefore, it can be seen that there is no strong coupling between the target scene image data and the target scene point cloud data input to the target model. Even if, due to certain special circumstances, the target model only obtains the target scene image data or only obtains the target scene point cloud data, the final perception result of the target scene obtained by the target model can still have high accuracy.

[0252] In one possible implementation, the model to be trained is used to encode image data and / or point cloud data to obtain a first BEV feature of the image data and / or a second BEV feature of the point cloud data.

[0253] In one possible implementation, the model to be trained is used to: process the first BEV feature and / or the second BEV feature based on a pre-set third BEV feature using a deformable attention mechanism to obtain a fourth BEV feature; add the third BEV feature and the fourth BEV feature to obtain a fifth BEV feature; process the fifth BEV feature based on a hybrid expert model to obtain a sixth BEV feature; add the fifth BEV feature and the sixth BEV feature to obtain a seventh BEV feature; and obtain the feature fusion result based on the seventh BEV feature.

[0254] In one possible implementation, the model to be trained is used to: process a pre-set third BEV feature using a first multilayer perceptron to obtain first position information of a first sub-feature of the first BEV feature and / or second position information of a second sub-feature of the second BEV feature; based on the first position information and / or the second position information, obtain a first sub-feature and / or a second sub-feature from the first BEV feature and / or the second BEV feature; process the third BEV feature using a second multilayer perceptron to obtain first weight information of the first sub-feature of the first BEV feature and / or second weight information of the second sub-feature of the second BEV feature; and based on the first weight information and / or the second weight information, perform a weighted summation of the first sub-feature and / or the second sub-feature to obtain a fourth BEV feature.

[0255] In one possible implementation, the model to be trained is used to: process the fifth BEV feature based on a first expert model to obtain the eighth BEV feature; process the fifth BEV feature based on a second expert model to obtain the ninth BEV feature; and process the eighth and ninth BEV features based on a third expert model to obtain the sixth BEV feature.

[0256] In one possible implementation, the model to be trained is used to: divide the fifth BEV feature into multiple third sub-features; divide the fifth BEV feature into multiple fourth sub-features; process the multiple third sub-features and multiple fourth sub-features based on multiple fourth expert models to obtain multiple fifth sub-features, wherein the processing based on one fourth expert model corresponds to at least one third sub-feature and at least one fourth sub-feature; and add the multiple fifth sub-features to obtain the sixth BEV feature.

[0257] In one possible implementation, the point cloud data is data acquired when the LiDAR is interfered with, and the image data is data acquired when the camera is interfered with.

[0258] It should be noted that the information interaction and execution process between the modules / units of the above-mentioned device are based on the same concept as the method embodiment of this application, and the resulting technical effects are the same as those of the method embodiment of this application. For details, please refer to the description in the method embodiment shown above in the embodiment of this application, and it will not be repeated here.

[0259] This application also relates to an execution device. Figure 17 This is a schematic diagram of the execution device provided in an embodiment of this application. Figure 17 As shown, the execution device 1700 can specifically manifest as a mobile phone, tablet, laptop, in-vehicle computer, smart wearable device, server, etc., without limitation here. Among them, the execution device 1700 can be deployed with... Figure 15 The scene perception device described in the corresponding embodiment is used to implement Figure 5 The scene perception function corresponds to the embodiment. Specifically, the execution device 1700 includes: a receiver 1701, a transmitter 1702, a processor 1703, and a memory 1704 (wherein the execution device 1700 may have one or more processors 1703). Figure 17 (Taking a processor as an example), processor 1703 may include application processor 17031 and communication processor 17032. In some embodiments of this application, receiver 1701, transmitter 1702, processor 1703 and memory 1704 may be connected via a bus or other means.

[0260] Memory 1704 may include read-only memory and random access memory, and provides instructions and data to processor 1703. A portion of memory 1704 may also include non-volatile random access memory (NVRAM). Memory 1704 stores processor and operation instructions, executable modules, or data structures, or subsets thereof, or extended sets thereof, wherein the operation instructions may include various operation instructions for implementing various operations.

[0261] Processor 1703 controls the operation of the execution device. In specific applications, the various components of the execution device are coupled together through a bus system, which may include not only the data bus, but also power buses, control buses, and status signal buses. However, for clarity, all buses are referred to as the bus system in the diagram.

[0262] The methods disclosed in the embodiments of this application can be applied to or implemented by processor 1703. Processor 1703 can be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above methods can be completed by integrated logic circuits in the hardware of processor 1703 or by instructions in software form. Processor 1703 can be a general-purpose processor, a digital signal processor (DSP), a microprocessor, or a microcontroller, and may further include application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. Processor 1703 can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. A general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly implemented by a hardware decoding processor, or implemented by a combination of hardware and software modules in the decoding processor. The software module can reside in a mature storage medium in the field, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory 1704. Processor 1703 reads the information in memory 1704 and, in conjunction with its hardware, completes the steps of the above method.

[0263] Receiver 1701 can be used to receive input digital or character information, and to generate signal inputs related to the settings and function control of the execution device. Transmitter 1702 can be used to output digital or character information through the first interface; transmitter 1702 can also be used to send instructions to the disk group through the first interface to modify the data in the disk group; transmitter 1702 may also include a display device such as a display screen.

[0264] In one embodiment of this application, the processor 1703 is used to... Figure 5 The target model in the corresponding embodiment is used to obtain the perception results of the target scene.

[0265] This application also relates to a training device. Figure 18 This is a schematic diagram of the structure of a training device provided in an embodiment of this application. Figure 18As shown, the training device 1800 is implemented by one or more servers. The training device 1800 can vary significantly due to different configurations or performance. It may include one or more central processing units (CPUs) 1818 (e.g., one or more processors) and memory 1832, and one or more storage media 1830 (e.g., one or more mass storage devices) for storing application programs 1842 or data 1844. The memory 1832 and storage media 1830 can be temporary or persistent storage. The program stored in the storage media 1830 may include one or more modules (not shown in the figure), each module may include a series of instruction operations on the training device. Furthermore, the CPU 1818 may be configured to communicate with the storage media 1830 and execute the series of instruction operations in the storage media 1830 on the training device 1800.

[0266] The training device 1800 may also include one or more power supplies 1826, one or more wired or wireless network interfaces 1850, one or more input / output interfaces 1858; or, one or more operating systems 1841, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0267] Specifically, the training equipment can perform Figure 14 The model training method in the corresponding embodiment.

[0268] This application also relates to a computer storage medium storing a program for signal processing, which, when run on a computer, causes the computer to perform steps as performed by the aforementioned execution device, or causes the computer to perform steps as performed by the aforementioned training device.

[0269] This application also relates to a computer program product that stores instructions that, when executed by a computer, cause the computer to perform steps as performed by the aforementioned execution device, or to perform steps as performed by the aforementioned training device.

[0270] The execution device, training device, or terminal device provided in this application embodiment can specifically be a chip. The chip includes a processing unit and a communication unit. The processing unit can be, for example, a processor, and the communication unit can be, for example, an input / output interface, pins, or circuits. The processing unit can execute computer execution instructions stored in the storage unit to cause the chip within the execution device to execute the data processing method described in the above embodiments, or to cause the chip within the training device to execute the data processing method described in the above embodiments. Optionally, the storage unit can be a storage unit within the chip, such as a register or cache. Alternatively, the storage unit can be a storage unit located outside the chip within the wireless access device, such as a read-only memory (ROM) or other types of static storage devices capable of storing static information and instructions, such as random access memory (RAM).

[0271] For details, please refer to Figure 19 , Figure 19 This is a schematic diagram of the chip provided in an embodiment of this application. The chip can be represented as a neural network processor (NPU) 1900. The NPU 1900 is mounted as a coprocessor on the host CPU, and tasks are assigned by the host CPU. The core part of the NPU is the arithmetic circuit 1903, which is controlled by the controller 1904 to extract matrix data from the memory and perform multiplication operations.

[0272] In some implementations, the arithmetic circuit 1903 internally includes multiple processing engines (PEs). In some implementations, the arithmetic circuit 1903 is a two-dimensional pulsating array. The arithmetic circuit 1903 can also be a one-dimensional pulsating array or other electronic circuitry capable of performing mathematical operations such as multiplication and addition. In some implementations, the arithmetic circuit 1903 is a general-purpose matrix processor.

[0273] For example, suppose we have an input matrix A, a weight matrix B, and an output matrix C. The arithmetic circuit retrieves the corresponding data of matrix B from the weight memory 1902 and caches it in each PE of the arithmetic circuit. The arithmetic circuit retrieves the data of matrix A from the input memory 1901 and performs matrix operations with matrix B. The partial result or the final result of the obtained matrix is ​​stored in the accumulator 1908.

[0274] Unified memory 1906 is used to store input and output data. Weight data is directly transferred to weight memory 1902 via Direct Memory Access Controller (DMAC) 1905. Input data is also transferred to unified memory 1906 via DMAC.

[0275] BIU stands for Bus Interface Unit, which is used for interaction between the AXI bus and the DMAC and the Instruction Fetch Buffer (IFB) 1909.

[0276] The Bus Interface Unit (BIU) 1913 is used by the instruction fetch memory 1909 to fetch instructions from external memory, and also by the memory access controller 1905 to fetch the original data of the input matrix A or the weight matrix B from external memory.

[0277] The DMAC is mainly used to move input data from external memory DDR to unified memory 1906, or to weight data to weight memory 1902, or to input data to input memory 1901.

[0278] The vector computation unit 1907 includes multiple processing units that further process the output of the computation circuit 1903 when necessary, such as vector multiplication, vector addition, exponential operations, logarithmic operations, size comparisons, etc. It is mainly used for computation in non-convolutional / fully connected layers of neural networks, such as batch normalization, pixel-level summation, and upsampling of the predicted label plane.

[0279] In some implementations, the vector computation unit 1907 can store the processed output vector in the unified memory 1906. For example, the vector computation unit 1907 can apply a linear function, or a nonlinear function, to the output of the computation circuit 1903, such as linearly interpolating the predicted label plane extracted from the convolutional layer, or, for example, accumulating a vector of values ​​to generate activation values. In some implementations, the vector computation unit 1907 generates normalized values, pixel-level summed values, or both. In some implementations, the processed output vector can be used as activation input to the computation circuit 1903, for example, for use in subsequent layers of the neural network.

[0280] The instruction fetch buffer 1909 connected to the controller 1904 is used to store the instructions used by the controller 1904;

[0281] Unified memory 1906, input memory 1901, weighted memory 1902, and instruction fetch memory 1909 are all on-chip memories. External memory is proprietary to this NPU hardware architecture.

[0282] The processor mentioned above can be a general-purpose central processing unit, a microprocessor, an ASIC, or one or more integrated circuits used to control the execution of the above program.

[0283] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the device embodiment drawings provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0284] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, training equipment, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0285] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0286] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, training device, or data center to another website, computer, training device, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device or data center that integrates one or more available media. The available medium may be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

Claims

1. A scene perception method, characterized in that, The method is implemented through a target model, and the method includes: Acquire image data or point cloud data of the target scene; Feature extraction is performed on the image data or the point cloud data to obtain the first bird's-eye view BEV feature of the image data or the second BEV feature of the point cloud data. Based on the preset third BEV feature, the first BEV feature or the second BEV feature is processed based on the deformable attention mechanism to obtain the fourth BEV feature. The third BEV feature and the fourth BEV feature are added together to obtain the fifth BEV feature; The fifth BEV feature is processed using a hybrid expert model to obtain the sixth BEV feature; The fifth BEV feature and the sixth BEV feature are added together to obtain the seventh BEV feature; Based on the seventh BEV feature, the feature fusion result is obtained; Based on the feature fusion results, the perception results of the target scene are obtained.

2. The method according to claim 1, characterized in that, The step of extracting features from the image data or the point cloud data to obtain the first BEV feature of the image data or the second BEV feature of the point cloud data includes: The image data or the point cloud data is encoded to obtain a first BEV feature of the image data or a second BEV feature of the point cloud data.

3. The method according to claim 1, characterized in that, The process of processing the first BEV feature or the second BEV feature based on a deformable attention mechanism, based on a preset third BEV feature, to obtain the fourth BEV feature includes: The preset third BEV feature is processed based on the first multilayer perceptron to obtain the first position information of the first sub-feature of the first BEV feature or the second position information of the second sub-feature of the second BEV feature. Based on the first location information or the second location information, obtain the first sub-feature or the second sub-feature from the first BEV feature or the second BEV feature; The third BEV feature is processed based on a second multilayer perceptron to obtain the first weight information of the first sub-feature of the first BEV feature or the second weight information of the second sub-feature of the second BEV feature. Based on the first weight information or the second weight information, the first sub-feature or the second sub-feature is weighted and summed to obtain the fourth BEV feature.

4. The method according to claim 3, characterized in that, The process of processing the fifth BEV feature based on a hybrid expert model to obtain the sixth BEV feature includes: The fifth BEV feature is processed based on the first expert model to obtain the eighth BEV feature; The fifth BEV feature is processed based on the second expert model to obtain the ninth BEV feature; The eighth BEV feature and the ninth BEV feature are processed based on a third expert model to obtain the sixth BEV feature.

5. The method according to claim 3, characterized in that, The process of processing the fifth BEV feature based on a hybrid expert model to obtain the sixth BEV feature includes: The fifth BEV feature is divided into multiple third sub-features; The fifth BEV feature is divided into multiple fourth sub-features; The plurality of third sub-features and the plurality of fourth sub-features are processed based on a plurality of fourth expert models to obtain a plurality of fifth sub-features, wherein the processing based on a fourth expert model corresponds to at least one third sub-feature and at least one fourth sub-feature. The sixth BEV feature is obtained by adding the multiple fifth sub-features together.

6. The method according to any one of claims 1 to 5, characterized in that, The point cloud data is data collected when the lidar is interfered with, and the image data is data collected when the camera is interfered with.

7. A model training method, characterized in that, The method includes: Acquire image data or point cloud data of the target scene; The image data or point cloud data is processed by a model to be trained to obtain a perception result of the target scene. The model to be trained is used to: extract features from the image data or point cloud data to obtain a first BEV feature of the image data or a second BEV feature of the point cloud data; based on a preset third BEV feature, process the first BEV feature or the second BEV feature using a deformable attention mechanism to obtain a fourth BEV feature; add the third BEV feature and the fourth BEV feature to obtain a fifth BEV feature; process the fifth BEV feature using a hybrid expert model to obtain a sixth BEV feature; add the fifth BEV feature and the sixth BEV feature to obtain a seventh BEV feature; obtain a feature fusion result based on the seventh BEV feature; and obtain a perception result of the target scene based on the feature fusion result. Based on the perception results of the target scene, the model to be trained is trained to obtain the target model.

8. The method according to claim 7, characterized in that, The model to be trained is used for: The image data or the point cloud data is encoded to obtain a first BEV feature of the image data or a second BEV feature of the point cloud data.

9. The method according to claim 7, characterized in that, The model to be trained is used for: The preset third BEV feature is processed based on the first multilayer perceptron to obtain the first position information of the first sub-feature of the first BEV feature or the second position information of the second sub-feature of the second BEV feature. Based on the first location information or the second location information, obtain the first sub-feature or the second sub-feature from the first BEV feature or the second BEV feature; The third BEV feature is processed based on a second multilayer perceptron to obtain the first weight information of the first sub-feature of the first BEV feature or the second weight information of the second sub-feature of the second BEV feature. Based on the first weight information or the second weight information, the first sub-feature or the second sub-feature is weighted and summed to obtain the fourth BEV feature.

10. The method according to claim 9, characterized in that, The model to be trained is used for: The fifth BEV feature is processed based on the first expert model to obtain the eighth BEV feature; The fifth BEV feature is processed based on the second expert model to obtain the ninth BEV feature; The eighth BEV feature and the ninth BEV feature are processed based on a third expert model to obtain the sixth BEV feature.

11. The method according to claim 9, characterized in that, The model to be trained is used for: The fifth BEV feature is divided into multiple third sub-features; The fifth BEV feature is divided into multiple fourth sub-features; The plurality of third sub-features and the plurality of fourth sub-features are processed based on a plurality of fourth expert models to obtain a plurality of fifth sub-features, wherein the processing based on a fourth expert model corresponds to at least one third sub-feature and at least one fourth sub-feature. The sixth BEV feature is obtained by adding the multiple fifth sub-features together.

12. The method according to any one of claims 7 to 11, characterized in that, The point cloud data is data collected when the lidar is interfered with, and the image data is data collected when the camera is interfered with.

13. A scene perception device, characterized in that, The device includes a target model, and the device comprises: The first acquisition module is used to acquire image data of the target scene or point cloud data of the target scene; The feature extraction module is used to extract features from the image data or the point cloud data to obtain the first BEV feature of the image data or the second BEV feature of the point cloud data. The feature fusion module is used to: process the first BEV feature or the second BEV feature based on a pre-set third BEV feature using a deformable attention mechanism to obtain a fourth BEV feature; add the third BEV feature and the fourth BEV feature to obtain a fifth BEV feature; process the fifth BEV feature based on a hybrid expert model to obtain a sixth BEV feature; add the fifth BEV feature and the sixth BEV feature to obtain a seventh BEV feature; and obtain the feature fusion result based on the seventh BEV feature. The second acquisition module is used to acquire the perception result of the target scene based on the feature fusion result.

14. A model training device, characterized in that, The device includes: The acquisition module is used to acquire image data of the target scene or point cloud data of the target scene; The processing module is used to process the image data or the point cloud data using a model to be trained, to obtain the perception result of the target scene. The model to be trained is used to: extract features from the image data or the point cloud data to obtain a first BEV feature of the image data or a second BEV feature of the point cloud data; based on a preset third BEV feature, process the first BEV feature or the second BEV feature using a deformable attention mechanism to obtain a fourth BEV feature; add the third BEV feature and the fourth BEV feature to obtain a fifth BEV feature; process the fifth BEV feature using a hybrid expert model to obtain a sixth BEV feature; add the fifth BEV feature and the sixth BEV feature to obtain a seventh BEV feature; obtain a feature fusion result based on the seventh BEV feature; and obtain the perception result of the target scene based on the feature fusion result. The training module is used to train the model to be trained based on the perception results of the target scene to obtain the target model.

15. A scene perception device, characterized in that, The device includes a memory and a processor; the memory stores code, and the processor is configured to execute the code, wherein when the code is executed, the scene perception device performs the method as described in any one of claims 1 to 12.

16. A computer storage medium, characterized in that, The computer storage medium stores one or more instructions that, when executed by one or more computers, cause the one or more computers to perform the method of any one of claims 1 to 12.

17. A computer program product, characterized in that, The computer program product stores instructions that, when executed by a computer, cause the computer to perform the method described in any one of claims 1 to 12.

Images