A multimodal model pre-training method, device, equipment and storage medium

By performing feature extraction and fusion processing on multimodal models, the problem of complementarity between modals in the prior art is solved, and the processing effect and information modeling capabilities of multimodal models in downstream tasks are improved.

CN118823332BActive Publication Date: 2024-12-03HANGZHOU HIKVISION DIGITAL TECHNOLOGY CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411318858.2
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-09-20
Publication Date
2024-12-03
Estimated Expiration
2044-09-20

AI Technical Summary

Technical Problem

The existing multimodal model ignores the complementarity between modes during pre-training, resulting in poorer results in the encoder processing downstream tasks.

Method used

By acquiring the sample point cloud and images of the same scene, voxelization and masking are performed, feature extraction and feature fusion are used using the encoder and decoder in the multimodal model, and the model parameters are adjusted to make full use of the point cloud and images information.

Benefits of technology

The processing effect of multimodal models in downstream tasks is improved, the modeling ability of point cloud and image information is enhanced, and the dependence on labeled data is reduced.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118823332B_ABST
    Figure CN118823332B_ABST
Patent Text Reader

Abstract

The embodiments of the present application provide a multimodal model pre-training method, device, equipment and storage medium, which relate to the field of deep learning technology. The specific implementation scheme is: masking the sample point cloud and the sample image respectively to obtain the first mask point cloud and the first mask image; extracting the features of the first mask point cloud to obtain the first BEV feature, and extracting the features of the first mask image to obtain the first image feature; converting the first image feature to the BEV perspective and fusing it with the first BEV feature to obtain the second BEV feature; decoding the second BEV feature and the second image feature respectively to obtain the first prediction result, the second prediction result and the third prediction result; adjusting the model parameters based on the differences between the first prediction result, the second prediction result and the third prediction result and the corresponding true value. It can be seen that through this scheme, the processing effect of the encoder in the trained multimodal model in processing downstream tasks can be improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of deep learning technology, and in particular to a multimodal model pre-training method, apparatus, device and storage medium. Background Art

[0002] Multimodal models can process and integrate data from different data types or information sources to obtain a more comprehensive and accurate understanding of information from various modalities. Currently, most multimodal models need to be pre-trained using a large amount of unlabeled multimodal data to enable the model to acquire certain multimodal information extraction and modeling capabilities.

[0003] In the related art, the encoder used to process unimodal data is pre-trained by constructing multiple unimodal reconstruction tasks. However, the encoder trained by this pre-training method ignores the complementarity of information between modalities, resulting in poor processing effect of the trained encoder when processing downstream tasks.

[0004] Therefore, how to improve the processing effect of the encoder in the trained multimodal model in handling downstream tasks has become a technical problem that needs to be solved urgently. Summary of the invention

[0005] The purpose of the embodiments of the present application is to provide a multimodal model pre-training method, apparatus, device and storage medium to improve the processing effect of the encoder in the trained multimodal model in processing downstream tasks. The specific technical solution is as follows:

[0006] In a first aspect, an embodiment of the present application provides a multimodal model pre-training method, the method comprising:

[0007] Obtaining sample point clouds and sample images corresponding to the same scene;

[0008] voxelize the sample point cloud to obtain a plurality of voxels, perform point cloud masking on some real points in the sample point cloud according to the bird's-eye view grid to which each voxel belongs to obtain a first mask point cloud, and perform image masking on the sample image to obtain a first mask image;

[0009] Using a first encoder in a multimodal model to perform feature extraction on the first mask point cloud to obtain features from a bird's-eye view (BEV) as first BEV features, and using a second encoder in the multimodal model to perform feature extraction on the first mask image to obtain first image features; wherein the multimodal model further includes a first converter, a first fusion network, a first decoder, and a second decoder;

[0010] Using the first converter, converting the first image feature to the BEV perspective to obtain a first conversion feature;

[0011] Using the first fusion network to perform feature fusion on the first conversion feature and the first BEV feature to obtain a second BEV feature;

[0012] The second BEV feature is decoded by the first decoder to obtain a first prediction result and a second prediction result, and the second image feature is decoded by the second decoder to obtain a third prediction result; wherein the first prediction result represents: the predicted number of prediction points contained in the bird's-eye view grid and the predicted coordinates of each prediction point, the second prediction result represents: the predicted RGB value of each prediction point in the bird's-eye view grid, and the third prediction result represents: the predicted RGB value of each pixel in the sample image; the second image feature is obtained based on the first image feature;

[0013] Based on the difference between the first prediction result and the corresponding true value, the difference between the second prediction result and the corresponding true value, and the difference between the third prediction result and the corresponding true value, the model parameters of the multimodal model are adjusted; wherein the true value corresponding to the second prediction result is the RGB value of the corresponding pixel point in the sample image of the real point in the bird's-eye view grid.

[0014] In a second aspect, an embodiment of the present application provides a point cloud processing method, the method comprising:

[0015] Get the point cloud to be processed;

[0016] Inputting the point cloud to be processed into a first encoder in a multimodal model trained by the multimodal model pre-training method provided in the first aspect above, to obtain a BEV feature to be used;

[0017] Target detection is performed based on the BEV features to be utilized to obtain a detection result.

[0018] In a third aspect, an embodiment of the present application provides a multimodal model pre-training method, the method comprising:

[0019] Obtaining sample point clouds and sample images corresponding to the same scene;

[0020] voxelize the sample point cloud to obtain a plurality of voxels, perform point cloud masking on some real points in the sample point cloud according to the bird's-eye view grid to which each voxel belongs to obtain a first mask point cloud, and perform image masking on the sample image to obtain a first mask image;

[0021] Using a first encoder in a multimodal model to perform feature extraction on the first mask point cloud to obtain features from a BEV perspective as first BEV features, and using a second encoder in the multimodal model to perform feature extraction on the first mask image to obtain first image features; wherein the multimodal model further includes a second converter, a second fusion network, a first decoder, and a second decoder;

[0022] Using the second converter, converting the first BEV feature to the image perspective of the sample image to obtain a second conversion feature;

[0023] Using the second fusion network to perform feature fusion on the second conversion feature and the first image feature to obtain a second image feature;

[0024] The first decoder is used to decode the second BEV feature to obtain a first prediction result, and the second decoder is used to decode the second image feature to obtain a third prediction result and a fourth prediction result; wherein the first prediction result represents: the predicted number of prediction points contained in the bird's-eye view grid and the predicted coordinates of each prediction point, the third prediction result represents: the predicted RGB value of each pixel in the sample image, and the fourth prediction result represents: the predicted depth value of each pixel in the sample image; the second BEV feature is obtained based on the first BEV feature;

[0025] Based on the difference between the first prediction result and the corresponding true value, the difference between the third prediction result and the corresponding true value, and the difference between the fourth prediction result and the corresponding true value, the model parameters of the multimodal model are adjusted; wherein the true value corresponding to the fourth prediction result is the depth value of the corresponding point of each pixel point of the sample image in the sample point cloud.

[0026] In a fourth aspect, an embodiment of the present application provides an image processing method, the method comprising:

[0027] Get the image to be processed;

[0028] Inputting the image to be processed into a second encoder in a multimodal model trained based on the multimodal model pre-training method provided in the third aspect above, to obtain image features to be used;

[0029] Target detection is performed based on the image features to be utilized to obtain a detection result.

[0030] In a fifth aspect, an embodiment of the present application provides a multimodal model pre-training device, the device comprising:

[0031] A first acquisition module acquires a sample point cloud and a sample image corresponding to the same scene;

[0032] A first mask module performs voxelization processing on the sample point cloud to obtain a plurality of voxels, performs point cloud mask processing on some real points in the sample point cloud according to the bird's-eye view grid to which each voxel belongs to obtain a first mask point cloud, and performs image mask processing on the sample image to obtain a first mask image;

[0033] A first feature extraction module, using a first encoder in a multimodal model to perform feature extraction on the first mask point cloud to obtain features under a BEV perspective as first BEV features, and using a second encoder in the multimodal model to perform feature extraction on the first mask image to obtain first image features; wherein the multimodal model further includes a first converter, a first fusion network, a first decoder, and a second decoder;

[0034] A first conversion module, using the first converter to convert the first image feature to the BEV perspective to obtain a first conversion feature;

[0035] a first fusion module, using the first fusion network to perform feature fusion on the first conversion feature and the first BEV feature to obtain a second BEV feature;

[0036] A first decoding module is configured to decode the second BEV feature using the first decoder to obtain a first prediction result and a second prediction result, and to decode the second image feature using the second decoder to obtain a third prediction result; wherein the first prediction result represents: the predicted number of prediction points contained in the bird's-eye view grid and the predicted coordinates of each prediction point, the second prediction result represents: the predicted RGB value of each prediction point in the bird's-eye view grid, and the third prediction result represents: the predicted RGB value of each pixel in the sample image; the second image feature is obtained based on the first image feature;

[0037] The first adjustment module adjusts the model parameters of the multimodal model based on the difference between the first prediction result and the corresponding true value, the difference between the second prediction result and the corresponding true value, and the difference between the third prediction result and the corresponding true value; wherein the true value corresponding to the second prediction result is the RGB value of the corresponding pixel point in the sample image of the real point in the bird's-eye view grid.

[0038] In a sixth aspect, an embodiment of the present application provides a point cloud processing device, the device comprising:

[0039] Point cloud acquisition module, used to acquire the point cloud to be processed;

[0040] A point cloud input module, used for inputting the point cloud to be processed into a first encoder in a multimodal model trained based on the multimodal model pre-training method provided in the first aspect above, to obtain a BEV feature to be used;

[0041] The point cloud detection module is used to perform target detection based on the BEV features to be utilized to obtain a detection result.

[0042] In a seventh aspect, an embodiment of the present application provides a multimodal model pre-training device, the device comprising:

[0043] A second acquisition module acquires a sample point cloud and a sample image corresponding to the same scene;

[0044] A second mask module performs voxelization processing on the sample point cloud to obtain a plurality of voxels, performs point cloud mask processing on some real points in the sample point cloud according to the bird's-eye view grid to which each voxel belongs to obtain a first mask point cloud, and performs image mask processing on the sample image to obtain a first mask image;

[0045] A second feature extraction module, using the first encoder in the multimodal model to perform feature extraction on the first mask point cloud to obtain features under the BEV perspective as first BEV features, and using the second encoder in the multimodal model to perform feature extraction on the first mask image to obtain first image features; wherein the multimodal model further includes a second converter, a second fusion network, a first decoder and a second decoder;

[0046] A second conversion module, using the second converter to convert the first BEV feature to the image perspective of the sample image to obtain a second conversion feature;

[0047] a second fusion module, using the second fusion network to perform feature fusion on the second conversion feature and the first image feature to obtain a second image feature;

[0048] A second decoding module is configured to decode the second BEV feature using the first decoder to obtain a first prediction result, and to decode the second image feature using the second decoder to obtain a third prediction result and a fourth prediction result; wherein the first prediction result indicates: the predicted number of prediction points contained in the bird's-eye view grid and the predicted coordinates of each prediction point, the third prediction result indicates: the predicted RGB value of each pixel in the sample image, and the fourth prediction result indicates: the predicted depth value of each pixel in the sample image; the second BEV feature is obtained based on the first BEV feature;

[0049] The second adjustment module adjusts the model parameters of the multimodal model based on the difference between the first prediction result and the corresponding true value, the difference between the third prediction result and the corresponding true value, and the difference between the fourth prediction result and the corresponding true value; wherein the true value corresponding to the fourth prediction result is the depth value of the corresponding point of each pixel point of the sample image in the sample point cloud.

[0050] In an eighth aspect, an embodiment of the present application provides an image processing device, the device comprising:

[0051] An image acquisition module, used for acquiring an image to be processed;

[0052] An image input module, used for inputting the image to be processed into a second encoder in the multimodal model trained based on the multimodal model pre-training method provided in the third aspect above, to obtain image features to be used;

[0053] The image detection module is used to perform target detection based on the image features to be utilized to obtain a detection result.

[0054] In a ninth aspect, an embodiment of the present application provides an electronic device, including:

[0055] Memory, used to store computer programs;

[0056] The processor is used to implement any of the multimodal model pre-training methods, point cloud processing methods, or image processing methods described above when executing the program stored in the memory.

[0057] In the tenth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements any one of the multimodal model pre-training methods, point cloud processing methods, or image processing methods described above.

[0058] In the eleventh aspect, an embodiment of the present application provides a computer program product, which includes executable instructions. When the executable instructions are executed on a computer, the computer executes any one of the multimodal model pre-training methods, point cloud processing methods, or image processing methods described above.

[0059] Beneficial effects of the embodiments of the present application:

[0060] The solution provided by the embodiment of the present application adjusts the model parameters based on the difference between the first prediction result and the corresponding true value, so that the model can learn the distribution of points in the point cloud, including the number of points and position information, from the sample point cloud. Adjust the model parameters based on the difference between the third prediction result and the corresponding true value, so that the model can learn color information from the sample image. Taking the RGB value of the corresponding pixel point in the sample image of the real point in the bird's-eye view grid as the true value corresponding to the second prediction result can give full play to the complementary role of the color information in the sample image to the color information of the point cloud. Then, based on the difference between the second prediction result and the corresponding true value, the model parameters are adjusted so that the model can make full use of the color information in the sample image to learn the color information of the point cloud, so that the first encoder in the trained multimodal model can accurately predict the RGB information in the input point cloud based on the distribution of points in the input point cloud. Thereby, the processing effect of the encoder in the trained multimodal model in processing downstream tasks can be improved.

[0061] Of course, implementing any product or method of the present application does not necessarily require achieving all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application, and for ordinary technicians in this field, other embodiments can also be obtained based on these drawings.

[0063] Figure 1 A flowchart of a multimodal model pre-training method provided in an embodiment of the present application;

[0064] Figure 2 A flowchart of another multimodal model pre-training method provided in an embodiment of the present application;

[0065] Figure 3 A schematic diagram of a specific example of a multimodal model pre-training method provided in an embodiment of the present application;

[0066] Figure 4 A flow chart of a point cloud processing method provided in an embodiment of the present application;

[0067] Figure 5 A flowchart of another multimodal model pre-training method provided in an embodiment of the present application;

[0068] Figure 6 A flowchart of an image processing method provided in an embodiment of the present application;

[0069] Figure 7 A schematic diagram of the structure of a multimodal model pre-training device provided in an embodiment of the present application;

[0070] Figure 8 A schematic diagram of the structure of a point cloud processing device provided in an embodiment of the present application;

[0071] Fig. 9 A schematic diagram of the structure of another multimodal model pre-training device provided in an embodiment of the present application;

[0072] Fig.10 A schematic diagram of the structure of an image processing device provided in an embodiment of the present application;

[0073] Fig.11 A block diagram of an electronic device for implementing the method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0074] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field based on the present application belong to the scope of protection of the present application.

[0075] First, the professional terms involved in the embodiments of the present application are introduced.

[0076] BEV (Bird's-Eye-View): Also known as a bird's-eye view, it refers to observing a scene or object from above.

[0077] Voxel: It is the smallest unit in three-dimensional space division and is used to represent volume elements.

[0078] Below, the multimodal model pre-training method provided in the embodiment of the present application is introduced.

[0079] The multimodal model pre-training method provided in the embodiment of the present application can be applied to various electronic devices, such as personal computers, servers, and other devices with data processing capabilities. In addition, it can be understood that the multimodal model pre-training method provided in the embodiment of the present application can be implemented by software, hardware, or a combination of software and hardware. It should be noted that the embodiment of the present application provides two multimodal model pre-training methods, specifically: a multimodal model pre-training method based on converting the first image feature to the BEV perspective and fusing it with the first BEV feature (hereinafter referred to as the first scheme), and a multimodal model pre-training method based on converting the first BEV feature to the image perspective and fusing it with the first image feature (hereinafter referred to as the second scheme).

[0080] The first solution provided by the embodiment of the present application is introduced below in conjunction with the accompanying drawings. Figure 1 The multimodal model pre-training method provided in the embodiment of the present application includes steps S101-S107:

[0081] S101, obtaining a sample point cloud and a sample image corresponding to the same scene;

[0082] In this embodiment, image acquisition and point cloud acquisition can be performed on the same scene respectively to obtain point clouds and images corresponding to the same scene as sample point clouds and sample images corresponding to the same scene. For example, for any scene, an image of the scene can be obtained by photographing the scene with a camera, and a point cloud of the scene can be obtained by scanning the scene with a laser radar or other sensor equipment.

[0083] S102, voxelizing the sample point cloud to obtain a plurality of voxels, performing point cloud masking processing on some real points in the sample point cloud according to the bird's-eye view grid to which each voxel belongs to obtain a first mask point cloud, and performing image masking processing on the sample image to obtain a first mask image;

[0084] In this embodiment, after obtaining the sample point cloud, the sample point cloud is voxelized, that is, a bounding box of a three-dimensional space is determined according to the coordinates of each point in the sample point cloud, and the determined bounding box is divided according to the spatial range of a cube of a predetermined size to obtain multiple voxels. Exemplarily, in practical applications, a VEF (Voxel feature Encoding) network layer can be used to voxelize the point cloud. It should be noted that the embodiment of the present application does not limit the size of the spatial range of the voxel.

[0085] Optionally, in one implementation, performing point cloud masking processing on some real points in the sample point cloud according to the bird's-eye view grid to which each voxel belongs to obtain a first mask point cloud may include steps A1-A3:

[0086] A1, grouping multiple voxels to obtain voxels belonging to the same bird's-eye view grid; A2, determining the bird's-eye view grid to be masked from each bird's-eye view grid according to the first ratio; A3, replacing the coordinates of each real point contained in the determined bird's-eye view grid to be masked with the current predetermined coordinates to obtain a first mask point cloud.

[0087] In this implementation, a mask strategy based on BEV-guide (bird's-eye view guide) is used for point cloud mask processing. After obtaining multiple voxels, all voxels are assigned to different bird's-eye view grids (BEV-grids) according to their voxel coordinates to achieve grouping of voxels, and each bird's-eye view grid contains multiple non-empty voxels. Exemplarily, the projection range of the sample point cloud on the bird's-eye view plane can be divided into multiple bird's-eye view grids, and the projection positions of the multiple voxels on the bird's-eye view plane are determined according to the voxel coordinates of the multiple voxels, so that all voxels are assigned to different bird's-eye view grids. Then, the first proportion of the bird's-eye view grid to be masked is determined from the voxels belonging to the same bird's-eye view grid. Exemplarily, the first proportion can be 10%, 20%, and so on. Exemplarily, the method of determining the first proportion of the bird's-eye view grid to be masked from the multiple bird's-eye view grids obtained by division can be randomly selected, or the first proportion of the bird's-eye view grid to be masked is selected according to a predetermined interval, which is reasonable.

[0088] After determining the bird's-eye view grid to be masked, the determined bird's-eye view grid to be masked is masked by replacing the coordinates of each real point contained in the determined bird's-eye view grid to be masked with the current predetermined coordinates to obtain a first mask point cloud. Exemplarily, in practical applications, the predetermined coordinates can be a fixed random number, or a learnable parameter Token. If the predetermined coordinates are learnable parameter Tokens, then when the multimodal model starts training, the Token can be set to a random number. Accordingly, when adjusting the model parameters based on the difference between each prediction result and the corresponding true value during the model training process, the Token is also adjusted based on the difference to improve the generalization ability of the model.

[0089] In this embodiment, the sample image is subjected to image masking processing by selecting a preset number of pixels from the sample image and then masking the RGB (Red Green Blue) values ​​of the selected pixels, for example, replacing the RGB values ​​of the selected pixels with 0 to obtain a first mask image.

[0090] Optionally, in one implementation, performing image mask processing on the sample image to obtain a first mask image may include steps B1-B3:

[0091] B1, dividing the sample image according to a predetermined size to obtain multiple image areas; B2, determining the image area to be masked from the multiple image areas according to a second ratio; B3, replacing the RGB values ​​of the pixels in the determined image area to be masked with predetermined RGB values ​​to obtain a first mask image.

[0092] In this implementation, the sample image is divided into regular and non-overlapping patches (blocks) according to a predetermined size to obtain multiple image areas. Then, an image area of ​​a second proportion is determined from the multiple image areas as the image area to be masked. Exemplarily, the second proportion can be 10%, 20%, and so on. After determining the image area to be masked, the RGB values ​​of the pixels in the image area to be masked are replaced with predetermined RGB values ​​to obtain a first mask image. It should be noted that the predetermined size and the predetermined RGB value can be set by relevant technicians based on experience, and the embodiments of the present application are not limited to this.

[0093] S103, using the first encoder in the multimodal model to extract features from the first mask point cloud to obtain features from a BEV (bird's eye view) perspective as first BEV features, and using the second encoder in the multimodal model to extract features from the first mask image to obtain first image features; wherein the multimodal model further includes a first converter, a first fusion network, a first decoder, and a second decoder;

[0094] In this embodiment, the first encoder is a 3D (3-dimensional) feature extraction network, and the second encoder is a 2D (2-dimensional) feature extraction network. Exemplarily, the first encoder can discretize and project the first mask point cloud on the XY coordinate plane to convert the first mask point cloud to the BEV perspective, and then extract features from the first mask point cloud converted to the BEV perspective to obtain the first BEV feature.

[0095] Optionally, in one implementation, the first encoder includes a BEV encoder.

[0096] In this implementation, 3D sparse convolution is used to extract features from the voxels in the first mask point cloud, so as to project the features of the voxels into the BEV space and obtain the features from the BEV perspective, that is, the voxels are compressed on the Z axis until the Z axis dimension is 1. Then, the BEV encoder is used to further extract features from the features converted to the BEV perspective to obtain the first BEV feature. The resolution of the first BEV feature output by the BEV encoder needs to be consistent with the number of grids of the bird's-eye view obtained by division.

[0097] For example, if the sample point cloud is represented as , that is, the sample point cloud contains n points, is the 3D coordinate of the point. The projection range of the sample point cloud on the bird's-eye view plane is divided into M bird's-eye view grids, and the first BEV feature output by the BEV encoder is expressed as , that is, For a feature map with width w, height h, and feature dimension (that is, channel dimension) c, then h×w=M. That is, each element position in the feature map corresponds to a bird's-eye view grid, so that the M bird's-eye view grids can be predicted later.

[0098] Exemplarily, in practical applications, the BEV encoder may be composed of a transformer network or a convolutional layer. It should be noted that the embodiment of the present application does not limit the network structure of the first encoder.

[0099] The second encoder extracts features from the first mask image to obtain the first image features .in, , h and w are the height and width of the first image feature, respectively, and c is the feature dimension of the first image feature. It should be noted that, since the values ​​of h and w of the first BEV feature are related to the number of bird's-eye view grids, and the resolution of the first image feature is independent of the bird's-eye view grids, the h and w of the first image feature may be different from the h and w of the first BEV feature.

[0100] In this embodiment, in addition to the first encoder and the second encoder, the multimodal model also includes a first converter, a first fusion network, a first decoder and a second decoder. The functions of the first converter, the first fusion network, the first decoder and the second decoder are introduced in the following steps.

[0101] S104, using a first converter to convert the first image feature to a BEV perspective to obtain a first conversion feature;

[0102] In this embodiment, after obtaining the first image feature, the first image feature is converted to the BEV perspective, so that when the first conversion feature and the first BEV feature are subsequently fused, the first conversion feature and the first BEV feature are features under the same BEV perspective, thereby increasing the amount of information in the fused data.

[0103] Optionally, in one implementation, the first transformer is a LSS Transformer (Lift-Splat-Shoot Transformer).

[0104] In this implementation, the LSS Transformer estimates the depth of each pixel in the first image feature, "lifts" these pixels into 3D space, and these 3D information are projected onto a planar view to form a first transformation feature. It should be noted that the embodiment of the present application does not limit the specific network type of the first transformer.

[0105] S105, using a first fusion network to perform feature fusion on the first conversion feature and the first BEV feature to obtain a second BEV feature;

[0106] In this embodiment, the feature fusion of the first conversion feature and the first BEV feature may be a weighted sum of the first conversion feature and the first BEV feature, or a concatenation of the first conversion feature and the first BEV feature, both of which are reasonable.

[0107] Optionally, in one implementation, the first fusion network is a first cross-attention network. It is understandable that using the attention mechanism to perform cross-modal interaction on features of the point cloud modality and the image modality can better perform cross-modal reconstruction tasks.

[0108] Accordingly, in the present implementation, in step S105, the first conversion feature and the first BEV feature are fused using the first fusion network to obtain the second BEV feature, which may include: mapping the first BEV feature to obtain Q in the input data of the first cross-attention network, and mapping the first conversion feature to obtain K and V in the input data of the first cross-attention network, and inputting the obtained Q, K and V into the first cross-attention network for feature fusion to obtain the second BEV feature.

[0109] Exemplarily, the first BEV feature may be processed using a preset first transformation function to map the first BEV feature to Q, and the first conversion feature may be processed using a preset second transformation function to map the first conversion feature to K and V. The first transformation function and the second transformation function may be preset linear variation functions, and in practical applications, the first transformation function and the second transformation function may be set by relevant technicians.

[0110] It can be understood that Q, K and V obtained based on the first BEV feature and the first conversion feature mapping are input into the cross attention network, so that the information of the sample point cloud and the sample image can fully interact. The first cross attention network maps the first BEV feature to Q, and the first conversion feature to K and V. The weighting method of V is determined by calculating the similarity between Q and K. The obtained second BEV feature pays more attention to the information related to the first BEV feature, which improves the expression ability of the model.

[0111] It should be noted that the embodiment of the present application does not limit the specific type of the first fusion network. For example, in another implementation, the first fusion network may also be a channel attention network.

[0112] S106, using the first decoder to decode the second BEV feature to obtain a first prediction result and a second prediction result, and using the second decoder to decode the second image feature to obtain a third prediction result; wherein the first prediction result represents: the predicted number of prediction points contained in the bird's-eye view grid and the predicted coordinates of each prediction point, the second prediction result represents: the predicted RGB value of each prediction point in the bird's-eye view grid, and the third prediction result represents: the predicted RGB value of each pixel in the sample image; the second image feature is obtained based on the first image feature;

[0113] It can be understood that since the second BEV feature is obtained by converting the first image feature to the BEV perspective and fusing it with the first BEV feature, and the first image feature contains the color information of the sample image, and the first BEV feature contains the number and position information of the midpoints of the sample point cloud, the second BEV feature contains the position and color information of each point under the BEV perspective. Then, by decoding the second BEV feature using the first decoder, a first prediction result indicating the predicted number of predicted points contained in the bird's-eye view grid and the predicted coordinates of each predicted point, and a second prediction result indicating the predicted RGB value of each predicted point in the bird's-eye view grid can be obtained.

[0114] For example, three output layers may be designed in the first decoder, which are respectively used to output the predicted number of predicted points contained in the bird's-eye view grid. , the predicted coordinates of each predicted point in the bird's-eye view grid , and the predicted RGB values ​​of each predicted point in the bird's-eye view grid .in, , as well as They can be expressed as:

[0115] ;

[0116] ;

[0117] ;

[0118] in, represents 2D convolution, is the second BEV feature. In the first decoder, the second BEV feature passes through a shared convolutional layer After processing, they are input into the convolutional layer respectively. and In. After the convolutional layer After processing, the predicted number of predicted points contained in the bird's-eye view grid is obtained, and the convolution layer After processing, the predicted coordinates of each predicted point in the bird's-eye view grid are obtained, and the convolution layer After processing, the predicted RGB value of each predicted point in the bird's-eye view grid is obtained. , , .in, is the number of bird’s-eye view grids, Indicates the number of prediction points in each bird's-eye view grid, that is, the number of prediction points in a bird's-eye view grid Since the number of real points in each bird's-eye view grid is not fixed, and the number of channels of the convolution kernel cannot be adjusted dynamically, the 3D coordinates of a fixed number of points are predicted, that is, is a hyperparameter.

[0119] In this embodiment, the second image feature may be the first image feature itself. In this case, the second decoder decodes the first image feature to predict the RGB of each pixel in the first mask image, and the third prediction result output by the second decoder includes the predicted RGB value of each pixel in the sample image. Exemplarily, the predicted RGB value of each pixel in the sample image can be expressed as:

[0120] ;

[0121] in, is the first image feature, represents 2D convolution, is the predicted RGB value of each pixel in the sample image. After the convolution layer After processing, input to the convolution layer In the convolutional layer After processing, the predicted RGB value of each pixel in the sample image is obtained. , 3 represents the pixel values ​​of the three RGB channels, and w and h represent the width and height of the first image feature respectively.

[0122] S107, adjusting the model parameters of the multimodal model based on the difference between the first prediction result and the corresponding true value, the difference between the second prediction result and the corresponding true value, and the difference between the third prediction result and the corresponding true value; wherein the true value corresponding to the second prediction result is the RGB value of the corresponding pixel point in the sample image of the real point in the bird's-eye view grid.

[0123] It can be understood that since the first prediction result represents: the predicted number of prediction points contained in the bird's-eye view grid and the predicted coordinates of each prediction point, the true value corresponding to the first prediction result contains two parts, namely the true number of real points contained in the bird's-eye view grid and the true coordinates of each real point. The true value corresponding to the first prediction result can be directly determined from the sample point cloud.

[0124] Exemplarily, the method of calculating the difference between the first prediction result and the corresponding true value can be: using the L1 loss function to calculate the difference A between the predicted number and the actual number of prediction points contained in the bird's-eye view grid, and using the chamfer distance loss function to calculate the difference B between the predicted coordinates and the actual coordinates of each prediction point in the bird's-eye view grid, and taking the sum or mean of A and B as the difference between the first prediction result and the corresponding true value.

[0125] Since the third prediction result is the predicted RGB value of each pixel in the sample image, the true value corresponding to the third prediction result is the true RGB value of each pixel in the sample image, which can be directly determined from the sample image. Exemplarily, the method of calculating the difference between the third prediction result and the corresponding true value can be: using the mean square error loss function to calculate the difference between the predicted RGB value and the true RGB value of each pixel in the sample image, and taking the sum or mean of the obtained differences as the difference between the third prediction result and the corresponding true value.

[0126] Since the second prediction result is the predicted RGB value of each prediction point in the bird's-eye view grid, and the sample point cloud does not contain RGB information, the true value corresponding to the second prediction result is determined using the RGB information in the sample image. In this embodiment, the RGB value of a specified pixel of the sample image is used as the true value corresponding to the second prediction result, and the specified pixel is the corresponding pixel of the real point in the sample point cloud in the sample image, that is, the corresponding pixel in the sample image after the real point in the sample point cloud is converted to the image coordinate system of the sample image.

[0127] It is understandable that after the sample point cloud is transformed into the image coordinate system of the sample image, each real point in the bird's-eye view grid has a corresponding designated pixel point in the sample image. Exemplarily, the method of calculating the difference between the second prediction result and the corresponding true value can be: using the chamfer distance loss function to calculate the difference between the predicted RGB value of each predicted point contained in the bird's-eye view grid and the real RGB value of the corresponding designated pixel point, and taking the sum or mean of the obtained differences as the difference between the second prediction result and the corresponding true value.

[0128] It should be noted that the embodiment of the present application does not limit the method of calculating the differences between the first prediction result, the second prediction result and the third prediction result and the corresponding true values.

[0129] It can be understood that by using the expression parameters of the camera that collects the sample image and the expression parameters of the laser radar that collects the sample point cloud, the sample point cloud can be projected onto the sample image to transform the sample point cloud into the image coordinate system of the sample image. For example, if the coordinates of the i-th real point in the sample point cloud are expressed as: , where i=1,2…n, n is the number of points in the sample point cloud, then the real points in the sample point cloud can be calculated using the following formula Transform to the image coordinate system of the sample image:

[0130] ;

[0131] Among them, P is the intrinsic parameter of the camera, T is the extrinsic parameter of the camera, is the transformation matrix from the point cloud coordinate system to the camera coordinate system, and point The pixel coordinates converted to the image coordinate system of the sample image, the RGB value of the pixel coordinate position on the sample image is the pixel coordinates of the sample image. The true value of the RGB value.

[0132] It is understandable that after calculating the difference between each prediction result and the corresponding true value, the model loss value can be calculated. Exemplarily, in one implementation, the sum or mean of the difference between the first prediction result and the corresponding true value, the difference between the second prediction result and the corresponding true value, and the difference between the third prediction result and the corresponding true value can be calculated as the model loss value, and then the model parameters are adjusted by back propagation to minimize the model loss value until the model converges, that is, the model loss value is less than a preset threshold, and the training is terminated.

[0133] It should be noted that the embodiments of the present application do not limit the method of calculating the model loss value based on the differences between each prediction result and the corresponding true value. For example, in another implementation method, the differences between each prediction result and the corresponding true value can be weighted summed to obtain the model loss value, which is reasonable.

[0134] It can be understood that by adjusting the model parameters of the multimodal model based on the differences between the first prediction result, the second prediction result, and the third prediction result and the corresponding true values, the encoder in the trained multimodal model can make full use of image information, point cloud information, and interactive information between the image and the point cloud, thereby improving the modeling capability of the RGB information of the point cloud, and further improving the ability of the trained encoder to handle single-modal tasks and multimodal tasks. In addition, since the true values ​​corresponding to the first prediction result, the second prediction result, and the third prediction result can be obtained from the input sample point cloud and sample image, the multimodal model can be self-supervised trained, reducing the dependence on labeled data.

[0135] The solution provided by the embodiment of the present application adjusts the model parameters based on the difference between the first prediction result and the corresponding true value, so that the model can learn the distribution of points in the point cloud, including the number of points and position information, from the sample point cloud. Adjust the model parameters based on the difference between the third prediction result and the corresponding true value, so that the model can learn color information from the sample image. Taking the RGB value of the corresponding pixel point in the sample image of the real point in the bird's-eye view grid as the true value corresponding to the second prediction result can give full play to the complementary role of the color information in the sample image to the color information of the point cloud. Then, based on the difference between the second prediction result and the corresponding true value, the model parameters are adjusted so that the model can make full use of the color information in the sample image to learn the color information of the point cloud, so that the first encoder in the trained multimodal model can accurately predict the RGB information in the input point cloud based on the distribution of points in the input point cloud. Thereby, the processing effect of the encoder in the trained multimodal model in processing downstream tasks can be improved.

[0136] Optionally, in another embodiment of the present application, the difference between the first prediction result and the corresponding true value includes: a first difference between the predicted number of predicted points contained in the bird's-eye view grid to be masked and the actual number of real points contained in the bird's-eye view grid to be masked, and a second difference between the predicted coordinates of each predicted point in the bird's-eye view grid to be masked and the actual coordinates of each real point in the bird's-eye view grid to be masked.

[0137] It can be understood that only the loss of the to-be-masked bird's-eye view grid before and after the point cloud reconstruction is calculated as the difference between the first prediction result and the corresponding true value, and there is no need to calculate the loss of all the bird's-eye view grids before and after the point cloud reconstruction, thereby reducing the amount of calculation. Among them, the loss of the to-be-masked bird's-eye view grid before and after the point cloud reconstruction includes: the loss of the number of points contained before and after the point cloud reconstruction and the position loss of each point, that is, the loss between the predicted number and the actual number of predicted points contained in the to-be-masked bird's-eye view grid, and the loss between the predicted coordinates of each predicted point in the to-be-masked bird's-eye view grid and the actual coordinates of each actual point in the to-be-masked bird's-eye view grid.

[0138] Optionally, in one implementation, a smooth L1 loss function is used to calculate a first difference between a predicted number of predicted points included in the to-be-masked bird's-eye view grid in the first prediction result and a real number of real points included in the to-be-masked bird's-eye view grid in the sample point cloud.

[0139] It is understandable that the smoothed L1 loss function is a loss function that combines the square error loss and the absolute error loss, which can reduce the impact of outliers when training the model, thereby improving the robustness of the model. It should be noted that the embodiment of the present application does not limit the method for calculating the first difference. For example, in another implementation, the first difference can also be calculated by the L1 loss function or the L2 loss function.

[0140] Optionally, in one implementation, the second difference is the sum of the first sub-differences corresponding to each bird's-eye view grid to be masked; the first sub-difference corresponding to any bird's-eye view grid to be masked is: calculated using the first chamfer distance loss function based on the predicted coordinates of each predicted point in the bird's-eye view grid to be masked and the real coordinates of each real point in the bird's-eye view grid to be masked.

[0141] It can be understood that for two sets of 3D point clouds S1 and S2, the chamfer distance is the sum of the minimum distances from any point of S1 to S2 plus the sum of the minimum distances from any point of S2 to S1. The larger the distance, the greater the difference between the two sets of point clouds; the smaller the distance, the smaller the difference between the two sets of point clouds, and the better the point cloud reconstruction effect. Therefore, the first chamfer distance loss function is used to calculate the first sub-difference corresponding to each masked bird's-eye view grid, so that after adjusting the model parameters using the calculated second difference, a point cloud that is closer to the real point cloud can be generated.

[0142] Exemplarily, in a specific implementation, the first chamfer distance loss function is:

[0143] ;

[0144] in, is the first sub-difference corresponding to the i-th bird's-eye view grid to be masked, L represents the predicted number of predicted points contained in the i-th bird's-eye view grid to be masked, Indicates the first grid in the i-th bird's-eye view to be masked The predicted coordinates of the predicted points, represents the predicted coordinates of each predicted point in the i-th grid of the bird's-eye view to be masked; N represents the actual number of real points contained in the i-th grid of the bird's-eye view to be masked, Indicates the first grid in the i-th bird's-eye view to be masked The real coordinates of a real point, Represents the real coordinates of each real point in the i-th bird's-eye view grid to be masked. The maximum value of i is the number of bird's-eye view grids to be masked determined during point cloud mask processing. Represents the square of the L2 (Euclidean) norm.

[0145] Optionally, in one implementation, the difference between the second prediction result and the corresponding true value is the sum of the second sub-differences corresponding to each bird's-eye view grid; the second sub-difference corresponding to any bird's-eye view grid is: calculated using the second chamfer distance loss function based on the predicted RGB value of each predicted point in the bird's-eye view grid and the RGB value of the corresponding pixel point of the real point in the bird's-eye view grid in the sample image.

[0146] In this implementation, since the sample point cloud does not contain RGB information, the second sub-differences corresponding to all bird's-eye view grids can be calculated during model pre-training so that the model can fully learn the RGB information in the sample image.

[0147] Exemplarily, in a specific implementation, the second chamfer distance loss function is:

[0148] ;

[0149] in, represents the second sub-difference corresponding to the i-th bird's-eye view grid, L represents the predicted number of predicted points contained in the i-th bird's-eye view grid, Indicates the first The predicted RGB value of the prediction point, represents the predicted RGB value of each predicted point in the i-th bird's-eye view grid; N represents the actual number of real points contained in the i-th bird's-eye view grid, Indicates the first The RGB value of the pixel corresponding to the real point in the sample image, represents the RGB value of the pixel corresponding to each real point in the i-th bird's-eye view grid in the sample image. The maximum value of i is the number of bird's-eye view grids obtained by dividing the projection range of the sample point cloud on the bird's-eye view plane.

[0150] It should be noted that the embodiment of the present application does not limit the method of calculating the first sub-difference corresponding to each bird's-eye view grid to be masked and the method of calculating the second sub-difference corresponding to each bird's-eye view grid. Any loss function that can calculate the difference between two different numbers of point sets can be used to calculate the first sub-difference and the second sub-difference.

[0151] It can be seen that through this solution, it is only necessary to calculate the loss of the masked bird's-eye view grid before and after point cloud reconstruction as the difference between the first prediction result and the corresponding true value, thereby reducing the amount of calculation.

[0152] Optionally, in another embodiment of the present application, Figure 1 Based on the illustrated embodiment, the multimodal model further includes a second converter and a second fusion network;

[0153] In the above step S106, the second image feature is decoded by the second decoder to obtain the third prediction result. Figure 2 As shown, the multimodal model pre-training method further includes steps S201-S202:

[0154] S201, using a second converter to convert a first BEV (bird's eye view) feature to an image perspective of a sample image to obtain a second converted feature;

[0155] In this embodiment, after obtaining the first BEV feature of the sample point cloud, the first BEV feature is converted to the image perspective of the sample image, so that when the second converted feature and the first image feature are subsequently fused, the second converted feature and the first image feature are features under the same image perspective, thereby increasing the amount of information in the fused data.

[0156] Optionally, in one implementation, the second converter is used to convert the first BEV feature to the image perspective of the sample image based on the intrinsic parameters, extrinsic parameters of the camera that captures the sample image and the conversion relationship between the point cloud coordinate system of the sample point cloud and the camera coordinate system.

[0157] It is understandable that the first BEV feature is converted to the image perspective of the sample image, that is, the first BEV feature is mapped to the image coordinate system of the sample image. It should be noted that the method of converting the first BEV feature to the image perspective of the sample image can refer to the method of converting the sample point cloud to the image coordinate system of the sample image in the above step S107, and will not be repeated here. It is understandable that since the points in the sample point cloud are sparser than the pixel points in the sample image, when the sample point cloud is mapped to the image coordinate system of the sample image, there are pixel positions that cannot be mapped, that is, the pixel points of the sample image have no corresponding points in the sample point cloud. At this time, the value of the pixel position that cannot be mapped is set to 0. It should be noted that the embodiment of the present application does not limit the network structure of the second converter.

[0158] S202, using a second fusion network to perform feature fusion on the second conversion feature and the first image feature to obtain a second image feature;

[0159] In this embodiment, the feature fusion of the second conversion feature and the first image feature may be a weighted sum of the second conversion feature and the first image feature, or a concatenation of the second conversion feature and the first image feature, both of which are reasonable.

[0160] Optionally, in one implementation, the second fusion network is a second cross-attention network; accordingly, in this implementation, in step S202, the second conversion feature and the first image feature are subjected to feature fusion using the second fusion network to obtain the second image feature, which may include: mapping the first image feature to obtain Q in the input data of the second cross-attention network, and mapping the second conversion feature to obtain K and V in the input data of the second cross-attention network, and inputting the obtained Q, K and V into the second cross-attention network for feature fusion to obtain the second image feature.

[0161] Exemplarily, the first image feature may be processed using a preset third transformation function to map the first image feature to Q, and the second transformation feature may be processed using a preset fourth transformation function to map the second transformation feature to K and V. The third transformation function and the fourth transformation function may be preset linear variation functions, and in practical applications, the third transformation function and the fourth transformation function may be set by relevant technicians.

[0162] It can be understood that Q, K and V obtained based on the first image feature and the second conversion feature mapping are input into the second cross-attention network, so that the information of the sample point cloud and the sample image can fully interact. The second cross-attention network maps the first image feature to Q, and the second conversion feature to K and V. The weighting method of V is determined by calculating the similarity between Q and K. The obtained second image feature pays more attention to the information related to the first image feature, thereby improving the expression ability of the model. It should be noted that the embodiment of the present application does not limit the specific type of the second fusion network. For example, in another implementation, the second fusion network can also be a multi-head attention network.

[0163] Accordingly, in this embodiment, in the above step S106, the second image feature is decoded using the second decoder to obtain a third prediction result, which may include: using the second decoder to decode the second image feature to obtain a third prediction result representing the predicted RGB value of each pixel in the sample image, and a fourth prediction result representing the predicted depth value of each pixel in the sample image.

[0164] That is, in this embodiment, the second image feature is obtained by fusion of the first image feature and the second conversion feature. It can be understood that since the second image feature is obtained by fusion of the first BEV feature after conversion to the image perspective and the first image feature, the second image feature contains the color information and depth information of each point under the image perspective. Then, by decoding the second image feature using the second decoder, a third prediction result representing the predicted RGB value of each pixel in the sample image and a fourth prediction result representing the predicted depth value of each pixel in the sample image can be obtained.

[0165] For example, two output layers can be designed in the second decoder, which are used to output the predicted RGB value of each pixel in the sample image. And the predicted depth value of each pixel in the sample image .in, and Can be expressed as:

[0166] ;

[0167] ;

[0168] in, represents 2D convolution, is the second image feature. , After the convolution layer After processing, they are input into the convolutional layer respectively. and In the convolutional layer After processing, the predicted RGB value of each pixel in the sample image is obtained, and the convolution layer After processing, the predicted depth value of each pixel in the image is obtained. , 3 represents the pixel value of the three channels of RGB, , w and h represent the width and height of the second image feature respectively.

[0169] Accordingly, in the present embodiment, adjusting the model parameters of the multimodal model based on the difference between the first prediction result and the corresponding true value, the difference between the second prediction result and the corresponding true value, and the difference between the third prediction result and the corresponding true value in the above-mentioned step S107 may include: adjusting the model parameters of the multimodal model based on the difference between the first prediction result and the corresponding true value, the difference between the second prediction result and the corresponding true value, the difference between the third prediction result and the corresponding true value, and the difference between the fourth prediction result and the corresponding true value; wherein the true value corresponding to the fourth prediction result is the depth value of the corresponding point of each pixel of the sample image in the sample point cloud.

[0170] Since the fourth prediction result is the predicted depth value of each pixel in the sample image, and the sample image does not contain depth information, the depth information in the sample point cloud is used to determine the true value corresponding to the fourth prediction result. In this embodiment, the depth value of the corresponding point of each pixel of the sample image in the sample point cloud is used as the true value corresponding to the fourth prediction result.

[0171] Exemplarily, the sample point cloud can be projected onto the sample image. If the position of point A in the sample point cloud projected onto the sample image is the pixel position of pixel point B in the sample image, then the corresponding point of pixel point B in the sample point cloud is point A. At this time, the true value of the depth value of pixel point B is the depth value of point A. In the case where the Z axis represents the depth, the depth value of point A is the Z axis coordinate of point A.

[0172] Exemplarily, the method for calculating the difference between the fourth prediction result and the corresponding true value can be: using the mean square error loss function to calculate the difference between the predicted depth value of each pixel in the sample image and the depth value of the corresponding point of each pixel of the sample image in the sample point cloud, and taking the sum or mean of the obtained differences as the difference between the third prediction result and the corresponding true value.

[0173] Exemplarily, the model loss value of the multimodal model can be calculated according to the following formula:

[0174] ;

[0175] in, Represents the model loss value of the multimodal model, represents the first difference, is the weight corresponding to the first difference, represents the second difference, is the weight corresponding to the second difference, represents the difference between the second prediction result and the corresponding true value, For The corresponding weights, represents the difference between the third prediction result and the corresponding true value, For The corresponding weights, represents the difference between the fourth prediction result and the corresponding true value, Indicates that The corresponding weight.

[0176] After the model loss value is calculated, the model parameters can be adjusted by minimizing the back propagation method of the model loss value until the model converges, that is, the model loss value is less than the preset threshold, and the training is terminated. It should be noted that the embodiment of the present application does not limit the method of calculating the model loss value based on the difference between each prediction result and the corresponding true value. For example, in another implementation method, the sum or mean of the differences between the first prediction result, the second prediction result, the third prediction result and the fourth prediction result and the corresponding true value can be calculated as the model loss value.

[0177] It can be understood that since the depth value of the corresponding point of each pixel of the sample image in the sample point cloud is used as the true value corresponding to the fourth prediction result, the model parameters are adjusted based on the differences between the first prediction result, the second prediction result and the third prediction result and the corresponding true value, respectively, combined with the difference between the fourth prediction result and the corresponding true value, so that the model can make full use of the depth information in the sample point cloud to learn the depth information of the image, that is, the second encoder in the trained multimodal model can accurately predict the depth information in the input image based on the color information in the input image. Thereby, the processing effect of the encoder in the trained multimodal model in processing downstream tasks can be further improved.

[0178] Exemplarily, in one implementation, the multimodal model provided in the embodiment of the present application can be as follows: Figure 3 As shown, the multimodal model includes a 3D (3-dimensional) encoder (corresponding to the first encoder mentioned above), a 2D (2-dimensional) encoder (corresponding to the second encoder mentioned above), a 3D-2D Projection (3-dimensional-2-dimensional conversion module, corresponding to the first converter mentioned above), an LSSTransformer (lift-flatten-shoot converter, corresponding to the second converter mentioned above), a first cross-modal interaction module (corresponding to the first fusion network mentioned above), a second cross-modal interaction module (corresponding to the second fusion network mentioned above), a point cloud decoder (corresponding to the first decoder mentioned above) and an image decoder (corresponding to the second decoder mentioned above).

[0179] like Figure 3 As shown, the sample point cloud is processed by point cloud mask to obtain a mask point cloud (corresponding to the first mask point cloud mentioned above), the sample image is processed by image mask to obtain a mask image (corresponding to the first mask image mentioned above), the mask point cloud is input into the 3D encoder for feature extraction to obtain the first BEV (bird's eye view) feature, and the mask image is input into the 2D encoder for feature extraction to obtain the first image feature. After the first image feature is input into the LSS Transformer and converted to the BEV perspective to obtain the first conversion feature, it is further input into the first cross-modal interaction module to perform feature fusion with the first BEV feature to obtain the second BEV feature. After the first BEV feature is input into the 3D-2D Projection and converted to the image perspective to obtain the second conversion feature, it is further input into the second cross-modal interaction module to perform feature fusion with the first image feature to obtain the second image feature. Among them, the first cross-modal interaction module and the second cross-modal interaction module are cross-attention networks.

[0180] The point cloud decoder decodes the second BEV feature to obtain the point cloud reconstruction result and the color information reconstruction result of the point cloud, wherein the point cloud reconstruction result includes the predicted number of predicted points in the reconstructed bird's-eye view grid and the predicted coordinates of each predicted point, and the color information reconstruction result of the point cloud includes the predicted RGB value of each predicted point in the bird's-eye view grid. The image decoder decodes the second image feature to obtain the image reconstruction result and the depth information reconstruction result of the image, wherein the image reconstruction result includes the predicted RGB value of each pixel, and the depth information reconstruction result of the image includes the predicted depth value of each pixel.

[0181] After obtaining the reconstruction results output by the decoder, the model loss value is calculated based on the difference between each reconstruction result and the true value corresponding to each reconstruction result. Then, the model parameters of the multimodal model are adjusted by back propagation to minimize the model loss value until the model converges and the training is completed.

[0182] It can be seen that through this scheme, on the basis of the single-modal reconstruction task, two cross-modal reconstruction tasks are designed, namely the color information reconstruction task of the point cloud and the depth information reconstruction task of the image, which can give full play to the complementarity of information between different modalities. Among them, the cross-modal reconstruction task of the color information of the point cloud can play the role of the image modality in supplementing the color information of the point cloud modality; the cross-modal reconstruction task of the depth information of the image can play the role of the point cloud modality in supplementing the depth information of the image modality. By using the perspective conversion and attention mechanism to interact cross-modally with the features of each modality, the cross-modal reconstruction task can be better performed and the convergence difficulty of the cross-modal reconstruction task can be reduced. After the model is pre-trained by the designed single-modal reconstruction tasks and cross-modal reconstruction tasks, all decoders in the multi-modal model can be removed, and other pre-trained parts can be used to fine-tune the model in downstream tasks to improve the performance of downstream tasks and reduce the dependence on labeled data. Among them, downstream tasks can include downstream tasks of laser perception, downstream tasks of image perception, and multi-modal perception tasks.

[0183] Corresponding to the embodiment of the first solution above, the embodiment of the present application further provides a point cloud processing method, such as Figure 4 As shown, steps S401-S403 may be included:

[0184] S401, obtaining a point cloud to be processed;

[0185] In this embodiment, the point cloud to be processed can be obtained from an electronic device storing point clouds, or collected in real time using a sensor device such as a laser radar, which is reasonable. For example, the point cloud to be processed can be a point cloud collected in real time by a laser device on an intelligent driving vehicle.

[0186] S402, inputting the point cloud to be processed into the first encoder of the multimodal model trained based on the first scheme to obtain a BEV (bird's eye view) feature to be used;

[0187] S403, performing target detection based on the BEV features to be utilized to obtain a detection result.

[0188] It can be understood that since the first encoder in the multimodal model trained according to the above-mentioned multimodal model pre-training method can accurately predict the RGB information in the input point cloud based on the distribution of points in the input point cloud, the point cloud to be processed is input into the first encoder obtained by the training, and the obtained BEV features to be used contain the distribution and color information in the input point cloud. Then, target detection based on the BEV features to be used can better understand the point cloud to be processed in combination with the distribution and color information in the point cloud contained in the BEV features to be used, so as to accurately identify and locate the target object in the point cloud to be processed, thereby obtaining accurate detection results. It should be noted that since the detection results obtained by target detection based on the BEV features to be used are more accurate, this solution can be applied to various point cloud perception tasks and multimodal perception tasks.

[0189] It can be seen that through this solution, the accuracy of the detection results of target detection on point clouds can be improved.

[0190] The present application embodiment also provides another multimodal model pre-training method, namely, the second solution, such as Figure 5 As shown, steps S501-S507 are included:

[0191] S501, obtaining a sample point cloud and a sample image corresponding to the same scene;

[0192] S502, voxelizing the sample point cloud to obtain a plurality of voxels, performing point cloud masking processing on some real points in the sample point cloud according to the bird's-eye view grid to which each voxel belongs to obtain a first mask point cloud, and performing image masking processing on the sample image to obtain a first mask image;

[0193] S503, using the first encoder in the multimodal model to perform feature extraction on the first mask point cloud to obtain features from a BEV (bird's eye view) perspective as first BEV features, and using the second encoder in the multimodal model to perform feature extraction on the first mask image to obtain first image features; wherein the multimodal model further includes a second converter, a second fusion network, a first decoder and a second decoder;

[0194] S504, using a second converter to convert the first BEV feature to an image perspective of the sample image to obtain a second conversion feature;

[0195] S505, using a second fusion network to perform feature fusion on the second conversion feature and the first image feature to obtain a second image feature;

[0196] S506, using the first decoder to decode the second BEV feature to obtain a first prediction result, and using the second decoder to decode the second image feature to obtain a third prediction result and a fourth prediction result; wherein the first prediction result represents: the predicted number of prediction points contained in the bird's-eye view grid and the predicted coordinates of each prediction point, the third prediction result represents: the predicted RGB value of each pixel in the sample image, and the fourth prediction result represents: the predicted depth value of each pixel in the sample image; the second BEV feature is obtained based on the first BEV feature;

[0197] S507, adjusting the model parameters of the multimodal model based on the difference between the first prediction result and the corresponding true value, the difference between the third prediction result and the corresponding true value, and the difference between the fourth prediction result and the corresponding true value; wherein the true value corresponding to the fourth prediction result is the depth value of the corresponding point of each pixel point of the sample image in the sample point cloud.

[0198] Optionally, the multimodal model further includes a first converter and a first fusion network;

[0199] Before using the first decoder to decode the second BEV feature to obtain the first prediction result, the method further includes:

[0200] Using a first converter, converting the first image feature to a BEV perspective to obtain a first conversion feature;

[0201] Using the first fusion network to fuse the first conversion feature and the first BEV feature to obtain a second BEV feature;

[0202] Decoding the second BEV feature using the first decoder to obtain a first prediction result includes:

[0203] Decoding the second BEV feature using the first decoder to obtain a first prediction result indicating the predicted number of prediction points included in the bird's-eye view grid and the predicted coordinates of each prediction point, and a second prediction result indicating the predicted RGB value of each prediction point in the bird's-eye view grid;

[0204] Adjusting a model parameter of the multimodal model based on a difference between the first prediction result and the corresponding true value, a difference between the third prediction result and the corresponding true value, and a difference between the fourth prediction result and the corresponding true value includes:

[0205] Based on the difference between the first prediction result and the corresponding true value, the difference between the second prediction result and the corresponding true value, the difference between the third prediction result and the corresponding true value, and the difference between the fourth prediction result and the corresponding true value, adjust the model parameters of the multimodal model; wherein the true value corresponding to the second prediction result is the RGB value of the corresponding pixel point in the sample image of the real point in the bird's-eye view grid.

[0206] Optionally, performing point cloud masking processing on some real points in the sample point cloud according to the bird's-eye view grid to which each voxel belongs to obtain a first mask point cloud, including: grouping a plurality of voxels to obtain voxels belonging to the same bird's-eye view grid; determining a bird's-eye view grid to be masked from each bird's-eye view grid according to a first ratio; replacing the coordinates of each real point contained in the determined bird's-eye view grid to be masked with the current predetermined coordinates to obtain the first mask point cloud;

[0207] and / or,

[0208] Performing image mask processing on a sample image to obtain a first mask image includes: dividing the sample image according to a predetermined size to obtain a plurality of image regions; determining an image region to be masked from the plurality of image regions according to a second ratio; and replacing the RGB values ​​of pixels in the determined image region to be masked with predetermined RGB values ​​to obtain the first mask image.

[0209] Optionally, the method further includes: adjusting the current predetermined coordinates based on the difference between the first prediction result and the corresponding true value, the difference between the third prediction result and the corresponding true value, and the difference between the fourth prediction result and the corresponding true value.

[0210] Optionally, the difference between the first prediction result and the corresponding true value includes: a first difference between the predicted number of predicted points contained in the bird's-eye view grid to be masked and the actual number of real points contained in the bird's-eye view grid to be masked, and a second difference between the predicted coordinates of each predicted point in the bird's-eye view grid to be masked and the actual coordinates of each real point in the bird's-eye view grid to be masked.

[0211] Optionally, the first difference is calculated using a smooth L1 loss function;

[0212] and / or,

[0213] The second difference is the sum of the first sub-differences corresponding to each bird's-eye view grid to be masked; the first sub-difference corresponding to any bird's-eye view grid to be masked is: calculated using the first chamfer distance loss function based on the predicted coordinates of each predicted point in the bird's-eye view grid to be masked and the real coordinates of each real point in the bird's-eye view grid to be masked.

[0214] Optionally, the first chamfer distance loss function is:

[0215] ;

[0216] in, is the first subdifference corresponding to the i-th bird's-eye view grid to be masked, L represents the predicted number of predicted points contained in the i-th bird's-eye view grid to be masked, Indicates the first grid in the i-th bird's-eye view to be masked The predicted coordinates of the predicted points, represents the predicted coordinates of each predicted point in the i-th grid of the bird's-eye view to be masked; N represents the actual number of real points contained in the i-th grid of the bird's-eye view to be masked, Indicates the first grid in the i-th bird's-eye view to be masked The real coordinates of a real point, Represents the real coordinates of each real point in the grid of the i-th bird's-eye view to be masked.

[0217] Optionally, the difference between the second prediction result and the corresponding true value is the sum of the second sub-differences corresponding to each bird's-eye view grid; the second sub-difference corresponding to any bird's-eye view grid is: calculated using the second chamfer distance loss function based on the predicted RGB value of each predicted point in the bird's-eye view grid and the RGB value of the corresponding pixel point of the real point in the bird's-eye view grid in the sample image.

[0218] Optionally, the second chamfer distance loss function is:

[0219]

[0220] in, represents the second sub-difference corresponding to the i-th bird's-eye view grid, L represents the predicted number of predicted points contained in the i-th bird's-eye view grid, Indicates the first The predicted RGB value of the prediction point, represents the predicted RGB value of each predicted point in the i-th bird's-eye view grid; N represents the actual number of real points contained in the i-th bird's-eye view grid, Indicates the first The RGB value of the pixel corresponding to the real point in the sample image, Represents the RGB value of the pixel point corresponding to each real point in the i-th bird's-eye view grid in the sample image.

[0221] Optionally, the first encoder includes a BEV encoder; and / or the first transformer is an LSSTransformer.

[0222] Optionally, the first fusion network is a first cross attention network; using the first fusion network to perform feature fusion on the first conversion feature and the first BEV feature to obtain a second BEV feature, including: mapping the first BEV feature to obtain a query Q in the input data of the first cross attention network, and mapping the first conversion feature to obtain a key K and a value V in the input data of the first cross attention network, and inputting the obtained Q, K and V into the first cross attention network for feature fusion to obtain the second BEV feature;

[0223] and / or,

[0224] The second fusion network is a second cross-attention network; the second fusion network is used to perform feature fusion on the second conversion feature and the first image feature to obtain a second image feature, including: mapping the first image feature to obtain a query Q in the input data of the second cross-attention network, and mapping the second conversion feature to obtain a key K and a value V in the input data of the second cross-attention network, and inputting the obtained Q, K and V into the second cross-attention network for feature fusion to obtain a second image feature.

[0225] Optionally, the difference between the third prediction result and the corresponding true value is calculated using a mean square error loss function based on the predicted RGB value of each pixel in the sample image and the true RGB value of each pixel in the sample image;

[0226] and / or,

[0227] The difference between the fourth prediction result and the corresponding true value is: calculated using the mean square error loss function based on the predicted depth value of each pixel in the sample image and the depth value of the corresponding point of each pixel of the sample image in the sample point cloud.

[0228] Optionally, the second converter is used to convert the first BEV feature to the image perspective of the sample image based on the intrinsic parameters, extrinsic parameters of the camera that captures the sample image and the conversion relationship between the point cloud coordinate system of the sample point cloud and the camera coordinate system of the camera.

[0229] It should be noted that the specific implementation of each step in the embodiment of the present application can refer to the above Figure 1 and Figure 2 Description of the illustrated embodiment.

[0230] The solution provided by the embodiment of the present application adjusts the model parameters based on the difference between the first prediction result and the corresponding true value, so that the model can learn the distribution of points in the point cloud, including the number of points and position information, from the sample point cloud. Adjust the model parameters based on the difference between the third prediction result and the corresponding true value, so that the model can learn color information from the sample image. Taking the depth value of the corresponding point of each pixel of the sample image in the sample point cloud as the true value corresponding to the fourth prediction result can give full play to the supplementary effect of the depth information in the sample point cloud on the color information of the image. Then, based on the difference between the fourth prediction result and the corresponding true value, the model parameters are adjusted so that the model can make full use of the depth information in the sample point cloud to learn the depth information of the image, so that the second encoder in the trained multimodal model can accurately predict the depth information in the input image based on the color information in the input image. Thus, the processing effect of the encoder in the trained multimodal model in processing downstream tasks can be improved.

[0231] Corresponding to the embodiment of the second solution above, the embodiment of the present application further provides an image processing method, such as Figure 6 As shown, steps S601-S603 may be included:

[0232] S601, obtaining an image to be processed;

[0233] In this embodiment, the point cloud to be processed can be obtained from an electronic device storing images, or collected in real time by a camera, which is reasonable. For example, the image to be processed can be an image collected in real time by a vehicle camera on an intelligent driving vehicle.

[0234] S602, inputting the image to be processed into the second encoder in the multimodal model trained based on the second solution to obtain image features to be used;

[0235] S603: Perform target detection based on the image features to be utilized to obtain a detection result.

[0236] It can be understood that since the second encoder in the multimodal model trained according to the above-mentioned second scheme can accurately predict the depth information in the input image based on the color information in the input image, when the image to be processed is input into the trained second encoder, the obtained image features to be used contain the color and depth information of the input image. Then, target detection based on the image features to be used can better understand the image to be processed in combination with the color and depth information contained in the image features to be used, so as to accurately identify and locate the target object in the image to be processed, thereby obtaining accurate detection results. It should be noted that since the detection results obtained by target detection based on the image features to be used are more accurate, this scheme can be applied to various image perception tasks and multimodal perception tasks.

[0237] It can be seen that through this solution, the accuracy of the detection results of target detection on the image can be improved.

[0238] Corresponding to the embodiment of the first solution above, the embodiment of the present application also provides a multimodal model pre-training device, such as Figure 7 As shown, the device comprises:

[0239] A first acquisition module 710, used to acquire a sample point cloud and a sample image corresponding to the same scene;

[0240] A first mask module 720 is used to voxelize the sample point cloud to obtain a plurality of voxels, perform point cloud masking on some real points in the sample point cloud according to the bird's-eye view grid to which each voxel belongs to obtain a first mask point cloud, and perform image masking on the sample image to obtain a first mask image;

[0241] A first feature extraction module 730 is used to extract features from the first mask point cloud using a first encoder in the multimodal model to obtain features from a BEV perspective as first BEV features, and to extract features from the first mask image using a second encoder in the multimodal model to obtain first image features; the multimodal model further includes a first converter, a first fusion network, a first decoder, and a second decoder;

[0242] A first conversion module 740, configured to convert the first image feature to the BEV perspective using the first converter to obtain a first conversion feature;

[0243] A first fusion module 750, configured to use the first fusion network to perform feature fusion on the first conversion feature and the first BEV feature to obtain a second BEV feature;

[0244] The first decoding module 760 is used to decode the second BEV feature using the first decoder to obtain a first prediction result and a second prediction result, and to decode the second image feature using the second decoder to obtain a third prediction result; wherein the first prediction result represents: the predicted number of prediction points contained in the bird's-eye view grid and the predicted coordinates of each prediction point, the second prediction result represents: the predicted RGB value of each prediction point in the bird's-eye view grid, and the third prediction result represents: the predicted RGB value of each pixel in the sample image; the second image feature is obtained based on the first image feature;

[0245] The first adjustment module 770 is used to adjust the model parameters of the multimodal model based on the difference between the first prediction result and the corresponding true value, the difference between the second prediction result and the corresponding true value, and the difference between the third prediction result and the corresponding true value; wherein the true value corresponding to the second prediction result is the RGB value of the corresponding pixel point in the sample image of the real point in the bird's-eye view grid.

[0246] Optionally, the multimodal model further includes a second converter and a second fusion network;

[0247] The device also includes:

[0248] A second conversion module, configured to convert the first BEV feature to the image perspective of the sample image using the second converter to obtain a second conversion feature before the first decoding module 760 decodes the second image feature using the second decoder to obtain a third prediction result;

[0249] A second fusion module, configured to use the second fusion network to perform feature fusion on the second conversion feature and the first image feature to obtain a second image feature;

[0250] The first decoding module 760 is specifically configured to:

[0251] Decoding the second image feature using the second decoder to obtain a third prediction result representing a predicted RGB value of each pixel in the sample image and a fourth prediction result representing a predicted depth value of each pixel in the sample image;

[0252] The first adjustment module 770 is specifically used to:

[0253] Based on the difference between the first prediction result and the corresponding true value, the difference between the second prediction result and the corresponding true value, the difference between the third prediction result and the corresponding true value, and the difference between the fourth prediction result and the corresponding true value, adjust the model parameters of the multimodal model; wherein the true value corresponding to the fourth prediction result is the depth value of the corresponding point of each pixel point of the sample image in the sample point cloud.

[0254] Optionally, the first mask module 720 is specifically configured to:

[0255] Grouping multiple voxels to obtain voxels belonging to the same bird's-eye view grid; determining a to-be-masked bird's-eye view grid from each bird's-eye view grid according to a first ratio; replacing the coordinates of each real point included in the determined to-be-masked bird's-eye view grid with the current predetermined coordinates to obtain a first mask point cloud;

[0256] and / or,

[0257] The sample image is divided according to a predetermined size to obtain a plurality of image regions; an image region to be masked is determined from the plurality of image regions according to a second ratio; and RGB values ​​of pixels in the determined image region to be masked are replaced with predetermined RGB values ​​to obtain a first mask image.

[0258] Optionally, the device further comprises:

[0259] A coordinate adjustment module is used to adjust the current predetermined coordinates based on the difference between the first prediction result and the corresponding true value, the difference between the second prediction result and the corresponding true value, and the difference between the third prediction result and the corresponding true value.

[0260] Optionally, the difference between the first prediction result and the corresponding true value includes:

[0261] a first difference between the predicted number of predicted points contained in the to-be-masked bird's-eye view grid and the actual number of actual points contained in the to-be-masked bird's-eye view grid, and a second difference between the predicted coordinates of each predicted point in the to-be-masked bird's-eye view grid and the actual coordinates of each actual point in the to-be-masked bird's-eye view grid.

[0262] Optionally, the first difference is calculated using a smooth L1 loss function;

[0263] and / or,

[0264] The second difference is the sum of the first sub-differences corresponding to each bird's-eye view grid to be masked; the first sub-difference corresponding to any bird's-eye view grid to be masked is: calculated using the first chamfer distance loss function based on the predicted coordinates of each predicted point in the bird's-eye view grid to be masked and the real coordinates of each real point in the bird's-eye view grid to be masked.

[0265] Optionally, the first chamfer distance loss function is:

[0266] ;

[0267] in, is the first sub-difference corresponding to the i-th bird's-eye view grid to be masked, L represents the predicted number of predicted points contained in the i-th bird's-eye view grid to be masked, Indicates the first grid in the i-th bird's-eye view to be masked The predicted coordinates of the predicted points, represents the predicted coordinates of each predicted point in the i-th grid of the bird's-eye view to be masked; N represents the actual number of real points contained in the i-th grid of the bird's-eye view to be masked, Indicates the first grid in the i-th bird's-eye view to be masked The real coordinates of a real point, Represents the real coordinates of each real point in the grid of the i-th bird's-eye view to be masked.

[0268] Optionally, the difference between the second prediction result and the corresponding true value is the sum of the second sub-differences corresponding to each bird's-eye view grid; the second sub-difference corresponding to any bird's-eye view grid is: calculated using the second chamfer distance loss function based on the predicted RGB value of each predicted point in the bird's-eye view grid and the RGB value of the corresponding pixel point of the real point in the bird's-eye view grid in the sample image.

[0269] Optionally, the second chamfer distance loss function is:

[0270] ;

[0271] in, represents the second sub-difference corresponding to the i-th bird's-eye view grid, L represents the predicted number of predicted points contained in the i-th bird's-eye view grid, Indicates the first The predicted RGB value of the prediction point, represents the predicted RGB value of each predicted point in the i-th bird's-eye view grid; N represents the actual number of real points contained in the i-th bird's-eye view grid, Indicates the first The RGB value of the pixel corresponding to the real point in the sample image, Represents the RGB value of the pixel point corresponding to each real point in the i-th bird's-eye view grid in the sample image.

[0272] Optionally, the first encoder includes a BEV encoder; and / or the first transformer is an LSSTransformer.

[0273] Optionally, the first fusion network is a first cross attention network; the first fusion module 750 is specifically used to: map the first BEV feature to obtain a query Q in the input data of the first cross attention network, and map the first conversion feature to obtain a key K and a value V in the input data of the first cross attention network, and input the obtained Q, K and V into the first cross attention network for feature fusion to obtain a second BEV feature;

[0274] and / or,

[0275] The second fusion network is a second cross-attention network; the second fusion module is specifically used to: map the first image feature to obtain the query Q in the input data of the second cross-attention network, and map the second conversion feature to obtain the key K and value V in the input data of the second cross-attention network, and input the obtained Q, K and V into the second cross-attention network for feature fusion to obtain the second image feature.

[0276] Optionally, the difference between the third prediction result and the corresponding true value is calculated using a mean square error loss function based on the predicted RGB value of each pixel in the sample image and the true RGB value of each pixel in the sample image;

[0277] and / or,

[0278] The difference between the fourth prediction result and the corresponding true value is: calculated using the mean square error loss function based on the predicted depth value of each pixel in the sample image and the depth value of the corresponding point of each pixel of the sample image in the sample point cloud.

[0279] Optionally, the second converter is used to convert the first BEV feature to the image perspective of the sample image based on the intrinsic parameters, extrinsic parameters of the camera that captures the sample image and the conversion relationship between the point cloud coordinate system of the sample point cloud and the camera coordinate system of the camera.

[0280] Corresponding to the above-mentioned point cloud processing method embodiment, the present application embodiment also provides a point cloud processing device, such as Figure 8 As shown, the device comprises:

[0281] The point cloud acquisition module 810 is used to acquire the point cloud to be processed;

[0282] A point cloud input module 820 is used to input the point cloud to be processed into a first encoder in the multimodal model trained based on the first scheme to obtain a BEV feature to be used;

[0283] The point cloud detection module 830 is used to perform target detection based on the BEV features to be utilized to obtain a detection result.

[0284] Corresponding to the above-mentioned another embodiment of the multimodal model pre-training method, the present application embodiment also provides a multimodal model pre-training device, such as Fig. 9 As shown, the device comprises:

[0285] A second acquisition module 910, used to acquire a sample point cloud and a sample image corresponding to the same scene;

[0286] A second mask module 920 is used to voxelize the sample point cloud to obtain a plurality of voxels, perform point cloud masking on some real points in the sample point cloud according to the bird's-eye view grid to which each voxel belongs to obtain a first mask point cloud, and perform image masking on the sample image to obtain a first mask image;

[0287] A second feature extraction module 930 is used to extract features from the first mask point cloud using the first encoder in the multimodal model to obtain features from the BEV perspective as first BEV features, and to extract features from the first mask image using the second encoder in the multimodal model to obtain first image features; wherein the multimodal model further includes a second converter, a second fusion network, a first decoder and a second decoder;

[0288] A second conversion module 940 is used to convert the first BEV feature to the image perspective of the sample image using the second converter to obtain a second conversion feature;

[0289] A second fusion module 950 is used to perform feature fusion on the second conversion feature and the first image feature by using the second fusion network to obtain a second image feature;

[0290] The second decoding module 960 is used to decode the second BEV feature using the first decoder to obtain a first prediction result, and decode the second image feature using the second decoder to obtain a third prediction result and a fourth prediction result; wherein the first prediction result represents: the predicted number of prediction points contained in the bird's-eye view grid and the predicted coordinates of each prediction point, the third prediction result represents: the predicted RGB value of each pixel in the sample image, and the fourth prediction result represents: the predicted depth value of each pixel in the sample image; the second BEV feature is obtained based on the first BEV feature;

[0291] The second adjustment module 970 is used to adjust the model parameters of the multimodal model based on the difference between the first prediction result and the corresponding true value, the difference between the third prediction result and the corresponding true value, and the difference between the fourth prediction result and the corresponding true value; wherein the true value corresponding to the fourth prediction result is the depth value of the corresponding point of each pixel point of the sample image in the sample point cloud.

[0292] Optionally, the multimodal model further includes a first converter and a first fusion network;

[0293] The device also includes:

[0294] A first conversion module, configured to convert the first image feature to the BEV perspective by using the first converter to obtain a first conversion feature before the second decoding module 960 decodes the second BEV feature by using the first decoder to obtain a first prediction result;

[0295] A first fusion module, configured to use the first fusion network to perform feature fusion on the first conversion feature and the first BEV feature to obtain a second BEV feature;

[0296] The second decoding module 960 is specifically configured to:

[0297] Decoding the second BEV feature using the first decoder to obtain a first prediction result indicating the predicted number of prediction points included in the bird's-eye view grid and the predicted coordinates of each prediction point, and a second prediction result indicating the predicted RGB value of each prediction point in the bird's-eye view grid;

[0298] The second adjustment module 970 is specifically used to:

[0299] Based on the difference between the first prediction result and the corresponding true value, the difference between the second prediction result and the corresponding true value, the difference between the third prediction result and the corresponding true value, and the difference between the fourth prediction result and the corresponding true value, adjust the model parameters of the multimodal model; wherein the true value corresponding to the second prediction result is the RGB value of the corresponding pixel point in the sample image of the real point in the bird's-eye view grid.

[0300] Optionally, the second mask module 920 is specifically configured to:

[0301] Grouping multiple voxels to obtain voxels belonging to the same bird's-eye view grid; determining a to-be-masked bird's-eye view grid from each bird's-eye view grid according to a first ratio; replacing the coordinates of each real point included in the determined to-be-masked bird's-eye view grid with the current predetermined coordinates to obtain a first mask point cloud;

[0302] and / or,

[0303] The sample image is divided according to a predetermined size to obtain a plurality of image regions; an image region to be masked is determined from the plurality of image regions according to a second ratio; and RGB values ​​of pixels in the determined image region to be masked are replaced with predetermined RGB values ​​to obtain a first mask image.

[0304] Optionally, the device further comprises:

[0305] A coordinate adjustment module is used to adjust the current predetermined coordinates based on the difference between the first prediction result and the corresponding true value, the difference between the third prediction result and the corresponding true value, and the difference between the fourth prediction result and the corresponding true value.

[0306] Optionally, the difference between the first prediction result and the corresponding true value includes:

[0307] a first difference between the predicted number of predicted points contained in the to-be-masked bird's-eye view grid and the actual number of actual points contained in the to-be-masked bird's-eye view grid, and a second difference between the predicted coordinates of each predicted point in the to-be-masked bird's-eye view grid and the actual coordinates of each actual point in the to-be-masked bird's-eye view grid.

[0308] Optionally, the first difference is calculated using a smooth L1 loss function;

[0309] and / or,

[0310] The second difference is the sum of the first sub-differences corresponding to each bird's-eye view grid to be masked; the first sub-difference corresponding to any bird's-eye view grid to be masked is: calculated using the first chamfer distance loss function based on the predicted coordinates of each predicted point in the bird's-eye view grid to be masked and the real coordinates of each real point in the bird's-eye view grid to be masked.

[0311] Optionally, the first chamfer distance loss function is:

[0312] ;

[0313] in, is the first subdifference corresponding to the i-th bird's-eye view grid to be masked, L represents the predicted number of predicted points contained in the i-th bird's-eye view grid to be masked, Indicates the first grid in the i-th bird's-eye view to be masked The predicted coordinates of the predicted points, represents the predicted coordinates of each predicted point in the i-th grid of the bird's-eye view to be masked; N represents the actual number of real points contained in the i-th grid of the bird's-eye view to be masked, Indicates the first grid in the i-th bird's-eye view to be masked The real coordinates of a real point, Represents the real coordinates of each real point in the grid of the i-th bird's-eye view to be masked.

[0314] Optionally, the difference between the second prediction result and the corresponding true value is the sum of the second sub-differences corresponding to each bird's-eye view grid; the second sub-difference corresponding to any bird's-eye view grid is: calculated using the second chamfer distance loss function based on the predicted RGB value of each predicted point in the bird's-eye view grid and the RGB value of the corresponding pixel point of the real point in the bird's-eye view grid in the sample image.

[0315] Optionally, the second chamfer distance loss function is:

[0316] ;

[0317] in, represents the second sub-difference corresponding to the i-th bird's-eye view grid, L represents the predicted number of predicted points contained in the i-th bird's-eye view grid, Indicates the first The predicted RGB value of the prediction point, represents the predicted RGB value of each predicted point in the i-th bird's-eye view grid; N represents the actual number of real points contained in the i-th bird's-eye view grid, Indicates the first The RGB value of the pixel corresponding to the real point in the sample image, Represents the RGB value of the pixel point corresponding to each real point in the i-th bird's-eye view grid in the sample image.

[0318] Optionally, the first encoder includes a BEV encoder; and / or the first transformer is an LSSTransformer.

[0319] Optionally, the first fusion network is a first cross attention network; the first fusion module is specifically used to: map the first BEV feature to obtain a query Q in the input data of the first cross attention network, and map the first conversion feature to obtain a key K and a value V in the input data of the first cross attention network, and input the obtained Q, K and V into the first cross attention network for feature fusion to obtain a second BEV feature;

[0320] and / or,

[0321] The second fusion network is a second cross-attention network; the second fusion module 950 is specifically used to: map the first image feature to obtain the query Q in the input data of the second cross-attention network, and map the second conversion feature to obtain the key K and value V in the input data of the second cross-attention network, and input the obtained Q, K and V into the second cross-attention network for feature fusion to obtain the second image feature.

[0322] Optionally, the difference between the third prediction result and the corresponding true value is calculated using a mean square error loss function based on the predicted RGB value of each pixel in the sample image and the true RGB value of each pixel in the sample image;

[0323] and / or,

[0324] The difference between the fourth prediction result and the corresponding true value is: calculated using the mean square error loss function based on the predicted depth value of each pixel in the sample image and the depth value of the corresponding point of each pixel of the sample image in the sample point cloud.

[0325] Optionally, the second converter is used to convert the first BEV feature to the image perspective of the sample image based on the intrinsic parameters, extrinsic parameters of the camera that captures the sample image and the conversion relationship between the point cloud coordinate system of the sample point cloud and the camera coordinate system of the camera.

[0326] Corresponding to the above-mentioned embodiment of the image processing method, the embodiment of the present application further provides an image processing device, such as Fig.10 As shown, the device comprises:

[0327] An image acquisition module 1010 is used to acquire an image to be processed;

[0328] An image input module 1020 is used to input the image to be processed into a second encoder in the multimodal model trained based on the second scheme to obtain image features to be used;

[0329] The image detection module 1030 is used to perform target detection based on the image features to be utilized to obtain a detection result.

[0330] In the technical solution of this application, the operations involved in the acquisition, storage, use, processing, transmission, provision and disclosure of point clouds and images are all carried out with the user's authorization.

[0331] The present application also provides an electronic device, such as Fig.11As shown, it includes: a memory 1101 for storing computer programs; a processor 1102, for implementing the steps of any of the multimodal model pre-training methods, point cloud processing methods or image processing methods described above when executing the programs stored in the memory 1101.

[0332] In addition, the electronic device may further include a communication bus and / or a communication interface, and the processor 1102, the communication interface, and the memory 1101 communicate with each other via the communication bus.

[0333] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0334] The communication interface is used for communication between the electronic device and other devices. The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the processor.

[0335] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0336] In another embodiment provided in the present application, a computer-readable storage medium is also provided, which stores a computer program. When the computer program is executed by a processor, the steps of any of the above-mentioned multimodal model pre-training methods, point cloud processing methods or image processing methods are implemented.

[0337] In another embodiment provided in the present application, a computer program product comprising instructions is also provided, which, when executed on a computer, enables the computer to execute any multimodal model pre-training method, point cloud processing method or image processing method in the above-mentioned embodiments.

[0338] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a solid-state hard disk (SSD), etc.

[0339] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0340] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0341] The above description is only a preferred embodiment of the present application and is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application are included in the protection scope of the present application.

Claims

1. A multimodal model pre-training method, characterized in that: The method comprises: Obtaining sample point clouds and sample images corresponding to the same scene; voxelize the sample point cloud to obtain a plurality of voxels, perform point cloud masking on some real points in the sample point cloud according to the bird's-eye view grid to which each voxel belongs to obtain a first mask point cloud, and perform image masking on the sample image to obtain a first mask image; Using a first encoder in a multimodal model to perform feature extraction on the first mask point cloud to obtain features from a bird's-eye view (BEV) as first BEV features, and using a second encoder in the multimodal model to perform feature extraction on the first mask image to obtain first image features; wherein the multimodal model further includes a first converter, a first fusion network, a first decoder, and a second decoder; Using the first converter, converting the first image feature to the BEV perspective to obtain a first conversion feature; Using the first fusion network to perform feature fusion on the first conversion feature and the first BEV feature to obtain a second BEV feature; The second BEV feature is decoded by the first decoder to obtain a first prediction result and a second prediction result, and the second image feature is decoded by the second decoder to obtain a third prediction result; wherein the first prediction result represents: the predicted number of prediction points contained in the bird's-eye view grid and the predicted coordinates of each prediction point, the second prediction result represents: the predicted RGB value of each prediction point in the bird's-eye view grid, and the third prediction result represents: the predicted RGB value of each pixel in the sample image; the second image feature is obtained based on the first image feature; Based on the difference between the first prediction result and the corresponding true value, the difference between the second prediction result and the corresponding true value, and the difference between the third prediction result and the corresponding true value, the model parameters of the multimodal model are adjusted; wherein the true value corresponding to the second prediction result is the RGB value of the corresponding pixel point in the sample image of the real point in the bird's-eye view grid.

2. The method according to claim 1, characterized in that The multimodal model also includes a second converter and a second fusion network; Before using the second decoder to decode the second image feature to obtain a third prediction result, the method further includes: Using the second converter, converting the first BEV feature to the image perspective of the sample image to obtain a second conversion feature; Using the second fusion network to perform feature fusion on the second conversion feature and the first image feature to obtain a second image feature; The using the second decoder to decode the second image feature to obtain a third prediction result includes: Decoding the second image feature using the second decoder to obtain a third prediction result representing a predicted RGB value of each pixel in the sample image and a fourth prediction result representing a predicted depth value of each pixel in the sample image; The adjusting the model parameters of the multimodal model based on the difference between the first prediction result and the corresponding true value, the difference between the second prediction result and the corresponding true value, and the difference between the third prediction result and the corresponding true value includes: Based on the difference between the first prediction result and the corresponding true value, the difference between the second prediction result and the corresponding true value, the difference between the third prediction result and the corresponding true value, and the difference between the fourth prediction result and the corresponding true value, adjust the model parameters of the multimodal model; wherein the true value corresponding to the fourth prediction result is the depth value of the corresponding point of each pixel point of the sample image in the sample point cloud.

3. The method according to claim 1 or 2, characterized in that: The step of performing point cloud masking on some real points in the sample point cloud according to the bird's-eye view grid to which each voxel belongs to obtain a first mask point cloud, including: grouping a plurality of voxels to obtain voxels belonging to the same bird's-eye view grid; determining a bird's-eye view grid to be masked from each bird's-eye view grid according to a first ratio; and replacing the coordinates of each real point contained in the determined bird's-eye view grid to be masked with the current predetermined coordinates to obtain a first mask point cloud; and / or, The sample image is subjected to image mask processing to obtain a first mask image, including: dividing the sample image according to a predetermined size to obtain a plurality of image regions; determining an image region to be masked from the plurality of image regions according to a second ratio; and replacing the RGB values ​​of pixels in the determined image region to be masked with predetermined RGB values ​​to obtain the first mask image.

4. The method according to claim 3, characterized in that The method further comprises: Based on the difference between the first prediction result and the corresponding true value, the difference between the second prediction result and the corresponding true value, and the difference between the third prediction result and the corresponding true value, the current predetermined coordinates are adjusted.

5. The method according to claim 3, characterized in that: The difference between the first prediction result and the corresponding true value includes: a first difference between the predicted number of predicted points contained in the to-be-masked bird's-eye view grid and the actual number of actual points contained in the to-be-masked bird's-eye view grid, and a second difference between the predicted coordinates of each predicted point in the to-be-masked bird's-eye view grid and the actual coordinates of each actual point in the to-be-masked bird's-eye view grid.

6. The method according to claim 5, characterized in that The first difference is calculated using a smooth L1 loss function; and / or, The second difference is the sum of the first sub-differences corresponding to each grid of the bird's-eye view to be masked; The first sub-difference corresponding to any unmasked bird's-eye view grid is calculated based on the predicted coordinates of each predicted point in the unmasked bird's-eye view grid and the real coordinates of each real point in the unmasked bird's-eye view grid using the first chamfer distance loss function.

7. The method according to claim 6, characterized in that The first chamfer distance loss function is: ; in, is the first subdifference corresponding to the i-th bird's-eye view grid to be masked, L represents the predicted number of predicted points contained in the i-th bird's-eye view grid to be masked, Indicates the first grid in the i-th bird's-eye view to be masked The predicted coordinates of the predicted points, represents the predicted coordinates of each predicted point in the i-th grid of the bird's-eye view to be masked; N represents the actual number of real points contained in the i-th grid of the bird's-eye view to be masked, Indicates the first grid in the i-th bird's-eye view to be masked The real coordinates of a real point, Represents the real coordinates of each real point in the grid of the i-th bird's-eye view to be masked.

8. The method according to claim 3, characterized in that The difference between the second prediction result and the corresponding true value is the sum of the second sub-differences corresponding to each bird's-eye view grid; the second sub-difference corresponding to any bird's-eye view grid is: calculated using the second chamfer distance loss function based on the predicted RGB value of each predicted point in the bird's-eye view grid and the RGB value of the corresponding pixel point of the real point in the bird's-eye view grid in the sample image.

9. The method according to claim 8, characterized in that The second chamfer distance loss function is: ; in, represents the second sub-difference corresponding to the i-th bird's-eye view grid, L represents the predicted number of predicted points contained in the i-th bird's-eye view grid, Indicates the first The predicted RGB value of the prediction point, represents the predicted RGB value of each predicted point in the i-th bird's-eye view grid; N represents the actual number of real points contained in the i-th bird's-eye view grid, Indicates the first The RGB value of the pixel corresponding to the real point in the sample image, Represents the RGB value of the pixel point corresponding to each real point in the i-th bird's-eye view grid in the sample image.

10. The method according to claim 1 or 2, characterized in that: The first encoder comprises a BEV encoder; and / or, The first transformer is a lift-squat-shoot transformer (LSS Transformer).

11. The method according to claim 2, characterized in that The first fusion network is a first cross attention network; using the first fusion network to perform feature fusion on the first conversion feature and the first BEV feature to obtain a second BEV feature, including: mapping the first BEV feature to obtain a query Q in the input data of the first cross attention network, and mapping the first conversion feature to obtain a key K and a value V in the input data of the first cross attention network, and inputting the obtained Q, K and V into the first cross attention network for feature fusion to obtain a second BEV feature; and / or, The second fusion network is a second cross-attention network; the second fusion network is used to perform feature fusion on the second conversion feature and the first image feature to obtain a second image feature, including: mapping the first image feature to obtain a query Q in the input data of the second cross-attention network, and mapping the second conversion feature to obtain a key K and a value V in the input data of the second cross-attention network, and inputting the obtained Q, K and V into the second cross-attention network for feature fusion to obtain a second image feature.

12. The method according to claim 2, characterized in that: The difference between the third prediction result and the corresponding true value is calculated based on the predicted RGB value of each pixel in the sample image and the true RGB value of each pixel in the sample image using a mean square error loss function; and / or, The difference between the fourth prediction result and the corresponding true value is: calculated using the mean square error loss function based on the predicted depth value of each pixel in the sample image and the depth value of the corresponding point of each pixel of the sample image in the sample point cloud.

13. The method according to claim 2, characterized in that The second converter is used to convert the first BEV feature to the image perspective of the sample image based on the intrinsic parameters, extrinsic parameters of the camera that collects the sample image and the conversion relationship between the point cloud coordinate system of the sample point cloud and the camera coordinate system of the camera.

14. A point cloud processing method, characterized in that: The method comprises: Get the point cloud to be processed; Inputting the point cloud to be processed into a first encoder of a multimodal model trained by the method according to any one of claims 1 to 13 to obtain a BEV feature to be used; Target detection is performed based on the BEV features to be utilized to obtain a detection result.

15. A multimodal model pre-training method, characterized in that: The method comprises: Obtaining sample point clouds and sample images corresponding to the same scene; voxelize the sample point cloud to obtain a plurality of voxels, perform point cloud masking on some real points in the sample point cloud according to the bird's-eye view grid to which each voxel belongs to obtain a first mask point cloud, and perform image masking on the sample image to obtain a first mask image; Using a first encoder in a multimodal model to perform feature extraction on the first mask point cloud to obtain features from a BEV perspective as first BEV features, and using a second encoder in the multimodal model to perform feature extraction on the first mask image to obtain first image features; wherein the multimodal model further includes a second converter, a second fusion network, a first decoder, and a second decoder; Using the second converter, converting the first BEV feature to the image perspective of the sample image to obtain a second conversion feature; Using the second fusion network to perform feature fusion on the second conversion feature and the first image feature to obtain a second image feature; The first decoder is used to decode the second BEV feature to obtain a first prediction result, and the second decoder is used to decode the second image feature to obtain a third prediction result and a fourth prediction result; wherein the first prediction result represents: the predicted number of prediction points contained in the bird's-eye view grid and the predicted coordinates of each prediction point, the third prediction result represents: the predicted RGB value of each pixel in the sample image, and the fourth prediction result represents: the predicted depth value of each pixel in the sample image; the second BEV feature is obtained based on the first BEV feature; Based on the difference between the first prediction result and the corresponding true value, the difference between the third prediction result and the corresponding true value, and the difference between the fourth prediction result and the corresponding true value, the model parameters of the multimodal model are adjusted; wherein the true value corresponding to the fourth prediction result is the depth value of the corresponding point of each pixel point of the sample image in the sample point cloud.

16. The method according to claim 15, characterized in that The multimodal model also includes a first converter and a first fusion network; Before using the first decoder to decode the second BEV feature to obtain a first prediction result, the method further includes: Using the first converter, converting the first image feature to the BEV perspective to obtain a first conversion feature; Using the first fusion network to perform feature fusion on the first conversion feature and the first BEV feature to obtain a second BEV feature; The using the first decoder to decode the second BEV feature to obtain a first prediction result includes: Decoding the second BEV feature using the first decoder to obtain a first prediction result indicating the predicted number of prediction points included in the bird's-eye view grid and the predicted coordinates of each prediction point, and a second prediction result indicating the predicted RGB value of each prediction point in the bird's-eye view grid; The adjusting the model parameters of the multimodal model based on the difference between the first prediction result and the corresponding true value, the difference between the third prediction result and the corresponding true value, and the difference between the fourth prediction result and the corresponding true value includes: Based on the difference between the first prediction result and the corresponding true value, the difference between the second prediction result and the corresponding true value, the difference between the third prediction result and the corresponding true value, and the difference between the fourth prediction result and the corresponding true value, adjust the model parameters of the multimodal model; wherein the true value corresponding to the second prediction result is the RGB value of the corresponding pixel point in the sample image of the real point in the bird's-eye view grid.

17. The method according to claim 16, characterized in that The step of performing point cloud masking on some real points in the sample point cloud according to the bird's-eye view grid to which each voxel belongs to obtain a first mask point cloud, including: grouping a plurality of voxels to obtain voxels belonging to the same bird's-eye view grid; determining a bird's-eye view grid to be masked from each bird's-eye view grid according to a first ratio; and replacing the coordinates of each real point contained in the determined bird's-eye view grid to be masked with the current predetermined coordinates to obtain a first mask point cloud; and / or, The sample image is subjected to image mask processing to obtain a first mask image, including: dividing the sample image according to a predetermined size to obtain a plurality of image regions; determining an image region to be masked from the plurality of image regions according to a second ratio; and replacing the RGB values ​​of pixels in the determined image region to be masked with predetermined RGB values ​​to obtain the first mask image.

18. The method according to claim 17, characterized in that The method further comprises: Based on the difference between the first prediction result and the corresponding true value, the difference between the third prediction result and the corresponding true value, and the difference between the fourth prediction result and the corresponding true value, the current predetermined coordinates are adjusted.

19. The method according to claim 17, characterized in that The difference between the first prediction result and the corresponding true value includes: a first difference between the predicted number of predicted points contained in the to-be-masked bird's-eye view grid and the actual number of actual points contained in the to-be-masked bird's-eye view grid, and a second difference between the predicted coordinates of each predicted point in the to-be-masked bird's-eye view grid and the actual coordinates of each actual point in the to-be-masked bird's-eye view grid.

20. The method according to claim 19, characterized in that The first difference is calculated using a smooth L1 loss function; and / or, The second difference is the sum of the first sub-differences corresponding to each grid of the bird's-eye view to be masked; The first sub-difference corresponding to any unmasked bird's-eye view grid is calculated based on the predicted coordinates of each predicted point in the unmasked bird's-eye view grid and the real coordinates of each real point in the unmasked bird's-eye view grid using the first chamfer distance loss function.

21. The method according to claim 20, characterized in that The first chamfer distance loss function is: ; in, is the first subdifference corresponding to the i-th bird's-eye view grid to be masked, L represents the predicted number of predicted points contained in the i-th bird's-eye view grid to be masked, Indicates the first grid in the i-th bird's-eye view to be masked The predicted coordinates of the predicted points, represents the predicted coordinates of each predicted point in the i-th grid of the bird's-eye view to be masked; N represents the actual number of real points contained in the i-th grid of the bird's-eye view to be masked, Indicates the first grid in the i-th bird's-eye view to be masked The real coordinates of a real point, Represents the real coordinates of each real point in the grid of the i-th bird's-eye view to be masked.

22. The method according to claim 17, characterized in that The difference between the second prediction result and the corresponding true value is the sum of the second sub-differences corresponding to each bird's-eye view grid; the second sub-difference corresponding to any bird's-eye view grid is: calculated using the second chamfer distance loss function based on the predicted RGB value of each predicted point in the bird's-eye view grid and the RGB value of the corresponding pixel point of the real point in the bird's-eye view grid in the sample image.

23. The method according to claim 22, characterized in that The second chamfer distance loss function is: ; in, represents the second sub-difference corresponding to the i-th bird's-eye view grid, L represents the predicted number of predicted points contained in the i-th bird's-eye view grid, Indicates the first The predicted RGB value of the prediction point, represents the predicted RGB value of each predicted point in the i-th bird's-eye view grid; N represents the actual number of real points contained in the i-th bird's-eye view grid, Indicates the first The RGB value of the pixel corresponding to the real point in the sample image, Represents the RGB value of the pixel point corresponding to each real point in the i-th bird's-eye view grid in the sample image.

24. The method according to claim 16, characterized in that The first encoder comprises a BEV encoder; and / or, The first transformer is a lift-squat-shoot transformer (LSS Transformer).

25. The method according to claim 16, characterized in that The first fusion network is a first cross attention network; using the first fusion network to perform feature fusion on the first conversion feature and the first BEV feature to obtain a second BEV feature, including: mapping the first BEV feature to obtain a query Q in the input data of the first cross attention network, and mapping the first conversion feature to obtain a key K and a value V in the input data of the first cross attention network, and inputting the obtained Q, K and V into the first cross attention network for feature fusion to obtain a second BEV feature; and / or, The second fusion network is a second cross-attention network; the second fusion network is used to perform feature fusion on the second conversion feature and the first image feature to obtain a second image feature, including: mapping the first image feature to obtain a query Q in the input data of the second cross-attention network, and mapping the second conversion feature to obtain a key K and a value V in the input data of the second cross-attention network, and inputting the obtained Q, K and V into the second cross-attention network for feature fusion to obtain a second image feature.

26. The method according to claim 16, characterized in that The difference between the third prediction result and the corresponding true value is calculated based on the predicted RGB value of each pixel in the sample image and the true RGB value of each pixel in the sample image using a mean square error loss function; and / or, The difference between the fourth prediction result and the corresponding true value is: calculated using the mean square error loss function based on the predicted depth value of each pixel in the sample image and the depth value of the corresponding point of each pixel of the sample image in the sample point cloud.

27. The method according to claim 15 or 16, characterized in that The second converter is used to convert the first BEV feature to the image perspective of the sample image based on the intrinsic parameters, extrinsic parameters of the camera that collects the sample image and the conversion relationship between the point cloud coordinate system of the sample point cloud and the camera coordinate system of the camera.

28. An image processing method, characterized in that: The method comprises: Get the image to be processed; Inputting the image to be processed into a second encoder in a multimodal model trained by the method according to any one of claims 15 to 27 to obtain image features to be used; Target detection is performed based on the image features to be utilized to obtain a detection result.

29. A multimodal model pre-training device, characterized in that: The device comprises: A first acquisition module acquires a sample point cloud and a sample image corresponding to the same scene; A first mask module performs voxelization processing on the sample point cloud to obtain a plurality of voxels, performs point cloud mask processing on some real points in the sample point cloud according to the bird's-eye view grid to which each voxel belongs to obtain a first mask point cloud, and performs image mask processing on the sample image to obtain a first mask image; A first feature extraction module, using a first encoder in a multimodal model to perform feature extraction on the first mask point cloud to obtain features under a BEV perspective as first BEV features, and using a second encoder in the multimodal model to perform feature extraction on the first mask image to obtain first image features; wherein the multimodal model further includes a first converter, a first fusion network, a first decoder, and a second decoder; A first conversion module, using the first converter to convert the first image feature to the BEV perspective to obtain a first conversion feature; a first fusion module, using the first fusion network to perform feature fusion on the first conversion feature and the first BEV feature to obtain a second BEV feature; A first decoding module is configured to decode the second BEV feature using the first decoder to obtain a first prediction result and a second prediction result, and to decode the second image feature using the second decoder to obtain a third prediction result; wherein the first prediction result represents: the predicted number of prediction points contained in the bird's-eye view grid and the predicted coordinates of each prediction point, the second prediction result represents: the predicted RGB value of each prediction point in the bird's-eye view grid, and the third prediction result represents: the predicted RGB value of each pixel in the sample image; the second image feature is obtained based on the first image feature; The first adjustment module adjusts the model parameters of the multimodal model based on the difference between the first prediction result and the corresponding true value, the difference between the second prediction result and the corresponding true value, and the difference between the third prediction result and the corresponding true value; wherein the true value corresponding to the second prediction result is the RGB value of the corresponding pixel point in the sample image of the real point in the bird's-eye view grid.

30. A point cloud processing device, characterized in that: The device comprises: Point cloud acquisition module, used to acquire the point cloud to be processed; A point cloud input module, used for inputting the point cloud to be processed into a first encoder in a multimodal model trained based on the method according to any one of claims 1 to 13, to obtain a BEV feature to be used; The point cloud detection module is used to perform target detection based on the BEV features to be utilized to obtain a detection result.

31. A multimodal model pre-training device, characterized in that: The device comprises: A second acquisition module acquires a sample point cloud and a sample image corresponding to the same scene; A second mask module performs voxelization processing on the sample point cloud to obtain a plurality of voxels, performs point cloud mask processing on some real points in the sample point cloud according to the bird's-eye view grid to which each voxel belongs to obtain a first mask point cloud, and performs image mask processing on the sample image to obtain a first mask image; A second feature extraction module, using the first encoder in the multimodal model to perform feature extraction on the first mask point cloud to obtain features under the BEV perspective as first BEV features, and using the second encoder in the multimodal model to perform feature extraction on the first mask image to obtain first image features; wherein the multimodal model further includes a second converter, a second fusion network, a first decoder and a second decoder; A second conversion module, using the second converter to convert the first BEV feature to the image perspective of the sample image to obtain a second conversion feature; a second fusion module, using the second fusion network to perform feature fusion on the second conversion feature and the first image feature to obtain a second image feature; A second decoding module is configured to decode the second BEV feature using the first decoder to obtain a first prediction result, and to decode the second image feature using the second decoder to obtain a third prediction result and a fourth prediction result; wherein the first prediction result indicates: the predicted number of prediction points contained in the bird's-eye view grid and the predicted coordinates of each prediction point, the third prediction result indicates: the predicted RGB value of each pixel in the sample image, and the fourth prediction result indicates: the predicted depth value of each pixel in the sample image; the second BEV feature is obtained based on the first BEV feature; The second adjustment module adjusts the model parameters of the multimodal model based on the difference between the first prediction result and the corresponding true value, the difference between the third prediction result and the corresponding true value, and the difference between the fourth prediction result and the corresponding true value; wherein the true value corresponding to the fourth prediction result is the depth value of the corresponding point of each pixel point of the sample image in the sample point cloud.

32. An image processing device, characterized in that: The device comprises: An image acquisition module, used for acquiring an image to be processed; An image input module, used for inputting the image to be processed into a second encoder in a multimodal model trained based on the method according to any one of claims 15 to 27, to obtain image features to be used; The image detection module is used to perform target detection based on the image features to be utilized to obtain a detection result.

33. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor, for implementing the method described in any one of claims 1 to 28 when executing a program stored in a memory.

34. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 28 is implemented.

35. A computer program product, characterized in that The computer program product comprises executable instructions, and when the executable instructions are executed on a computer, the computer is caused to perform the method according to any one of claims 1 to 28.