A processing method and device for self-supervised learning based on multimodal information

Abstract

Embodiments of the present invention relate to a processing method and apparatus for self-supervised learning based on multimodal information. The method comprises: extracting an original point cloud and a first surround view; performing data augmentation on the original point cloud to generate first and second point clouds; filtering the ground point cloud; inputting the first and second point clouds into a point cloud bird's-eye view feature extraction network to generate first and second feature maps; inputting the first surround view into an image bird's-eye view feature extraction network to generate a third feature map; performing first multimodal information sample preparation processing; performing bird's-eye view plane clustering on the first and second point clouds; performing sample clustering processing; performing second multimodal information sample preparation processing; and constructing first multimodal, second multimodal, and overall loss functions to learn network parameters of the point cloud bird's-eye view feature extraction network. The present invention can shorten data preparation time and improve network learning accuracy and training adequacy.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a processing method and device for self-supervised learning based on multimodal information. Background Art

[0002] In the field of autonomous driving and robotics, the neural network that extracts features from point cloud data is called a point cloud feature extraction network, and the network that converts three-dimensional point cloud features to a bird's-eye view (BEV) is called a point cloud bird's-eye view feature extraction network. Before using the point cloud bird's-eye view feature extraction network for point cloud feature extraction, it needs to be trained. Currently, most common training methods are based on supervised learning, that is, the point cloud data used for training needs to be feature-labeled before training. Through practice, we found that this conventional supervised learning method has the following problems: 1) It is difficult to label three-dimensional point clouds, and it takes a long time to prepare enough training data; 2) The sparsity of point clouds is large, and using only point clouds as training data may result in insufficient training, which will affect the feature extraction accuracy of the point cloud bird's-eye view feature extraction network. Summary of the Invention

[0003] The purpose of the present invention is to address the shortcomings of the existing technology and provide a processing method, device, electronic device, and computer-readable storage medium for self-supervised learning based on multimodal information. The method uses self-supervised learning to train a point cloud bird's-eye view feature extraction network, introducing a ring view rich in environmental information and combining it with the point cloud to form a multimodal information sample set, and constructing a multimodal loss function based on the InfoNCE loss function. The present invention eliminates the need to annotate three-dimensional point clouds, shortening data preparation time. Training a point cloud bird's-eye view feature extraction network based on multimodal information can improve the network's learning accuracy and training adequacy.

[0004] To achieve the above objectives, a first aspect of an embodiment of the present invention provides a processing method for self-supervised learning based on multimodal information, the method comprising:

[0005] Selecting any first training data record from the first training data set to perform original point cloud and surround view extraction processing to generate a corresponding original point cloud and first surround view;

[0006] Select two methods from a plurality of data enhancement methods to perform corresponding data enhancement processing on the original point cloud to generate corresponding first and second point clouds; and perform ground point cloud filtering processing on the first and second point clouds;

[0007] Inputting the first point cloud and the second point cloud into a point cloud bird's-eye view feature extraction network for feature extraction processing to generate corresponding first feature maps and second feature maps; inputting the first surround view into a well-trained image bird's-eye view feature extraction network for feature extraction processing to generate a corresponding third feature map; and performing first multimodal information sample preparation processing based on the first, second, and third feature maps to obtain corresponding first, second, and third sample sequences;

[0008] and performing bird's-eye view plane clustering on the first and second point clouds according to a preset number of clusters Q to obtain corresponding first and second plane cluster space sequences; and performing sample clustering processing on the first, second, and third sample sequences based on the first and second plane cluster space sequences to generate corresponding first, second, and third cluster sample sequence sets; and performing second multimodal information sample preparation processing on the first, second, and third cluster sample sequence sets to obtain corresponding fourth, fifth, and sixth sample sequences;

[0009] A first multimodal loss function L1, a second multimodal loss function L2 and an overall loss function L are constructed according to the first, second, third, fourth, fifth and sixth sample sequences. all , and according to the first multimodal loss function L1, the second multimodal loss function L2 and the overall loss function L all The network parameters of the point cloud bird's-eye view feature extraction network are learned.

[0010] Preferably, the first training data set includes a plurality of the first training data records; the first training data record includes a first scene identification field, a first scene information field, a first point cloud field and a first surround view field;

[0011] The multiple data enhancement methods include rotation, noise addition, downsampling and random occlusion;

[0012] The plane spatial range of the bird's-eye view plane of the point cloud bird's-eye view feature extraction network and the image bird's-eye view feature extraction network is consistent and recorded as the corresponding first bird's-eye view plane; the output feature shape of the point cloud bird's-eye view feature extraction network and the image bird's-eye view feature extraction network is the same as that of the image bird's-eye view feature extraction network, both of which are H×W×D, where H, W, and D are all positive integers, H and W are the height and width of the preset bird's-eye view feature map, and D is the feature dimension of the preset bird's-eye view feature map; the first bird's-eye view plane includes H×W first bird's-eye view grids r i,j ,1≤i≤H,1≤j≤W;

[0013] The shapes of the first, second and third feature maps are all H×W×D; the first feature map consists of H*W first feature vectors a with a length of D i,j Composition, each of the first eigenvectors a i,jCorresponding to the first bird's-eye view grid r i,j The second feature map consists of H*W second feature vectors b of length D i,j The second eigenvector b i,j Corresponding to the first bird's-eye view grid r i,j ; The third feature map consists of H*W third feature vectors c of length D i,j The second eigenvector b i,j Corresponding to the first bird's-eye view grid r i,j ;

[0014] The first planar clustering space sequence includes the first planar clustering space s of the cluster quantity Q. 1,f , 1≤f≤Q; the second plane clustering space sequence includes the second plane clustering space s of the cluster number Q 2,f ; The first plane clustering space s with the same sequence index f 1,f With the second plane clustering space s 2,f The spatial distance is the shortest;

[0015] The first cluster sample sequence set includes multiple first cluster sample sequences; the first cluster sample sequence includes multiple first cluster samples; the second cluster sample sequence set includes multiple second cluster sample sequences; the second cluster sample sequence includes multiple second cluster samples; the third cluster sample sequence set includes multiple third cluster sample sequences; and the third cluster sample sequence includes multiple third cluster samples.

[0016] Preferably, the point cloud bird's-eye view feature extraction network is a point cloud pillar feature network of a PointPillars network.

[0017] Preferably, the extracting and processing of the original point cloud and the surround view from any one first training data record in the first training data set to generate the corresponding original point cloud and the first surround view specifically includes:

[0018] Select any one of the first training data records from the first training data set as the current training data record; and extract the first point cloud field and the first surround view field of the current training data record as the corresponding original point cloud and the first surround view.

[0019] Preferably, the step of performing corresponding data enhancement processing on the original point cloud to generate the corresponding first and second point clouds by selecting two of the plurality of data enhancement methods specifically includes:

[0020] Select any two methods from the multiple data enhancement methods as the corresponding first and second methods; and based on the first method, perform corresponding first data enhancement processing on the original point cloud to generate the corresponding first point cloud; and based on the second method, perform corresponding second data enhancement processing on the original point cloud to generate the corresponding second point cloud.

[0021] Preferably, the performing ground point cloud filtering processing on the first and second point clouds specifically includes:

[0022] All points with a height coordinate of 0 in the first and second point clouds are marked as ground points; and all the ground points in the first and second point clouds are deleted.

[0023] Preferably, performing the first multimodal information sample preparation process according to the first, second, and third feature maps to obtain corresponding first, second, and third sample sequences specifically includes:

[0024] For each of the first bird's-eye view grids r of the first bird's-eye view plane i,j Traverse; when traversing, the first bird's-eye view grid r currently traversed i,j As the corresponding current grid; and the first feature vector a corresponding to the current grid i,j The second eigenvector b i,j and the third eigenvector c i,j Are they all zero vectors? If so, the first feature vector a corresponding to the current grid is used. i,j The second eigenvector b i,j and the third eigenvector c i,j Form the corresponding first non-zero vector group;

[0025] The total number of the obtained first non-zero vector groups is counted to generate a corresponding first quantity M; and the M first feature vectors a corresponding to the M first non-zero vector groups are i,j , M second eigenvectors b i,j and the M third eigenvectors c i,j The first, second and third sample sequences are formed accordingly; the first sample sequence includes M first samples P 1,k , each of the first samples P 1,k Corresponding to the first eigenvector a i,j , 1≤k≤M; the second sample sequence includes M second samples P 2,k , each of the second samples P 2,k Corresponding to the second eigenvector b i,j The third sample sequence includes M third samples P 3,k , each of the third samples P3,k Corresponding to the third eigenvector c i,j The first sample P with the same subscript k 1,k The second sample P 2,k and the third sample P 3,k The corresponding first eigenvector a i,j The second eigenvector b i,j and the third eigenvector c i,j belong to the same first non-zero vector group.

[0026] Preferably, performing bird's-eye view plane clustering on the first and second point clouds according to a preset number of clusters Q to obtain corresponding first and second plane cluster space sequences specifically includes:

[0027] Based on a preset unsupervised clustering algorithm and the set number of clusters Q, point cloud clustering processing is performed on the first point cloud to obtain first point cloud clusters of the number of clusters Q; and based on the unsupervised clustering algorithm and the set number of clusters Q, point cloud clustering processing is performed on the second point cloud to obtain second point cloud clusters of the number of clusters Q; and the three-dimensional point cloud spaces of each of the first and second point cloud clusters are used as corresponding first and second point cloud cluster spaces;

[0028] Project each of the first point cloud clustering spaces onto the first bird's-eye view plane to obtain the corresponding first plane clustering space s 1,f , 1≤f≤Q; and all the first plane clustering spaces s obtained 1,f forming the corresponding first plane clustering space sequence;

[0029] Project each of the second point cloud clustering spaces onto the first bird's-eye view plane to obtain the corresponding first projection space; and combine it with the first plane clustering space s 1,f＝1 The first projection space with the shortest spatial distance is used as the corresponding second planar clustering space s 2,f＝1 , and will be clustered with the first plane space s 1,f＝2 The first projection space with the shortest spatial distance is used as the corresponding second planar clustering space s 2,f＝2 , and so on, until the first plane clustering space s 1,f＝Q The first projection space with the shortest spatial distance is used as the corresponding second planar clustering space s 2,f＝Q and the second plane clustering space s of the obtained cluster number Q 2,f The corresponding second plane clustering space sequence is formed.

[0030] Preferably, performing sample clustering processing on the first, second, and third sample sequences according to the first and second planar clustering space sequences to generate corresponding first, second, and third cluster sample sequence sets specifically includes:

[0031] Randomly select one of the first and second planar clustering space sequences as the corresponding third planar clustering space sequence; the third planar clustering space sequence includes the third planar clustering space s of the cluster quantity Q. 3,f ;

[0032] The first plane clustering space s of the first plane clustering space sequence 1,f Traverse one by one; and when traversing, the first plane clustering space s currently traversed 1,f as the corresponding current space; and the first bird's-eye view grid r on the first bird's-eye view plane that is fully or partially covered by the current space i,j are all marked as a type of grid; and the first feature vector a corresponding to each of the first type of grids in the first feature map is i,j are all marked as a class of vectors; and the first sample P corresponding to each of the first class of vectors in the first sample sequence is 1,k extracting them as the corresponding first cluster samples; and forming the corresponding first cluster sample sequence from all the obtained first cluster samples; and at the end of the traversal, forming the corresponding first cluster sample sequence set from all the obtained first cluster sample sequences;

[0033] The second plane clustering space s of the second plane clustering space sequence 2,f Traverse one by one; and when traversing, the second plane clustering space s currently traversed 2,f as the corresponding current space; and the first bird's-eye view grid r on the first bird's-eye view plane that is fully or partially covered by the current space i,j are all marked as second-class grids; and the second feature vector b corresponding to each of the second-class grids in the second feature map is i,j are all marked as second-class vectors; and the second samples P corresponding to each of the second-class vectors in the second sample sequence are marked as second-class vectors; 2,k extracting them as the corresponding second cluster samples; and forming a corresponding second cluster sample sequence from all the obtained second cluster samples; and forming a corresponding second cluster sample sequence set from all the obtained second cluster sample sequences at the end of the traversal;

[0034] The third plane clustering space s of the third plane clustering space sequence 3,f Traverse one by one; and when traversing, the third plane clustering space s currently traversed3,f as the corresponding current space; and the first bird's-eye view grid r on the first bird's-eye view plane that is fully or partially covered by the current space i,j are all marked as three types of grids; and the third eigenvector c corresponding to each of the three types of grids in the third feature map is i,j are all marked as three types of vectors; and the third samples P corresponding to each of the three types of vectors in the third sample sequence are ... 3,k Extract them as the corresponding third cluster samples; and form the corresponding third cluster sample sequence from all the obtained third cluster samples; and at the end of the traversal, form the corresponding third cluster sample sequence set from all the obtained third cluster sample sequences.

[0035] Preferably, performing the second multimodal information sample preparation process according to the first, second and third clustered sample sequence sets to obtain corresponding fourth, fifth and sixth sample sequences specifically includes:

[0036] Input each of the first cluster sample sequences in the first cluster sample sequence set into a maturely trained maximum pooling network for calculation, and use the output vector of the maximum pooling network as the corresponding fourth sample P 4,f ; and all the fourth samples P obtained 4,f forming the corresponding fourth sample sequence;

[0037] Each second cluster sample sequence in the second cluster sample sequence set is input into the maturely trained maximum pooling network for calculation, and the output vector of the maximum pooling network is used as the corresponding fifth sample P 5,f ; and all the fifth samples P obtained 5,f forming the corresponding fifth sample sequence;

[0038] Each of the third cluster sample sequences in the third cluster sample sequence set is input into a maturely trained maximum pooling network for calculation, and the output vector of the maximum pooling network is used as the corresponding sixth sample P 6,f ; and all the sixth samples P obtained 6,f to form the corresponding sixth sample sequence.

[0039] Preferably, the first multimodal loss function L1, the second multimodal loss function L2 and the overall loss function L are constructed according to the first, second, third, fourth, fifth and sixth sample sequences. all , and according to the first multimodal loss function L1, the second multimodal loss function L2 and the overall loss function L all The network parameters of the point cloud bird's-eye view feature extraction network are learned, specifically including:

[0040] According to the first, second and third sample sequences, the first multimodal loss function L1 is confirmed to be

[0041]

[0042] L infoNCE () is the InfoNCE loss function, M is the number of samples in the first, second and third sample sequences;

[0043] The second multimodal loss function L2 is confirmed to be:

[0044]

[0045] Q is the number of clusters;

[0046] The overall loss function L is determined based on the first multimodal loss function L1 and the second multimodal loss function L2. all for

[0047] L all =L1+L2;

[0048] Substituting each sample of the first, second, and third sample sequences into the first multimodal loss function L1 to calculate and generate a corresponding first loss value; and identifying whether the first loss value satisfies a preset first loss value convergence range; if not, modulating the network parameters of the point cloud bird's-eye view feature extraction network based on a direction that minimizes the first multimodal loss function L1;

[0049] Substituting each sample of the fourth, fifth, and sixth sample sequences into the second multimodal loss function L2 to calculate a corresponding second loss value; and identifying whether the second loss value satisfies a preset second loss value convergence range; if not, modulating the network parameters of the point cloud bird's-eye view feature extraction network based on a direction that minimizes the second multimodal loss function L2;

[0050] Substitute each sample of the first, second, third, fourth, fifth and sixth sample sequences into the overall loss function L all Calculate the corresponding overall loss value; and identify whether the overall loss value meets the preset overall loss value convergence range; if not, based on making the overall loss function L all The direction in which the minimum value is reached modulates the network parameters of the point cloud bird's-eye view feature extraction network.

[0051] A second aspect of an embodiment of the present invention provides a device for implementing the processing method for self-supervised learning based on multimodal information described in the first aspect above, the device comprising: an acquisition module, a data preprocessing module, a first multimodal information processing module, a second multimodal information processing module, and a multimodal information learning processing module;

[0052] The acquisition module is used to select a first training data record from the first training data set to perform original point cloud and surround view extraction processing to generate a corresponding original point cloud and first surround view;

[0053] The data preprocessing module is used to select two methods from a plurality of data enhancement methods to perform corresponding data enhancement processing on the original point cloud to generate corresponding first and second point clouds; and perform ground point cloud filtering processing on the first and second point clouds;

[0054] The first multimodal information processing module is configured to input the first point cloud and the second point cloud into a point cloud bird's-eye view feature extraction network for feature extraction processing to generate corresponding first feature maps and second feature maps; input the first surround view into a well-trained image bird's-eye view feature extraction network for feature extraction processing to generate a corresponding third feature map; and perform first multimodal information sample preparation processing based on the first, second, and third feature maps to obtain corresponding first, second, and third sample sequences;

[0055] The second multimodal information processing module is configured to perform bird's-eye view planar clustering on the first and second point clouds according to a preset number of clusters Q to obtain corresponding first and second planar cluster space sequences; perform sample clustering processing on the first, second, and third sample sequences based on the first and second planar cluster space sequences to generate corresponding first, second, and third cluster sample sequence sets; and perform second multimodal information sample preparation processing on the first, second, and third cluster sample sequence sets to obtain corresponding fourth, fifth, and sixth sample sequences;

[0056] The multimodal information learning processing module is used to construct a first multimodal loss function L1, a second multimodal loss function L2 and an overall loss function L according to the first, second, third, fourth, fifth and sixth sample sequences. all , and according to the first multimodal loss function L1, the second multimodal loss function L2 and the overall loss function L all The network parameters of the point cloud bird's-eye view feature extraction network are learned.

[0057] A third aspect of an embodiment of the present invention provides an electronic device, including: a memory, a processor, and a transceiver;

[0058] The processor is configured to be coupled to the memory, read and execute instructions in the memory, so as to implement the method steps described in the first aspect above;

[0059] The transceiver is coupled to the processor, and the processor controls the transceiver to send and receive messages.

[0060] A fourth aspect of an embodiment of the present invention provides a computer-readable storage medium, which stores computer instructions. When the computer instructions are executed by a computer, the computer executes the instructions of the method described in the first aspect above.

[0061] Embodiments of the present invention provide a processing method, apparatus, electronic device, and computer-readable storage medium for self-supervised learning based on multimodal information. This method uses self-supervised learning to train a neural network, combining a ring view rich in environmental information with a point cloud to form a multimodal information sample set. A multimodal loss function is constructed based on the InfoNCE loss function. This method eliminates the need for annotating three-dimensional point clouds, significantly reducing data preparation time. Furthermore, the point cloud bird's-eye view feature extraction network is trained based on multimodal information, improving its learning accuracy and training adequacy. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Figure 1 A schematic diagram of a processing method for self-supervised learning based on multimodal information provided in Example 1 of the present invention;

[0063] Figure 2 A module structure diagram of a processing device for self-supervised learning based on multimodal information provided in the second embodiment of the present invention;

[0064] Figure 3 This is a structural diagram of an electronic device provided in Example 3 of the present invention. DETAILED DESCRIPTION

[0065] To make the objectives, technical solutions, and advantages of the present invention more apparent, the present invention will be further described in detail below with reference to the accompanying drawings. It should be understood that the embodiments described herein are merely some, rather than all, of the present invention. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present invention without inventive effort are intended to fall within the scope of protection of the present invention.

[0066] The first embodiment of the present invention provides a processing method for self-supervised learning based on multimodal information, such as Figure 1 A schematic diagram of a processing method for self-supervised learning based on multimodal information provided in the first embodiment of the present invention is shown. The method mainly includes the following steps:

[0067] Step 1: Select a first training data record from the first training data set, perform original point cloud and surround view extraction processing to generate a corresponding original point cloud and first surround view;

[0068] The first training data set includes a plurality of first training data records; the first training data record includes a first scene identification field, a first scene information field, a first point cloud field, and a first surround view field;

[0069] Specifically, the method includes: selecting any first training data record from the first training data set as the current training data record; and extracting the first point cloud field and the first surround view field of the current training data record as the corresponding original point cloud and the first surround view.

[0070] Here, each first training data record of the first training data set corresponds to a group of training data; the first scene identification field is the scene code corresponding to the group of training data, and the first scene information field is the scene introduction corresponding to the group of training data; the first point cloud field is the 360° point cloud collected by the on-board laser radar in the scene (it can also be a point cloud of other surround degrees); the first surround view field is the 360° surround image or surround image set collected by the on-board camera in the scene (such as an image set composed of images from several directions such as left front, front, right front, right side, right rear, rear, and left rear).

[0071] Step 2: Select two data enhancement methods from multiple data enhancement methods to perform corresponding data enhancement processing on the original point cloud to generate corresponding first and second point clouds; and perform ground point cloud filtering processing on the first and second point clouds;

[0072] Among them, multiple data enhancement methods include rotation, noise addition, downsampling and random occlusion;

[0073] Specifically comprising: step 21, selecting two methods from a plurality of data enhancement methods to perform corresponding data enhancement processing on the original point cloud to generate corresponding first and second point clouds;

[0074] Specifically, the method includes: selecting two methods from a plurality of data enhancement methods as corresponding first and second methods; performing a corresponding first data enhancement process on the original point cloud based on the first method to generate a corresponding first point cloud; and performing a corresponding second data enhancement process on the original point cloud based on the second method to generate a corresponding second point cloud;

[0075] Conventional practice indicates that traditional training datasets contain insufficient 3D point cloud data. Learning a neural network based on an insufficiently rich training dataset may result in missing data or insufficient learning. To improve the richness of the training data, embodiments of the present invention utilize multiple data augmentation methods to transform the original point cloud. The first and second point clouds obtained after the transformation can be considered as point clouds from different angles of the same scene.

[0076] For example, the first and second methods selected are rotation and random occlusion, respectively; then, the corresponding first data enhancement process is point cloud rotation, and the corresponding second data enhancement process is point cloud random occlusion. The original point cloud is subjected to point cloud rotation to generate the corresponding first point cloud, and the original point cloud is subjected to point cloud random occlusion to generate the corresponding second point cloud.

[0077] Step 22, performing ground point cloud filtering processing on the first and second point clouds;

[0078] Specifically, the method includes: marking all points with a height coordinate of 0 in the first and second point clouds as ground points; and deleting all ground points in the first and second point clouds.

[0079] Here, the coordinates of each point in the point cloud are three-dimensional coordinates (x, y, z) in the world coordinate system, where z is the height coordinate; points with a height coordinate z of 0 are all ground points. In the embodiment of the present invention, it is stipulated that the ground points need to be filtered out from the first and second point clouds before training.

[0080] Step 3: Input the first point cloud and the second point cloud into the point cloud bird's-eye view feature extraction network for feature extraction to generate corresponding first feature maps and second feature maps; input the first ring view into the well-trained image bird's-eye view feature extraction network for feature extraction to generate corresponding third feature maps; and perform first multimodal information sample preparation processing based on the first, second, and third feature maps to obtain corresponding first, second, and third sample sequences;

[0081] Among them, the plane spatial range of the bird's-eye view plane of the point cloud bird's-eye view feature extraction network and the image bird's-eye view feature extraction network is consistent and recorded as the corresponding first bird's-eye view plane; the output feature shape of the point cloud bird's-eye view feature extraction network and the image bird's-eye view feature extraction network is the same, both of which are H×W×D, where H, W, and D are all positive integers, H and W are the height and width of the preset bird's-eye view feature map, and D is the feature dimension of the preset bird's-eye view feature map; the first bird's-eye view plane includes H×W first bird's-eye view grids r i,j ,1≤i≤H,1≤j≤W;

[0082] The shapes of the first, second and third feature maps are all H×W×D; the first feature map consists of H*W first feature vectors a of length D. i,jComposition, each first eigenvector a i,j Corresponding to a first bird's-eye view grid r i,j ; The second feature map consists of H*W second feature vectors b of length D i,j Composition, each second eigenvector b i,j Corresponding to a first bird's-eye view grid r i,j ; The third feature map consists of H*W third feature vectors c of length D i,j Composition, each second eigenvector b i,j Corresponding to a first bird's-eye view grid r i,j ;

[0083] Specifically comprising: step 31, inputting the first point cloud and the second point cloud into the point cloud bird's-eye view feature extraction network respectively for feature extraction processing to generate corresponding first feature map and second feature map;

[0084] Among them, the point cloud bird's-eye view feature extraction network is the point cloud pillar feature network of the PointPillars network;

[0085] Here, the embodiment of the present invention uses the point cloud column feature network (PillarFeature Net) of the PointPillars network mentioned in the technical paper "PointPillars: Fast Encoders for Object Detection from Point Clouds" as the point cloud bird's-eye view feature extraction network; the point cloud bird's-eye view feature extraction network of the embodiment of the present invention is similar to the point cloud column feature network of the PointPillars network, and a point cloud column (Pillar) grid space (size is H×W, number of grids is H*W) is constructed on the first bird's-eye view plane, and point cloud clustering and point cloud column feature extraction are performed on the first and second point clouds input based on the point cloud columns (Pillar) corresponding to each grid to obtain the corresponding point cloud column feature vector, that is, the first feature vector a i,j , the second eigenvector b i,j , and the obtained H*W first eigenvectors a i,j , the second eigenvector b i,j The corresponding feature maps are the first and second feature maps;

[0086] Step 32: Input the first ring view into a well-trained bird's-eye view feature extraction network to perform feature extraction processing to generate a corresponding third feature map;

[0087] Here, the embodiment of the present invention uses a neural network that can perform image bird's-eye view features on the surrounding view as an image bird's-eye view feature extraction network (for example, the encoder of BEVformer, the BEV feature encoder of HDMapNet, etc.) and completes relevant training for it in advance; it should be noted that the planar spatial range of the bird's-eye view plane of the image bird's-eye view feature extraction network of the embodiment of the present invention should be consistent with the planar spatial range of the bird's-eye view plane of the point cloud bird's-eye view feature extraction network, that is, both are the first bird's-eye view plane, and the feature shape output by the image bird's-eye view feature extraction network should also be the same as the output feature shape of the point cloud bird's-eye view feature extraction network. If the output of the original network (for example, the encoder of BEVformer, the BEV feature encoder of HDMapNet, etc.) is different from the output shape of the point cloud bird's-eye view feature extraction network, then it is necessary to add a fully connected or convolutional network for shaping after the output of the original network to ensure that the output shape of the image bird's-eye view feature extraction network can correspond to the output shape of the image bird's-eye view feature extraction network;

[0088] Step 33: performing first multimodal information sample preparation processing according to the first, second, and third feature maps to obtain corresponding first, second, and third sample sequences;

[0089] Specifically, step 331 includes: i,j Traverse; when traversing, the first bird's-eye view grid r currently traversed i,j As the corresponding current grid; and the first eigenvector a corresponding to the current grid i,j , the second eigenvector b i,j and the third eigenvector c i,j Are they all zero vectors for identification? If so, the first eigenvector a corresponding to the current grid is used. i,j , the second eigenvector b i,j and the third eigenvector c i,j Form the corresponding first non-zero vector group;

[0090] Here, the all-zero vector is a vector whose D vector data are all 0, and the all-zero vector is also called a null vector;

[0091] For example, the shape of the first bird's-eye view plane is 4×3, including 12 first bird's-eye view grids r i,j , respectively: r 1,1 、r 1,2 、r 1,3 、r 2,1 、r 2,2 、r 2,3 、r 3,1 、r 3,2 、r 3,3 、r 4,1 、r 4,2、r 4,3 ; The 12 first eigenvectors a of the known first eigenmap i,j There is only a 1,1 、a 1,2 、a 1,3 、a 4,1 Not all 0 vectors, the 12 second eigenvectors b of the second eigenmap i,j Only b 1,1 、b 1,2 、b 1,3 、b 4,2 Not all 0 vectors, the 12 third eigenvectors c of the third eigenmap i,j None of them are all zero vectors;

[0092] Then, for each first bird's-eye view grid r of the first bird's-eye view plane i,j To traverse,

[0093] The current grid is r 1,1 When, because a 1,1 、b 1,1 、c 1,1 Are not all zero vectors, so we get the first non-zero vector group 1 (a 1,1 ,b 1,1 ,c 1,1 );

[0094] The current grid is r 1,2 When, because a 1,2 、b 1,2 、c 1,2 are not all zero vectors, so we get the first non-zero vector group 2 (a 1,2 ,b 1,2 ,c 1,2 );

[0095] The current grid is r 1,3 When, because a 1,3 、b 1,3 、c 1,3 are not all zero vectors, so we get the first non-zero vector group 3 (a 1,3 ,b 1,3 ,c 1,3 );

[0096] The current grid is r 2,1 、r 2,2 、r 2,3 、r 3,1 、r 3,2 、r 3,3 、r 4,1 、r 4,2 、r 4,3 When , because the corresponding first eigenvector a i,j , the second eigenvector bi,j and the third eigenvector c i,j There is always at least one all-zero vector in , so no first non-zero vector group is generated;

[0097] Step 332: Count the total number of the obtained first non-zero vector groups to generate a corresponding first quantity M; and calculate the M first eigenvectors a corresponding to the M first non-zero vector groups. i,j , M second eigenvectors b i,j and M third eigenvectors c i,j forming corresponding first, second and third sample sequences;

[0098] The first sample sequence includes M first samples P 1,k , each first sample P 1,k Corresponding to a first eigenvector a i,j , 1≤k≤M; the second sample sequence includes M second samples P 2,k , each second sample P 2,k Corresponding to a second eigenvector b i,j The third sample sequence includes M third samples P 3,k , each third sample P 3,k Corresponding to a third eigenvector c i,j ; The first sample P with the same subscript k 1,k , the second sample P 2,k and the third sample P 3,k The corresponding first eigenvector a i,j , the second eigenvector b i,j and the third eigenvector c i,j Belong to the same first non-zero vector group.

[0099] For example, it is known that all the first non-zero vector groups obtained are: first non-zero vector group 1 (a 1,1 ,b 1,1 ,c 1,1 ), the first non-zero vector group 2 (a 1,2 ,b 1,2 ,c 1,2 ) and the first non-zero vector group 3(a 1,3 ,b 1,3 ,c 1,3 ); then, the total number of the obtained first non-zero vector groups is statistically generated to generate a first number M=3; the first sample sequence is {P 1,1 =a 1,1 ,P 1,2 =a 1,2 ,P 1,3 =a 1,3}, the second sample sequence is {P 2,1 =b 1,1 ,P2,2 =b 1,2 ,P 2,3 =b 1,3}, the third sample sequence is {P 3,1 =c 1,1 ,P 3,2 =c 1,2 , P 3,3 =c 1,3}.

[0100] Here, the first and second sample sequences obtained are feature sample sequences in the point cloud modality, and the third sample sequence is a feature sample sequence in the image modality. Moreover, the position codes of the samples in these three sequences are one-to-one corresponding.

[0101] Step 4: Perform bird's-eye view plane clustering on the first and second point clouds according to the preset number of clusters Q to obtain corresponding first and second plane cluster space sequences; perform sample clustering processing on the first, second, and third sample sequences based on the first and second plane cluster space sequences to generate corresponding first, second, and third cluster sample sequence sets; and perform second multimodal information sample preparation processing on the first, second, and third cluster sample sequence sets to obtain corresponding fourth, fifth, and sixth sample sequences;

[0102] The first plane clustering space sequence includes the first plane clustering space s of the cluster number Q. 1,f , 1≤f≤Q; the second plane clustering space sequence includes the second plane clustering space s with the number of clusters Q 2,f ; The first plane clustering space s with the same sequence index f 1,f With the second plane clustering space s 2,f The spatial distance is the shortest;

[0103] The first cluster sample sequence set includes a plurality of first cluster sample sequences; the first cluster sample sequence includes a plurality of first cluster samples; the second cluster sample sequence set includes a plurality of second cluster sample sequences; the second cluster sample sequence includes a plurality of second cluster samples; the third cluster sample sequence set includes a plurality of third cluster sample sequences; the third cluster sample sequence includes a plurality of third cluster samples;

[0104] Specifically, the method includes: step 41, performing bird's-eye view plane clustering on the first and second point clouds according to a preset cluster quantity Q to obtain corresponding first and second plane cluster space sequences;

[0105] Specifically, the method includes: step 411, performing point cloud clustering processing on the first point cloud based on a preset unsupervised clustering algorithm and a set number of clusters Q to obtain a first point cloud cluster with a number of clusters Q; performing point cloud clustering processing on the second point cloud based on the unsupervised clustering algorithm and the set number of clusters Q to obtain a second point cloud cluster with a number of clusters Q; and using the three-dimensional point cloud spaces of the first and second point cloud clusters as the corresponding first and second point cloud cluster spaces;

[0106] Among them, unsupervised clustering algorithms include KMeans clustering algorithm, binary KMeans clustering algorithm, KMeans++ clustering algorithm, KMeansll clustering algorithm, Canopy clustering algorithm, Mini Batch KMeans clustering algorithm;

[0107] Here, when the embodiment of the present invention obtains the first or second point cloud cluster, six extreme coordinate values ​​on the XYZ axes are extracted from each first or second point cloud cluster: x max / x min 、y max / y min 、z max / z min , and based on these six coordinate extremes, determine the center point coordinates and shape (length, width, height) of the corresponding three-dimensional point cloud space (the first or second point cloud cluster space), and then determine the eight vertex coordinates of the corresponding first or second point cloud cluster space from the obtained center point + shape (length, width, height); It should be noted that if the actual number of clusters of the first point cloud does not reach the number of clusters Q, then one or more preset all-zero point cloud clusters are used to supplement it;

[0108] Step 412: Project each first point cloud cluster space onto the first bird's-eye view plane to obtain the corresponding first plane cluster space s 1,f , 1≤f≤Q; and all the first plane clustering spaces s obtained 1,f Composing the corresponding first plane clustering space sequence;

[0109] Here, projecting each first point cloud cluster space onto the first bird's-eye view plane is actually projecting each first point cloud cluster space onto the XY plane;

[0110] Step 413: Project each second point cloud cluster space onto the first bird's-eye view plane to obtain the corresponding first projection space; and combine it with the first plane cluster space s 1,f＝1 The first projection space with the shortest spatial distance is used as the corresponding second plane clustering space s 2,f＝1 , and will be clustered with the first plane space s 1,f＝2 The first projection space with the shortest spatial distance is used as the corresponding second plane clustering space s 2,f＝2 , and so on, until the first plane clustering space s1,f＝Q The first projection space with the shortest spatial distance is used as the corresponding second plane clustering space s 2,f＝Q So far; and the second plane clustering space s of the obtained cluster number Q 2,f Composing the corresponding second plane clustering space sequence;

[0111] Here, the embodiment of the present invention associates the first and second planar clustering spaces of the first and second planar clustering space sequences through the index f of the clustering space;

[0112] Step 42: performing sample clustering processing on the first, second and third sample sequences according to the first and second planar clustering space sequences to generate corresponding first, second and third cluster sample sequence sets;

[0113] Specifically, the method includes: step 421, randomly selecting one of the first and second planar clustering space sequences as the corresponding third planar clustering space sequence;

[0114] Among them, the third plane clustering space sequence includes the third plane clustering space s with the number of clusters Q 3,f ;

[0115] For example, if the number of clusters Q is 2, the first plane clustering space sequence includes two first plane clustering spaces s 1,1 、s 1,2 , the second plane clustering space sequence includes 2 second plane clustering spaces s 2,1 、s 2,2 , randomly select the second plane clustering space sequence from the first and second plane clustering space sequences, then the third plane clustering space sequence includes 2 second plane clustering spaces s 3,1 =s 2,1 、s 3,2 =s 2,2 ;

[0116] Step 422: the first plane clustering space s of the first plane clustering space sequence 1,f Traverse one by one; and when traversing, the first plane clustering space s currently traversed 1,f As the corresponding current space; and the first bird's-eye view grid r on the first bird's-eye view plane that is fully or partially covered by the current space i,j Mark them as a type of grid; and the first eigenvector a corresponding to each type of grid in the first feature map i,j are all marked as first-class vectors; and the first sample P corresponding to each first-class vector in the first sample sequence is 1,kExtract them as the corresponding first cluster samples; and form a corresponding first cluster sample sequence from all the obtained first cluster samples; and at the end of the traversal, form a corresponding first cluster sample sequence set from all the obtained first cluster sample sequences;

[0117] For example, the shape of the first bird's-eye view plane is 4×3, including 12 first bird's-eye view grids r i,j , respectively: r 1,1 、r 1,2 、r 1,3 、r 2,1 、r 2,2 、r 2,3 、r 3,1 、r 3,2 、r 3,3 、r 4,1 、r 4,2 、r 4,3 ; M = 3, the first sample sequence is {P 1,1 =a 1,1 ,P 1,2 =a 1,2 ,P 1,3 =a 1,3}, the second sample sequence is {P 2,1 =b 1,1 ,P 2,2 =b 1,2 ,P 2,3 =b 1,3}, the third sample sequence is {P 3,1 =c 1,1 ,P 3,2 =c 1,2 , P 3,3 =c 1,3};

[0118] The first plane clustering space s 1,1 With r 1,1 、r 1,2 There is an intersection, the first plane clustering space s 1,2 With r 1,3 There is an intersection;

[0119] Then, for the first plane clustering space s of the first plane clustering space sequence 1,f Traverse one by one; the current space is s 1,1 When r 1,1 、r 1,2 Marked as a type of grid, the first feature map a 1,1 、a 1,2 is marked as a class of vectors, P in the first sample sequence 1,1 、P 1,2 is the first cluster sample, thus obtaining the first cluster sample sequence 11 (P1,1 , P 1,2 ); the current space is s 1,2 When r 1,3 Marked as a type of grid, the first feature map a 1,3 is marked as a class of vectors, P in the first sample sequence 1,3 is the first cluster sample, thus obtaining the first cluster sample sequence 12 (P 1,3 ); The first cluster sample sequence set finally obtained is {first cluster sample sequence 11(P 1,1 , P 1,2 ), the first cluster sample sequence 12 (P 1,3 )};

[0120] Step 423: the second plane clustering space s of the second plane clustering space sequence 2,f Traverse one by one; and when traversing, the second plane clustering space s currently traversed 2,f As the corresponding current space; and the first bird's-eye view grid r on the first bird's-eye view plane that is fully or partially covered by the current space i,j are all marked as second-class grids; and the second eigenvector b corresponding to each second-class grid in the second feature map is i,j are all marked as second-class vectors; and the second sample P corresponding to each second-class vector in the second sample sequence is 2,k Extract them as the corresponding second cluster samples; and form a corresponding second cluster sample sequence from all the obtained second cluster samples; and at the end of the traversal, form a corresponding second cluster sample sequence set from all the obtained second cluster sample sequences;

[0121] For example, the second plane clustering space s 2,1 With r 1,1 、r 1,2 There is an intersection, the second plane clustering space s 2,2 With r 1,3 There is an intersection;

[0122] Then, for the second plane clustering space sequence s 2,f Traverse one by one; the current space is s 2,1 When r 1,1 、r 1,2 Marked as the second type of grid, b of the second feature map 1,1 、b 1,2 Marked as a second-class vector, P in the second sample sequence 2,1 、P 2,2 is the second cluster sample, thus obtaining the second cluster sample sequence 21 (P 2,1 , P 2,2 ); the current space is s 2,2 When r1,3 Marked as the second type of grid, b of the second feature map 1,3 Marked as a second-class vector, P in the second sample sequence 2,3 is the second cluster sample, thus obtaining the second cluster sample sequence 22 (P 2,3 ); The final set of second cluster sample sequences is {second cluster sample sequence 21(P 2,1 , P 2,2 ), the second cluster sample sequence 22 (P 2,3 )};

[0123] Step 424: the third plane clustering space s of the third plane clustering space sequence 3,f Traverse one by one; and when traversing, the third plane clustering space s currently traversed 3,f As the corresponding current space; and the first bird's-eye view grid r on the first bird's-eye view plane that is fully or partially covered by the current space i,j are all marked as three types of grids; and the third eigenvector c corresponding to each of the three types of grids in the third feature map is i,j are all marked as three-category vectors; and the third sample P corresponding to each of the three-category vectors in the third sample sequence is 3,k Extract them as the corresponding third cluster samples; and form a corresponding third cluster sample sequence from all the obtained third cluster samples; and at the end of the traversal, form a corresponding third cluster sample sequence set from all the obtained third cluster sample sequences;

[0124] For example, the third plane clustering space sequence includes two second plane clustering spaces s 3,1 =s 2,1 、s 3,2 =s 2,2 ; Second plane clustering space s 2,1 With r 1,1 、r 1,2 There is an intersection, the second plane clustering space s 2,2 With r 1,3 There is an intersection;

[0125] Then, for the third plane clustering space sequence s 3,f Traverse one by one; the current space is s 3,1 When r 1,1 、r 1,2 Marked as three types of grids, c of the third feature map 1,1 、c 1,2 Marked as three types of vectors, P in the third sample sequence 3,1 、P 3,2 is the third cluster sample, thus obtaining the third cluster sample sequence 31 (P 3,1 , P 3,2); the current space is s 3,2 When r 1,3 Marked as three types of grids, c of the third feature map 1,3 Marked as three types of vectors, P in the third sample sequence 3,3 is the third cluster sample, thus obtaining the third cluster sample sequence 32 (P 3,3 ); The final set of the third cluster sample sequence is {third cluster sample sequence 31 (P 3,1 , P 3,2 ), the third cluster sample sequence 32 (P 3,3 )};

[0126] Step 43: performing second multimodal information sample preparation processing on the first, second, and third clustered sample sequence sets to obtain corresponding fourth, fifth, and sixth sample sequences;

[0127] Specifically, step 431 includes: inputting each first cluster sample sequence of the first cluster sample sequence set into the maturely trained maximum pooling network for calculation, and using the output vector of the maximum pooling network as the corresponding fourth sample P 4,f ; and all the fourth samples P obtained 4,f forming a corresponding fourth sample sequence;

[0128] Here, the embodiment of the present invention performs comprehensive clustering feature extraction on each first cluster sample sequence through a maximum pooling network;

[0129] For example, given that the number of clusters Q=2, the first cluster sample sequence set is {first cluster sample sequence 11(P 1,1 , P 1,2 ), the first cluster sample sequence 12 (P 1,3 )}; Then the first cluster sample sequence 11(P 1,1 , P 1,2 ) Input the maximum pooling network to obtain the corresponding fourth sample P 4,1 , the first cluster sample sequence 12 (P 1,3 ) Input the maximum pooling network to obtain the corresponding fourth sample P 4,2 ; The fourth sample sequence obtained is {P 4,1 ,P 4,2};

[0130] Step 432: Input each second cluster sample sequence of the second cluster sample sequence set into the maturely trained maximum pooling network for calculation, and use the output vector of the maximum pooling network as the corresponding fifth sample P 5,f ; and all the fifth samples P obtained 5,f forming a corresponding fifth sample sequence;

[0131] Here, the embodiment of the present invention performs comprehensive clustering feature extraction on each second cluster sample sequence through a maximum pooling network;

[0132] For example, given that the number of clusters Q=2, the second cluster sample sequence set is {second cluster sample sequence 21(P 2,1 , P 2,2 ), the second cluster sample sequence 22 (P 2,3 )}; Then the second cluster sample sequence 21 (P 2,1 , P 2,2 ) Input the maximum pooling network to obtain the corresponding fifth sample P 5,1 , the second cluster sample sequence 22 (P 2,3 ) Input the maximum pooling network to obtain the corresponding fifth sample P 5,2 ; The fifth sample sequence obtained is {P 5,1 ,P 5,2};

[0133] Step 433: Input each third cluster sample sequence of the third cluster sample sequence set into the maturely trained maximum pooling network for calculation, and use the output vector of the maximum pooling network as the corresponding sixth sample P 6,f ; and all the sixth samples P obtained 6,f to form a corresponding sixth sample sequence.

[0134] Here, the embodiment of the present invention performs comprehensive clustering feature extraction on each third cluster sample sequence through a maximum pooling network;

[0135] For example, given that the number of clusters Q=2, the set of the third cluster sample sequences is {third cluster sample sequence 31(P 3,1 , P 3,2 ), the third cluster sample sequence 32 (P 3,3 )}; Then the third cluster sample sequence 31 (P 3,1 , P 3,2 ) Input the maximum pooling network to obtain the corresponding sixth sample P 6,1 , the third cluster sample sequence 32 (P 3,3 ) Input the maximum pooling network to obtain the corresponding sixth sample P 6,2 ; The sixth sample sequence obtained is {P 6,1 ,P 6,2}.

[0136] Step 5: Construct the first multimodal loss function L1, the second multimodal loss function L2 and the overall loss function L according to the first, second, third, fourth, fifth and sixth sample sequences. all , and according to the first multimodal loss function L1, the second multimodal loss function L2 and the overall loss function L allLearning the network parameters of the point cloud bird's-eye view feature extraction network;

[0137] Specifically, the method includes: step 51, confirming the first multimodal loss function L1 according to the first, second, and third sample sequences as follows:

[0138]

[0139] L infoNCE () is the InfoNCE loss function, M is the number of samples in the first, second and third sample sequences;

[0140] Here, the InfoNCE loss function is a contrastive learning loss function. In the embodiment of the present invention, the first multimodal loss function L1 is constructed based on the InfoNCE loss function to perform pairwise comparison on the grid-level features (or pillar features) of the first point cloud, the second point cloud, and the first surround view on the first bird's-eye view plane.

[0141] Step 52: confirm the second multimodal loss function L2 based on the fourth, fifth and sixth sample sequences as follows:

[0142]

[0143] Q is the number of clusters;

[0144] Here, in an embodiment of the present invention, the second multimodal loss function L2 is constructed based on the InfoNCE loss function, and the cluster features (or cluster features) of the first point cloud, the second point cloud, and the first surround view on the first bird's-eye view plane are compared pairwise based on the clustering result of the point cloud.

[0145] Step 53: Determine the overall loss function L according to the first multimodal loss function L1 and the second multimodal loss function L2. all for:

[0146] L all =L1+L2;

[0147] Here, the overall loss function L of the embodiment of the present invention is all In fact, it is necessary to perform a pairwise comparison of the grid-level features (or pillar features) of the first point cloud, the second point cloud, and the first ring view on the first bird's-eye view plane, and also to perform a pairwise comparison of the cluster features (or cluster features) of the first point cloud, the second point cloud, and the first ring view on the first bird's-eye view plane based on the clustering results of the point clouds;

[0148] Step 54: Substitute each sample of the first, second, and third sample sequences into the first multimodal loss function L1 to calculate and generate a corresponding first loss value; and determine whether the first loss value meets a preset first loss value convergence range; if not, modulate the network parameters of the point cloud bird's-eye view feature extraction network based on the direction that minimizes the first multimodal loss function L1;

[0149] Here, if the first loss value does not satisfy the preset first loss value convergence range, it means that the network parameters of the point cloud bird's-eye view feature extraction network need to be modulated. At this time, the network parameters of the bird's-eye view feature extraction network can be substituted into the expression of the first multimodal loss function L1 to generate the corresponding first objective function, and each network parameter in the first objective function is solved in the direction of the first objective function reaching the minimum value, and the network parameters of the bird's-eye view feature extraction network are reset using the solved network parameter values;

[0150] Step 55: Substitute each sample of the fourth, fifth, and sixth sample sequences into the second multimodal loss function L2 to calculate a corresponding second loss value; and determine whether the second loss value meets a preset second loss value convergence range; if not, modulate the network parameters of the point cloud bird's-eye view feature extraction network based on the direction that minimizes the second multimodal loss function L2;

[0151] Here, if the second loss value does not satisfy the preset second loss value convergence range, it means that the network parameters of the point cloud bird's-eye view feature extraction network need to be modulated. At this time, the network parameters of the bird's-eye view feature extraction network can be substituted into the expression of the second multimodal loss function L2 to generate the corresponding second objective function, and each network parameter in the second objective function is solved in the direction of the second objective function reaching the minimum value, and the network parameters of the bird's-eye view feature extraction network are reset using the solved network parameter values;

[0152] Step 56: Substitute each sample of the first, second, third, fourth, fifth and sixth sample sequences into the overall loss function L all Calculate the corresponding overall loss value; and identify whether the overall loss value meets the preset overall loss value convergence range; if not, based on the overall loss function L all The direction in which the minimum value is reached modulates the network parameters of the point cloud bird's-eye view feature extraction network.

[0153] Here, if the overall loss value does not meet the preset overall loss value convergence range, it means that the network parameters of the point cloud bird's-eye view feature extraction network need to be modulated. At this time, the network parameters of the bird's-eye view feature extraction network can be substituted into the overall loss function L allThe expression generates the corresponding overall objective function, and solves each network parameter in the overall objective function in the direction of minimizing the overall objective function, and uses the solved network parameter values ​​to reset the network parameters of the bird's-eye view feature extraction network.

[0154] It should be noted that in any of the above steps 54-56, as long as the network parameters of the point cloud bird's-eye view feature extraction network are modulated, the subsequent steps will no longer be executed, but the process will directly return to step 3 to re-perform the feature extraction operation based on the modulated point cloud bird's-eye view feature extraction network; if the loss values ​​obtained in the above steps 54-56 have satisfied the corresponding loss value convergence range, indicating that the current self-supervised learning is successful, the process will then return to step 1 to re-acquire a new first training data record and continue the next self-supervised learning of the point cloud bird's-eye view feature extraction network until the number of consecutive successful learning reaches a preset threshold.

[0155] Figure 2 This is a module structure diagram of a processing device for self-supervised learning based on multimodal information provided in the second embodiment of the present invention. The device is a terminal device or server that implements the aforementioned method embodiment, and can also be a device that enables the aforementioned terminal device or server to implement the aforementioned method embodiment. For example, the device can be a device or chip system of the aforementioned terminal device or server. Figure 2 As shown, the device includes: an acquisition module 201, a data preprocessing module 202, a first multimodal information processing module 203, a second multimodal information processing module 204 and a multimodal information learning processing module 205.

[0156] The acquisition module 201 is used to select any first training data record from the first training data set, perform original point cloud and surround view extraction processing, and generate a corresponding original point cloud and first surround view.

[0157] The data preprocessing module 202 is used to select two methods from multiple data enhancement methods to perform corresponding data enhancement processing on the original point cloud to generate corresponding first and second point clouds; and perform ground point cloud filtering processing on the first and second point clouds.

[0158] The first multimodal information processing module 203 is used to input the first point cloud and the second point cloud into the point cloud bird's-eye view feature extraction network respectively for feature extraction processing to generate corresponding first feature maps and second feature maps; and input the first surround view into the well-trained image bird's-eye view feature extraction network for feature extraction processing to generate a corresponding third feature map; and perform first multimodal information sample preparation processing based on the first, second and third feature maps to obtain corresponding first, second and third sample sequences.

[0159] The second multimodal information processing module 204 is used to perform bird's-eye view plane clustering on the first and second point clouds according to a preset cluster quantity Q to obtain corresponding first and second plane cluster space sequences; and perform sample clustering processing on the first, second and third sample sequences according to the first and second plane cluster space sequences to generate corresponding first, second and third cluster sample sequence sets; and perform second multimodal information sample preparation processing on the first, second and third cluster sample sequence sets to obtain corresponding fourth, fifth and sixth sample sequences.

[0160] The multimodal information learning processing module 205 is used to construct a first multimodal loss function L1, a second multimodal loss function L2 and an overall loss function L according to the first, second, third, fourth, fifth and sixth sample sequences. all , and according to the first multimodal loss function L1, the second multimodal loss function L2 and the overall loss function L all Learning the network parameters of the point cloud bird's-eye view feature extraction network.

[0161] An embodiment of the present invention provides a processing device for self-supervised learning based on multimodal information, which can execute the method steps in the above method embodiment. Its implementation principles and technical effects are similar and will not be repeated here.

[0162] It should be noted that it should be understood that the division of the various modules of the above device is merely a division of logical functions. In actual implementation, they can be fully or partially integrated into one physical entity, or they can be physically separated. Moreover, these modules can all be implemented in the form of software called by a processing element; or they can all be implemented in the form of hardware; or some modules can be implemented in the form of software called by a processing element, and some modules can be implemented in the form of hardware. For example, the acquisition module can be a separately established processing element, or it can be integrated into a chip of the above device. In addition, it can also be stored in the memory of the above device in the form of program code, and called by a processing element of the above device to perform the functions of the above determination module. The implementation of other modules is similar. In addition, these modules can all or partly be integrated together, or they can be implemented independently. The processing element described here can be an integrated circuit with signal processing capabilities. In the implementation process, each step of the above method or each of the above modules can be completed by the hardware integrated logic circuit in the processor element or by instructions in the form of software.

[0163] For example, the above modules may be one or more integrated circuits configured to implement the above methods, such as one or more application-specific integrated circuits (ASICs), one or more digital signal processors (DSPs), or one or more field programmable gate arrays (FPGAs). For another example, when a module is implemented by scheduling program code through a processing element, the processing element may be a general-purpose processor, such as a central processing unit (CPU) or other processor that can call program code. For another example, these modules may be integrated together and implemented in the form of a system-on-a-chip (SOC).

[0164] In the above embodiments, all or part of the embodiments may be implemented using software, hardware, firmware, or any combination thereof. When implemented using software, all or part of the embodiments may be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions described in the above method embodiments are generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The above-mentioned computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the above-mentioned computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center via wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, Bluetooth, microwave, etc.) means. The above-mentioned computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that includes one or more available media. The above-mentioned available medium can be a magnetic medium (such as a floppy disk, hard disk, tape), an optical medium (such as a DVD), or a semiconductor medium (such as a solid state disk (SSD)), etc.

[0165] Figure 3 This is a schematic diagram of the structure of an electronic device provided in the third embodiment of the present invention. The electronic device may be the aforementioned terminal device or server, or may be a terminal device or server connected to the aforementioned terminal device or server to implement the method of the embodiment of the present invention. Figure 3As shown, the electronic device may include: a processor 301 (such as a CPU), a memory 302, and a transceiver 303; the transceiver 303 is coupled to the processor 301, and the processor 301 controls the transceiver 303's transceiver actions. Various instructions may be stored in the memory 302 for completing various processing functions and implementing the processing steps described in the aforementioned method embodiment. Preferably, the electronic device involved in the embodiment of the present invention further includes: a power supply 304, a system bus 305, and a communication port 306. The system bus 305 is used to realize communication connections between components. The above-mentioned communication port 306 is used for connection and communication between the electronic device and other peripherals.

[0166] exist Figure 3 The system bus 305 mentioned in the figure can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The system bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus. The communication interface is used to realize communication between the database access device and other devices (such as clients, read-write libraries, and read-only libraries). The memory may include random access memory (RAM) and may also include non-volatile memory (Non-Volatile Memory), such as at least one disk storage.

[0167] The above-mentioned processors can be general-purpose processors, including central processing units (CPUs), network processors (NPs), graphics processing units (GPUs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0168] It should be noted that an embodiment of the present invention further provides a computer-readable storage medium, which stores instructions. When the computer-readable storage medium is run on a computer, it enables the computer to execute the methods and processing procedures provided in the above embodiments.

[0169] An embodiment of the present invention further provides a chip for executing instructions, which is used to execute the processing steps described in the above method embodiment.

[0170] Embodiments of the present invention provide a processing method, apparatus, electronic device, and computer-readable storage medium for self-supervised learning based on multimodal information. This method uses self-supervised learning to train a neural network, combining a ring view rich in environmental information with a point cloud to form a multimodal information sample set. A multimodal loss function is constructed based on the InfoNCE loss function. This method eliminates the need for annotating three-dimensional point clouds, significantly reducing data preparation time. Furthermore, the point cloud bird's-eye view feature extraction network is trained based on multimodal information, improving its learning accuracy and training adequacy.

[0171] Professionals should also be further aware that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0172] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein may be implemented using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0173] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A processing method for self-supervised learning based on multimodal information, characterized in that: The method comprises: Selecting any first training data record from the first training data set to perform original point cloud and surround view extraction processing to generate a corresponding original point cloud and first surround view; Select two methods from a plurality of data enhancement methods to perform corresponding data enhancement processing on the original point cloud to generate corresponding first and second point clouds; and perform ground point cloud filtering processing on the first and second point clouds; Inputting the first point cloud and the second point cloud into a point cloud bird's-eye view feature extraction network for feature extraction processing to generate corresponding first feature maps and second feature maps; inputting the first surround view into a well-trained image bird's-eye view feature extraction network for feature extraction processing to generate a corresponding third feature map; and performing first multimodal information sample preparation processing based on the first, second, and third feature maps to obtain corresponding first, second, and third sample sequences; and performing bird's-eye view plane clustering on the first and second point clouds according to a preset number of clusters Q to obtain corresponding first and second plane cluster space sequences; and performing sample clustering processing on the first, second, and third sample sequences based on the first and second plane cluster space sequences to generate corresponding first, second, and third cluster sample sequence sets; and performing second multimodal information sample preparation processing on the first, second, and third cluster sample sequence sets to obtain corresponding fourth, fifth, and sixth sample sequences; A first multimodal loss function L1, a second multimodal loss function L2 and an overall loss function L are constructed according to the first, second, third, fourth, fifth and sixth sample sequences. all , and according to the first multimodal loss function L1, the second multimodal loss function L2 and the overall loss function L all The network parameters of the point cloud bird's-eye view feature extraction network are learned.

2. The method for self-supervised learning based on multimodal information according to claim 1, characterized in that: The first training data set includes a plurality of first training data records; the first training data record includes a first scene identification field, a first scene information field, a first point cloud field and a first surround view field; The multiple data enhancement methods include rotation, noise addition, downsampling and random occlusion; The plane spatial range of the bird's-eye view plane of the point cloud bird's-eye view feature extraction network and the image bird's-eye view feature extraction network is consistent and recorded as the corresponding first bird's-eye view plane; the output feature shape of the point cloud bird's-eye view feature extraction network and the image bird's-eye view feature extraction network is the same as that of the image bird's-eye view feature extraction network, both of which are H×W×D, where H, W, and D are all positive integers, H and W are the height and width of the preset bird's-eye view feature map, and D is the feature dimension of the preset bird's-eye view feature map; the first bird's-eye view plane includes H×W first bird's-eye view grids r i,j ,1≤i≤H,1≤j≤W; The shapes of the first, second and third feature maps are all H×W×D; the first feature map consists of H*W first feature vectors a with a length of D i,j Composition, each of the first eigenvectors a i,j Corresponding to the first bird's-eye view grid r i,j The second feature map consists of H*W second feature vectors b of length D i,j The second eigenvector b i,j Corresponding to the first bird's-eye view grid r i,j ; The third feature map consists of H*W third feature vectors c of length D i,j The second eigenvector b i,j Corresponding to the first bird's-eye view grid r i,j ; The first planar clustering space sequence includes the first planar clustering space s of the cluster quantity Q. 1,f , 1≤f≤Q; the second plane clustering space sequence includes the second plane clustering space s of the cluster number Q 2,f ; The first plane clustering space s with the same sequence index f 1,f With the second plane clustering space s 2,f The spatial distance is the shortest; The first cluster sample sequence set includes multiple first cluster sample sequences; the first cluster sample sequence includes multiple first cluster samples; the second cluster sample sequence set includes multiple second cluster sample sequences; the second cluster sample sequence includes multiple second cluster samples; the third cluster sample sequence set includes multiple third cluster sample sequences; and the third cluster sample sequence includes multiple third cluster samples.

3. The method for self-supervised learning based on multimodal information according to claim 2, characterized in that: The point cloud bird's-eye view feature extraction network is a point cloud pillar feature network of the PointPillars network.

4. The method for self-supervised learning based on multimodal information according to claim 2, characterized in that: The step of extracting and processing the original point cloud and the surround view from any one of the first training data sets to generate the corresponding original point cloud and the first surround view specifically includes: Select any one of the first training data records from the first training data set as the current training data record; and extract the first point cloud field and the first surround view field of the current training data record as the corresponding original point cloud and the first surround view.

5. The method for self-supervised learning based on multimodal information according to claim 2, characterized in that: The step of performing corresponding data enhancement processing on the original point cloud to generate the corresponding first and second point clouds by selecting two of the plurality of data enhancement methods specifically includes: Select any two methods from the multiple data enhancement methods as the corresponding first and second methods; and based on the first method, perform corresponding first data enhancement processing on the original point cloud to generate the corresponding first point cloud; and based on the second method, perform corresponding second data enhancement processing on the original point cloud to generate the corresponding second point cloud.

6. The method for self-supervised learning based on multimodal information according to claim 2, characterized in that: The performing ground point cloud filtering processing on the first and second point clouds specifically includes: All points with a height coordinate of 0 in the first and second point clouds are marked as ground points; and all the ground points in the first and second point clouds are deleted.

7. The method for self-supervised learning based on multimodal information according to claim 2, characterized in that: The performing the first multimodal information sample preparation processing according to the first, second, and third feature maps to obtain corresponding first, second, and third sample sequences specifically includes: For each of the first bird's-eye view grids r of the first bird's-eye view plane i,j Traverse; when traversing, the first bird's-eye view grid r currently traversed i,j As the corresponding current grid; and the first feature vector a corresponding to the current grid i,j The second eigenvector b i,j and the third eigenvector c i,j Are they all zero vectors? If so, the first feature vector a corresponding to the current grid is used. i,j The second eigenvector b i,j and the third eigenvector c i,j Form the corresponding first non-zero vector group; The total number of the obtained first non-zero vector groups is counted to generate a corresponding first quantity M; and the M first feature vectors a corresponding to the M first non-zero vector groups are i,j , M second eigenvectors b i,j and the M third eigenvectors c i,j The first, second and third sample sequences are formed accordingly; the first sample sequence includes M first samples P 1,k , each of the first samples P 1,k Corresponding to the first eigenvector a i,j , 1≤k≤M; the second sample sequence includes M second samples P 2,k , each of the second samples P 2,k Corresponding to the second eigenvector b i,j The third sample sequence includes M third samples P 3,k , each of the third samples P 3,k Corresponding to the third eigenvector c i,j The first sample P with the same subscript k 1,k The second sample P 2,k and the third sample P 3,k The corresponding first eigenvector a i,j The second eigenvector b i,j and the third eigenvector c i,j belong to the same first non-zero vector group.

8. The method for self-supervised learning based on multimodal information according to claim 2, characterized in that: The step of performing bird's-eye view planar clustering on the first and second point clouds according to the preset number of clusters Q to obtain corresponding first and second planar clustering space sequences specifically includes: Based on a preset unsupervised clustering algorithm and the set number of clusters Q, point cloud clustering processing is performed on the first point cloud to obtain first point cloud clusters of the number of clusters Q; and based on the unsupervised clustering algorithm and the set number of clusters Q, point cloud clustering processing is performed on the second point cloud to obtain second point cloud clusters of the number of clusters Q; and the three-dimensional point cloud spaces of each of the first and second point cloud clusters are used as corresponding first and second point cloud cluster spaces; Project each of the first point cloud clustering spaces onto the first bird's-eye view plane to obtain the corresponding first plane clustering space s 1,f , 1≤f≤Q; and all the first plane clustering spaces s obtained 1,f forming the corresponding first plane clustering space sequence; Project each of the second point cloud clustering spaces onto the first bird's-eye view plane to obtain the corresponding first projection space; and combine it with the first plane clustering space s 1,f＝1 The first projection space with the shortest spatial distance is used as the corresponding second planar clustering space s 2,f＝1 , and will be clustered with the first plane space s 1,f＝2 The first projection space with the shortest spatial distance is used as the corresponding second planar clustering space s 2,f＝2 , and so on, until the first plane clustering space s 1,f＝Q The first projection space with the shortest spatial distance is used as the corresponding second planar clustering space s 2,f＝Q and the second plane clustering space s of the obtained cluster number Q 2,f The corresponding second plane clustering space sequence is formed.

9. The method for self-supervised learning based on multimodal information according to claim 7, characterized in that: The performing sample clustering processing on the first, second, and third sample sequences according to the first and second planar clustering space sequences to generate corresponding first, second, and third cluster sample sequence sets specifically includes: Randomly select one of the first and second planar clustering space sequences as the corresponding third planar clustering space sequence; the third planar clustering space sequence includes the third planar clustering space s of the cluster quantity Q. 3,f ; The first plane clustering space s of the first plane clustering space sequence 1,f Traverse one by one; and when traversing, the first plane clustering space s currently traversed 1,f as the corresponding current space; and the first bird's-eye view grid r on the first bird's-eye view plane that is fully or partially covered by the current space i,j are all marked as a type of grid; and the first feature vector a corresponding to each of the first type of grids in the first feature map is i,j are all marked as a class of vectors; and the first sample P corresponding to each of the first class of vectors in the first sample sequence is 1,k extracting them as the corresponding first cluster samples; and forming the corresponding first cluster sample sequence from all the obtained first cluster samples; and at the end of the traversal, forming the corresponding first cluster sample sequence set from all the obtained first cluster sample sequences; The second plane clustering space s of the second plane clustering space sequence 2,f Traverse one by one; and when traversing, the second plane clustering space s currently traversed 2,f as the corresponding current space; and the first bird's-eye view grid r on the first bird's-eye view plane that is fully or partially covered by the current space i,j are all marked as second-class grids; and the second feature vector b corresponding to each of the second-class grids in the second feature map is i,j are all marked as second-class vectors; and the second samples P corresponding to each of the second-class vectors in the second sample sequence are marked as second-class vectors; 2,k extracting them as the corresponding second cluster samples; and forming a corresponding second cluster sample sequence from all the obtained second cluster samples; and forming a corresponding second cluster sample sequence set from all the obtained second cluster sample sequences at the end of the traversal; The third plane clustering space s of the third plane clustering space sequence 3,f Traverse one by one; and when traversing, the third plane clustering space s currently traversed 3,f as the corresponding current space; and the first bird's-eye view grid r on the first bird's-eye view plane that is fully or partially covered by the current space i,j are all marked as three types of grids; and the third eigenvector c corresponding to each of the three types of grids in the third feature map is i,j are all marked as three types of vectors; and the third samples P corresponding to each of the three types of vectors in the third sample sequence are ... 3,k Extract them as the corresponding third cluster samples; and form the corresponding third cluster sample sequence from all the obtained third cluster samples; and at the end of the traversal, form the corresponding third cluster sample sequence set from all the obtained third cluster sample sequences.

10. The method for self-supervised learning based on multimodal information according to claim 9, characterized in that: The performing second multimodal information sample preparation processing according to the first, second, and third clustered sample sequence sets to obtain corresponding fourth, fifth, and sixth sample sequences specifically includes: Input each of the first cluster sample sequences in the first cluster sample sequence set into a maturely trained maximum pooling network for calculation, and use the output vector of the maximum pooling network as the corresponding fourth sample P 4,f ; and all the fourth samples P obtained 4,f forming the corresponding fourth sample sequence; Each second cluster sample sequence in the second cluster sample sequence set is input into the maturely trained maximum pooling network for calculation, and the output vector of the maximum pooling network is used as the corresponding fifth sample P 5,f ; and all the fifth samples P obtained 5,f forming the corresponding fifth sample sequence; Each of the third cluster sample sequences in the third cluster sample sequence set is input into a maturely trained maximum pooling network for calculation, and the output vector of the maximum pooling network is used as the corresponding sixth sample P 6,f ; and all the sixth samples P obtained 6,f to form the corresponding sixth sample sequence.

11. The method for self-supervised learning based on multimodal information according to claim 10, characterized in that: The first multimodal loss function L1, the second multimodal loss function L2 and the overall loss function L are constructed according to the first, second, third, fourth, fifth and sixth sample sequences. all , and according to the first multimodal loss function L1, the second multimodal loss function L2 and the overall loss function L all The network parameters of the point cloud bird's-eye view feature extraction network are learned, specifically including: According to the first, second and third sample sequences, the first multimodal loss function L1 is confirmed to be L infoNCE () is the InfoNCE loss function, M is the number of samples in the first, second and third sample sequences; The second multimodal loss function L2 is confirmed to be: Q is the number of clusters; The overall loss function L is determined based on the first multimodal loss function L1 and the second multimodal loss function L2. all for L all =L1+L2; Substituting each sample of the first, second, and third sample sequences into the first multimodal loss function L1 to calculate and generate a corresponding first loss value; and identifying whether the first loss value satisfies a preset first loss value convergence range; if not, modulating the network parameters of the point cloud bird's-eye view feature extraction network based on a direction that minimizes the first multimodal loss function L1; Substituting each sample of the fourth, fifth, and sixth sample sequences into the second multimodal loss function L2 to calculate a corresponding second loss value; and identifying whether the second loss value satisfies a preset second loss value convergence range; if not, modulating the network parameters of the point cloud bird's-eye view feature extraction network based on a direction that minimizes the second multimodal loss function L2; Substitute each sample of the first, second, third, fourth, fifth and sixth sample sequences into the overall loss function L all Calculate the corresponding overall loss value; and identify whether the overall loss value meets the preset overall loss value convergence range; if not, based on making the overall loss function L all The direction in which the minimum value is reached modulates the network parameters of the point cloud bird's-eye view feature extraction network.

12. A device for executing the processing method for self-supervised learning based on multimodal information according to any one of claims 1 to 11, characterized in that: The device includes: an acquisition module, a data preprocessing module, a first multimodal information processing module, a second multimodal information processing module and a multimodal information learning processing module; The acquisition module is used to select a first training data record from the first training data set to perform original point cloud and surround view extraction processing to generate a corresponding original point cloud and first surround view; The data preprocessing module is used to select two methods from a plurality of data enhancement methods to perform corresponding data enhancement processing on the original point cloud to generate corresponding first and second point clouds; and perform ground point cloud filtering processing on the first and second point clouds; The first multimodal information processing module is configured to input the first point cloud and the second point cloud into a point cloud bird's-eye view feature extraction network for feature extraction processing to generate corresponding first feature maps and second feature maps; input the first surround view into a well-trained image bird's-eye view feature extraction network for feature extraction processing to generate a corresponding third feature map; and perform first multimodal information sample preparation processing based on the first, second, and third feature maps to obtain corresponding first, second, and third sample sequences; The second multimodal information processing module is configured to perform bird's-eye view planar clustering on the first and second point clouds according to a preset number of clusters Q to obtain corresponding first and second planar cluster space sequences; perform sample clustering processing on the first, second, and third sample sequences based on the first and second planar cluster space sequences to generate corresponding first, second, and third cluster sample sequence sets; and perform second multimodal information sample preparation processing on the first, second, and third cluster sample sequence sets to obtain corresponding fourth, fifth, and sixth sample sequences; The multimodal information learning processing module is used to construct a first multimodal loss function L1, a second multimodal loss function L2 and an overall loss function L according to the first, second, third, fourth, fifth and sixth sample sequences. all , and according to the first multimodal loss function L1, the second multimodal loss function L2 and the overall loss function L all The network parameters of the point cloud bird's-eye view feature extraction network are learned.

13. An electronic device, characterized in that: include: memory, processors, and transceivers; The processor is configured to be coupled to the memory, read and execute instructions in the memory, so as to implement the method steps described in any one of claims 1 to 11; The transceiver is coupled to the processor, and the processor controls the transceiver to send and receive messages.

14. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer instructions, which, when executed by a computer, enable the computer to execute the method according to any one of claims 1 to 11.

Images