Multimodal data fusion method and device, computer device and storage medium

Abstract

The application relates to a multi-modal data fusion method and device, computer equipment and a storage medium. The method comprises the following steps: acquiring to-be-processed multi-modal data; the to-be-processed multi-modal data comprises to-be-processed point cloud data and to-be-processed image data; a pre-constructed feature extraction model is called; the feature extraction model is obtained by training a deep mining result of sample multi-modal data; the to-be-processed point cloud data and the to-be-processed image data are subjected to feature extraction by the feature extraction model respectively, so that first deep layer feature data corresponding to the to-be-processed point cloud data and second deep layer feature data corresponding to the to-be-processed image data are obtained; and the first deep layer feature data and the second deep layer feature data are fused by the feature extraction model, so that deep layer fusion features corresponding to the to-be-processed multi-modal data are obtained. The method can improve the fusion accuracy of multi-modal data.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of automatic driving, and in particular to a multi-modal data fusion method and device, computer equipment, a storage medium and a computer program product. BACKGROUND

[0002] The rapid development of sensor technologies such as cameras, lidar, and millimeter wave radar has promoted the progress of automatic driving perception capabilities. The data collected by different sensors corresponds to different modalities and also reflects the perception of the real world from different angles by the automatic driving system. The images collected by the camera contain more texture information such as color, while the point cloud collected by the radar contains more comprehensive spatial position information. The fusion method based on the point cloud modality and the image modality can further promote the development of automatic driving perception technology.

[0003] In the traditional way, the point cloud data and the image data are superimposed in the visible range to realize multi-modal data fusion.

[0004] However, the traditional way cannot obtain more target and scene information close to the real world, resulting in low fusion accuracy of multi-modal data. SUMMARY

[0005] Therefore, it is necessary to provide a multi-modal data fusion method, device, computer equipment, computer readable storage medium and computer program product capable of improving the fusion accuracy of multi-modal data to solve the above technical problems.

[0006] In a first aspect, the present application provides a multi-modal data fusion method. The method comprises:

[0007] obtaining to-be-processed multi-modal data; the to-be-processed multi-modal data comprises to-be-processed point cloud data and to-be-processed image data;

[0008] calling a pre-constructed feature extraction model;

[0009] extracting features from the to-be-processed point cloud data and the to-be-processed image data respectively through the feature extraction model to obtain first deep feature data corresponding to the to-be-processed point cloud data and second deep feature data corresponding to the to-be-processed image data;

[0010] fusing the first deep feature data and the second deep feature data through the feature extraction model to obtain deep fusion features corresponding to the to-be-processed multi-modal data.

[0011] In one embodiment, before obtaining the to-be-processed multi-modal data, the method further comprises:

[0012] obtaining sample multi-modal data; the sample multi-modal data comprises sample point cloud data and sample image data;

[0013] inputting the sample multi-modal data into a deep learning model to be trained; the deep learning model comprises a feature extraction layer, a deep mining layer, and a perception layer;

[0014] extracting, by the feature extraction layer, sample point cloud features corresponding to the sample point cloud data and sample image features corresponding to the sample image data, fusing the sample point cloud features and the sample image features to obtain sample fused features;

[0015] deep mining, by the deep mining layer, the sample point cloud features and the sample image features to obtain a deep mining result;

[0016] training, by the perception layer, the deep learning model according to the sample fused features and the deep mining result to obtain a pre-constructed feature extraction model.

[0017] In one of the embodiments, the deep mining result is a relationship loss value; the deep mining, by the deep mining layer, of the sample point cloud features and the sample image features to obtain the deep mining result comprises:

[0018] deep mining, by the deep mining layer, the sample point cloud features and the sample image features to obtain mining feature data;

[0019] determining the relationship loss value according to the mining feature data.

[0020] In one of the embodiments, the perception layer comprises a perception task layer and a training optimization layer; the training, by the perception layer, of the deep learning model according to the sample fused features and the deep mining result comprises:

[0021] executing, by the perception task layer, a preset perception task according to the sample fused features to obtain a task execution result;

[0022] determining, by the training optimization layer, an overall loss value of the deep learning model according to the task execution result and the relationship loss value, and training the deep learning model according to the overall loss value.

[0023] In one of the embodiments, the preset perception task is a target detection task; the determining, by the training optimization layer, of the overall loss value of the deep learning model according to the task execution result and the relationship loss value comprises:

[0024] determining, by the training optimization layer, a position loss value, a direction loss value, and a category loss value of the deep learning model according to the task execution result;

[0025] determining, by the training optimization layer, the overall loss value of the deep learning model according to the position loss value, the direction loss value, the category loss value, and the relationship loss value.

[0026] In one of the embodiments, the feature extraction model is used to extract features from the to-be-processed point cloud data and the to-be-processed image data respectively, to obtain first deep feature data corresponding to the to-be-processed point cloud data and second deep feature data corresponding to the to-be-processed image data, including:

[0027] The feature extraction model is used to extract features from the to-be-processed point cloud data, to obtain first deep feature data corresponding to the to-be-processed point cloud data; the first deep feature data includes deep point cloud features and associated features between the to-be-processed point cloud data and the to-be-processed image data;

[0028] The feature extraction model is used to perform semantic segmentation on the to-be-processed image data, to obtain a semantic segmentation result, and the semantic segmentation result is clustered to obtain a clustering result;

[0029] The feature extraction model is used to up-sample the clustering result, to obtain second deep feature data corresponding to the to-be-processed image data; the second deep feature data includes deep image features and associated features between the to-be-processed image data and the to-be-processed point cloud data.

[0030] In a second aspect, the present application further provides a multi-modal data fusion device. The device includes:

[0031] A data acquisition module is configured to acquire to-be-processed multi-modal data; the to-be-processed multi-modal data includes to-be-processed point cloud data and to-be-processed image data;

[0032] A model calling module is configured to call a pre-constructed feature extraction model;

[0033] A feature extraction module is configured to use the feature extraction model to extract features from the to-be-processed point cloud data and the to-be-processed image data respectively, to obtain first deep feature data corresponding to the to-be-processed point cloud data and second deep feature data corresponding to the to-be-processed image data;

[0034] A feature fusion module is configured to use the feature extraction model to fuse the first deep feature data and the second deep feature data, to obtain deep fusion features corresponding to the to-be-processed multi-modal data.

[0035] In a third aspect, the present application further provides a computer device. The computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the following steps when executing the computer program:

[0036] Acquire to-be-processed multi-modal data; the to-be-processed multi-modal data includes to-be-processed point cloud data and to-be-processed image data;

[0037] Call a pre-constructed feature extraction model;

[0038] The feature extraction model is used for respectively extracting features of the to-be-processed point cloud data and the to-be-processed image data, to obtain first deep feature data corresponding to the to-be-processed point cloud data and second deep feature data corresponding to the to-be-processed image data.

[0039] The first deep feature data and the second deep feature data are fused by the feature extraction model to obtain deep fusion features corresponding to the to-be-processed multi-modal data.

[0040] In a fourth aspect, the present application further provides a computer readable storage medium. The computer readable storage medium has a computer program stored thereon, and the computer program is executed by a processor to implement the following steps:

[0041] Obtaining to-be-processed multi-modal data; the to-be-processed multi-modal data includes to-be-processed point cloud data and to-be-processed image data;

[0042] Calling a pre-constructed feature extraction model;

[0043] The feature extraction model is used for respectively extracting features of the to-be-processed point cloud data and the to-be-processed image data, to obtain first deep feature data corresponding to the to-be-processed point cloud data and second deep feature data corresponding to the to-be-processed image data;

[0044] The first deep feature data and the second deep feature data are fused by the feature extraction model to obtain deep fusion features corresponding to the to-be-processed multi-modal data.

[0045] In a fifth aspect, the present application further provides a computer program product. The computer program product includes a computer program, and the computer program is executed by a processor to implement the following steps:

[0046] Obtaining to-be-processed multi-modal data; the to-be-processed multi-modal data includes to-be-processed point cloud data and to-be-processed image data;

[0047] Calling a pre-constructed feature extraction model;

[0048] The feature extraction model is used for respectively extracting features of the to-be-processed point cloud data and the to-be-processed image data, to obtain first deep feature data corresponding to the to-be-processed point cloud data and second deep feature data corresponding to the to-be-processed image data;

[0049] The first deep feature data and the second deep feature data are fused by the feature extraction model to obtain deep fusion features corresponding to the to-be-processed multi-modal data.

[0050] The aforementioned multimodal data fusion methods, devices, computer equipment, storage media, and computer program products, because the feature extraction model is obtained by deep mining of sample multimodal data and trained based on the deep mining results, can fully mine and extract the non-intuitively visible deep feature data of multimodal data and fuse them. The resulting deep fused features are closer to the target and scene information of the real world, which greatly improves the accuracy of multimodal data fusion and thus enhances the perception ability of autonomous driving perception technology to perceive the surrounding environment. Attached Figure Description

[0051] Figure 1 This is a diagram illustrating the application environment of a multimodal data fusion method in one embodiment.

[0052] Figure 2 This is a flowchart illustrating a multimodal data fusion method in one embodiment;

[0053] Figure 3 This is a flowchart illustrating the training steps of a feature extraction model in one embodiment;

[0054] Figure 4 This is a schematic diagram of the network structure of a deep learning model in one embodiment;

[0055] Figure 5 This is a structural block diagram of a multimodal data fusion device in one embodiment;

[0056] Figure 6 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0057] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0058] The multimodal data fusion method provided in this application embodiment can be applied to, for example... Figure 1The application environment is shown. Among them, the multi-modal data fusion method is mainly executed on the computer device 102. Specifically, in an automatic driving environment, a vehicle is pre-installed with a vehicle-mounted sensor 102 and a vehicle-mounted computer device 104. The vehicle-mounted computer device can be referred to as a computer device. The vehicle-mounted sensor 102 collects multi-modal data to be processed, and transmits the collected multi-modal data to be processed to the computer device 104, so that the computer device 104 calls a pre-constructed feature extraction model. The feature extraction model is obtained by training according to the deep mining result of the sample multi-modal data, so as to extract features from the multi-modal data to be processed by the feature extraction model, and obtain the first deep feature data corresponding to the multi-modal data to be processed. The second deep feature data corresponding to the image data to be processed is obtained, and then the first deep feature data and the second deep feature data are fused by the feature extraction model to obtain the deep fusion feature corresponding to the multi-modal data to be processed. Among them, the vehicle-mounted sensor 102 can include various image acquisition devices and video acquisition devices for collecting image data to be processed, and various radar sensors for collecting point cloud data to be processed. The computer device 104 can be, but is not limited to, various personal computers, notebook computers, smart phones, tablet computers and smart vehicle-mounted devices.

[0059] In one embodiment, as shown in Figure 2 , a multi-modal data fusion method is provided. The method is applied to the computer device in Figure 1 for example, including the following steps:

[0060] Step 202, obtaining multi-modal data to be processed; the multi-modal data to be processed includes point cloud data to be processed and image data to be processed.

[0061] Among them, the multi-modal data to be processed refers to multi-modal perception data that needs to be fused.

[0062] Specifically, in an automatic driving environment, a plurality of vehicle-mounted sensors installed on a vehicle scan the environment around the vehicle to obtain point cloud data to be processed and image data to be processed. Since different types of sensors collect data corresponding to different modalities, the point cloud data to be processed and the image data to be processed can be referred to as multi-modal data to be processed. Optionally, the plurality of vehicle-mounted sensors can include a first sensor and a second sensor. The first sensor can include various image acquisition devices and video acquisition devices for collecting image data to be processed, such as cameras, cameras, etc. The second sensor can include various radar sensors for collecting point cloud data to be processed, such as laser radar, millimeter wave radar, etc. Thus, the vehicle-mounted sensor transmits the collected multi-modal data to be processed to the computer device.

[0063] Step 204, calling a pre-constructed feature extraction model.

[0064] The deep mining refers to mining non-intuitive visible deep feature data in the sample multi-modal data.

[0065] The computer device stores a pre-constructed feature extraction model, and the feature extraction model is used to extract deep feature data of multi-modal data. The feature extraction model is obtained by deep mining a large amount of sample multi-modal data to obtain a deep mining result, and is trained according to the deep mining result. The deep mining result can include deep semantic features and internal correlation of the sample multi-modal data. The computer device calls the pre-constructed feature extraction model to extract deep features of the to-be-processed multi-modal data.

[0066] In step 206, the feature extraction model is used to extract features of the to-be-processed point cloud data and the to-be-processed image data respectively, to obtain first deep feature data corresponding to the to-be-processed point cloud data and second deep feature data corresponding to the to-be-processed image data.

[0067] The first deep feature data refers to non-intuitive visible deep feature data in the to-be-processed point cloud data. The second deep feature data refers to non-intuitive visible deep feature data in the to-be-processed image data. The first deep feature data and the second deep feature data are both high-dimensional features.

[0068] Specifically, the feature extraction model can include a feature extraction layer. The feature extraction layer is used to extract features of the to-be-processed point cloud data and the to-be-processed image data, and fuse the extracted features. Further, the feature extraction layer can include two network branches, a point cloud branch and an image branch. The point cloud branch is used to extract features of the to-be-processed point cloud data to obtain first deep feature data corresponding to the to-be-processed point cloud data. The first deep feature data can include deep semantic features corresponding to the to-be-processed point cloud data, and internal correlation between the to-be-processed point cloud data and the to-be-processed image data. The image branch is used to extract features of the to-be-processed image data to obtain second deep feature data corresponding to the to-be-processed image data. The second deep feature data can include deep semantic features corresponding to the to-be-processed image data, and internal correlation between the to-be-processed image data and the to-be-processed point cloud data.

[0069] In one of the embodiments, the feature extraction model is used to extract features from the to-be-processed point cloud data and the to-be-processed image data respectively, to obtain first deep feature data corresponding to the to-be-processed point cloud data and second deep feature data corresponding to the to-be-processed image data, including: extracting features from the to-be-processed point cloud data by the feature extraction model to obtain the first deep feature data corresponding to the to-be-processed point cloud data; the first deep feature data includes deep point cloud features and associated features between the to-be-processed point cloud data and the to-be-processed image data; performing semantic segmentation on the to-be-processed image data by the feature extraction model to obtain a semantic segmentation result, and clustering the semantic segmentation result to obtain a clustering result; upsampling the clustering result by the feature extraction model to obtain the second deep feature data corresponding to the to-be-processed image data; the second deep feature data includes deep image features and associated features between the to-be-processed image data and the to-be-processed point cloud data.

[0070] The deep point cloud features refer to deep semantic features in the to-be-processed point cloud data. The associated features between the to-be-processed point cloud data and the to-be-processed image data refer to the internal association relationship between the to-be-processed point cloud data and the to-be-processed image data. The deep image features refer to deep semantic features in the to-be-processed image data. The associated features between the to-be-processed image data and the to-be-processed point cloud data refer to the internal association relationship between the to-be-processed image data and the to-be-processed point cloud data.

[0071] Specifically, the feature extraction model can include a feature extraction layer, and the to-be-processed point cloud data is extracted by a point cloud branch in the feature extraction layer to obtain the first deep feature data including the deep point cloud features and the associated features between the to-be-processed point cloud data and the to-be-processed image data. The to-be-processed image data is processed by an image branch in the feature extraction layer to perform semantic segmentation on the to-be-processed image data and obtain a semantic segmentation result. The semantic segmentation result includes the category of each pixel point in the to-be-processed image data. Thus, the semantic segmentation result is clustered to cluster each pixel point according to the category of each pixel point. The clustering result is then upsampled so that each pixel point in the to-be-processed image data has the same feature dimension as the point cloud in the first deep feature data, and the second deep feature data including the deep image features and the associated features between the to-be-processed point cloud data is obtained. For example, the deep point cloud features corresponding to the to-be-processed point cloud data and the deep image features corresponding to the to-be-processed image data can be more abstract high-dimensional floating-point features. The key features can be similar features or identical features.

[0072] Further, the point cloud branch and the image branch mainly utilize an encoder, i.e., a convolution network, to extract deep feature data from the to-be-processed point cloud data and the to-be-processed image data respectively. The point cloud branch can include an encoder, which increases the feature dimension of the to-be-processed point cloud data to a preset dimension, so as to extract high-dimensional and deep feature data, i.e., first deep feature data, in the to-be-processed point cloud data. For example, the to-be-processed point cloud data can be represented as (P, [x, y, z, i]), and the feature dimension of (P, [x, y, z, i]) is increased from 4 to C by the encoder in the point cloud branch, and the dimension of the output first deep feature data F_pt is (P, C), wherein P represents the number of points in the to-be-processed point cloud data, and C represents the preset dimension, which is an intermediate feature number, such as 10, 64, etc.

[0073] Unlike the point cloud branch, the image branch can include two encoders, the first encoder performs semantic segmentation on the to-be-processed image, and the clustering processing is performed on the semantic segmentation result, so as to perform upsampling on the clustering result by the second encoder to obtain second deep feature data. For example, when the dimension of the first deep feature data F_pt is (P, C), each pixel point in the second deep feature data F_img has the same feature dimension C as the point cloud in the first deep feature data, and the dimension of the second deep feature data F_img is (H, W, C), wherein H represents the height of the second deep feature data, W represents the width of the second deep feature data, and C represents the feature dimension of the second deep feature data.

[0074] In step 208, the first deep feature data and the second deep feature data are fused by the feature extraction model to obtain deep fusion features corresponding to the to-be-processed multi-modal data.

[0075] The first deep feature data and the second deep feature data are fused by the feature extraction layer in the feature extraction model.

[0076] Further, the feature extraction layer further includes a fusion layer (Fusion layer), such as a 1x1 convolution kernel, which is connected to the output ends of the encoders in the point cloud branch and the image branch respectively. The first deep feature data output by the encoder in the point cloud branch and the second deep feature data output by the second encoder in the image branch are input to the fusion layer, and the fusion layer aligns each pixel point in the second deep feature data with the points in the first deep feature data according to a preset extrinsic parameter matrix to obtain aligned second deep feature data, so as to fuse the aligned second deep feature data with the first deep feature data to obtain deep fusion features F_fuse corresponding to the to-be-processed multi-modal data, and the shape of F_fuse is (P, C1), wherein P represents the number of point cloud points, and C1 represents the feature dimension.

[0077] In the conventional manner, the point cloud data and the image data are superimposed in the visible range by models such as PointPainting (image laser fusion model) and RoarNet (RegiOn Approximation Refinement Network, target detection network) to realize multi-modal data fusion, which focuses on superimposing shallow features, i.e., shapes, colors, spatial positions, etc. that can be seen by the naked eye and easily understood intuitively, and ignores the internal deep semantic relationship of multi-modal data, i.e., high-dimensional features that cannot be clearly represented by two-dimensional or three-dimensional features, and is only a one-way superposition of multiple modalities, such as superimposing images onto point clouds, without mutual supervision and mutual learning between multiple modalities. Therefore, the conventional manner cannot fully mine the deep semantic features and internal relationship of multi-modal data.

[0078] In the above multi-modal data fusion method, a pre-constructed feature extraction model is called to extract features from the to-be-processed point cloud data and the to-be-processed image data in the to-be-processed multi-modal data, respectively, to obtain first deep feature data corresponding to the to-be-processed point cloud data and second deep feature data corresponding to the to-be-processed image data, and then the first deep feature data and the second deep feature data are fused to obtain deep fusion features corresponding to the to-be-processed multi-modal data. Since the feature extraction model is trained according to the deep mining results of sample multi-modal data, it can fully mine and extract deep feature data of multi-modal data that is not directly visible, and the deep fusion features obtained by fusion are closer to the target and scene information in the real world, greatly improving the fusion accuracy of multi-modal data and thus improving the perception ability of automatic driving perception technology to the surrounding environment.

[0079] In one of the embodiments, the method further includes: extracting features from the deep fusion features to obtain target features; and performing an automatic driving task according to the target features.

[0080] The feature extraction model can further include a perception task layer configured to perform the automatic driving task.

[0081] After the feature extraction layer in the feature extraction model outputs the deep fusion features, the deep fusion features are input to the perception task layer, which can include an extraction layer (Extract layer) and a perception task head (3D Task Head). For example, the extraction layer can be Voxel, Pillar, etc. The deep fusion features are extracted by the extraction layer to obtain target features. The target features refer to features obtained by performing point cloud feature extraction on the deep fusion features. The target features are input to the perception task head, and the perception task head performs an automatic driving task according to the target features. For example, the automatic driving task can be a 3D target detection task, a target segmentation task, etc.

[0082] In the embodiment, since the deep fusion feature can be closer to the real-world scene, the target feature is obtained by performing feature extraction on the deep fusion feature, which can provide the perception task head with data meeting the requirements, and is beneficial to the subsequent perception task head to perform the automatic driving task.

[0083] In one embodiment, as shown in Figure 3 Before the multi-modal data to be processed is acquired, the method further includes a training step of the feature extraction model, which includes:

[0084] In step 302, sample multi-modal data is acquired; the sample multi-modal data includes sample point cloud data and sample image data.

[0085] In step 304, the sample multi-modal data is input into the deep learning model to be trained; the deep learning model includes a feature extraction layer, a deep mining layer, and a perception layer.

[0086] The sample multi-modal data refers to multi-modal data used for training the deep learning model.

[0087] In the model training process, the computer device can first acquire sample multi-modal data. The sample multi-modal data can include sample point cloud data and sample image data. The sample multi-modal data is input into the deep learning model to be trained, and the deep learning model can include a feature extraction layer, a deep mining layer, and a perception layer. The feature extraction layer and the deep mining layer each include a point cloud branch and an image branch. The point cloud branch in the feature extraction layer is connected to the point cloud branch in the deep mining layer, and the image branch in the feature extraction layer is connected to the image branch in the deep mining layer. The point cloud data to be processed and the image data to be processed are processed through two different branches, i.e., the point cloud data to be processed is processed through the point cloud branch in the deep learning model, and the image data to be processed is processed through the image branch.

[0088] In step 306, the sample point cloud feature corresponding to the sample point cloud data and the sample image feature corresponding to the sample image data are extracted through the feature extraction layer, the sample point cloud feature and the sample image feature are fused, and a sample fusion feature is obtained.

[0089] The sample point cloud feature corresponding to the sample point cloud data is extracted through the point cloud branch in the feature extraction layer, and the sample image feature corresponding to the sample image data is extracted through the image branch. The sample point cloud feature and the sample image feature are both high-dimensional features. The sample point cloud feature and the sample image feature are fused through the feature extraction layer to obtain a sample fusion feature. Further, the structure of the feature extraction layer of the deep learning model is the same as that of the feature extraction layer of the feature extraction model, which will not be described herein.

[0090] At step 308, the sample point cloud features and the sample image features are deeply mined by the deep mining layer to obtain a deep mining result.

[0091] The sample point cloud features are deeply mined by the point cloud branch in the deep mining layer, and the sample image features are deeply mined by the image branch in the deep mining layer to obtain mining feature data. The mining feature data refers to deep features of the sample point cloud data and the sample image data, and an inherent correlation between the sample point cloud data and the sample image data. Thus, the deep mining result is calculated by the deep mining layer according to the mining feature data.

[0092] In one of the embodiments, the deep mining result is a relationship loss value. The deep mining of the sample point cloud features and the sample image features by the deep mining layer includes: the deep mining of the sample point cloud features and the sample image features by the deep mining layer to obtain the mining feature data; and the determination of the relationship loss value according to the mining feature data.

[0093] The deep mining layer includes the point cloud branch and the image branch. The sample point cloud features are deeply mined by the point cloud branch in the deep mining layer to obtain first mining features. The first mining features refer to deep features corresponding to the sample point cloud data and an inherent correlation between the sample point cloud data and the sample image data. The sample image features are deeply mined by the image branch in the deep mining layer to obtain second mining features. The second mining features refer to deep features corresponding to the sample image data and an inherent correlation between the sample image data and the sample point cloud data. The mining feature data is obtained according to the first mining features and the second mining features. Further, the relationship loss value of the deep learning model is calculated according to the mining feature data, and the relationship loss value is determined as the deep mining result.

[0094] Further, the point cloud branch and the image branch of the deep mining layer each include an encoder and a multilayer perceptron (MLP), which can also be referred to as a projection. The encoder is used to deeply mine the sample point cloud features and the sample image features. The multilayer perceptron is used to map the mining feature data to a sample label space, that is, to integrate the mining feature data into a value, which can reduce the influence of feature positions on the mining feature data and improve the robustness of the entire model. The relationship loss value of the deep learning model is calculated by the deep mining layer according to the first mining features and the second mining features, and the parameters of the encoders in the point cloud branch and the image branch in the feature extraction layer are driven to update in a direction capable of mining deep features and inherent correlations between multi-modal data according to the relationship loss value.

[0095] Further, the sample multi-modal data can be multi-modal data of multiple targets, and therefore, the mining feature data refers to mining features of multiple targets. For example, the relationship loss function of the relationship loss value can adopt an InfoNCE loss function to calculate, as shown in the following formula:

[0096]

[0097]

[0098] wherein, L relation represents the relationship loss value, N represents the number of image or point cloud frames in a batch, z i represents a deep feature of a target in the first mining feature or the second mining feature, represents a feature of the target in another modality mining feature that is the same as z i , j represents any target within a batch, Sim(z i ,z j ) represents the cosine similarity of z i and z j , and t represents a temperature hyperparameter.

[0099] From the relationship loss function, minimizing the relationship loss means that the similarity of z i and should be as large as possible, and the similarity of z i and other different data should be as small as possible. The relationship loss function makes the similarity of the region features of the same target between different modalities higher, and the similarity of the region features of different targets lower. Thus, the model can extract deeper semantic features of the mining data and the feature relationship between different modalities, which is more conducive to improving the perception ability of autonomous driving.

[0100] In step 310, the deep learning model is trained according to the sample fusion feature and the deep mining result through the perception layer, and the pre-constructed feature extraction model is obtained.

[0101] The preset perception task is performed according to the sample fusion feature through the perception layer, and a task execution result is obtained. The preset perception task refers to an autonomous driving task, which can be a 3D target detection task, a target segmentation task, etc. Then, the deep learning model is trained according to the task execution result and the deep mining result until a preset condition is met, and the pre-constructed feature extraction model is obtained. The preset condition can be that the loss value of the deep learning model no longer decreases or reaches a preset number of iterations.

[0102] In one of the embodiments, the perception layer includes a perception task layer and a training optimization layer; training the deep learning model according to the sample fusion feature and the deep mining result by the perception layer includes: performing a preset perception task according to the sample fusion feature by the perception task layer to obtain a task execution result; determining an overall loss value of the deep learning model according to the task execution result and the relationship loss value by the training optimization layer, and training the deep learning model according to the overall loss value.

[0103] The perception task layer in the perception layer performs a preset perception task according to the sample fusion feature. The structure of the perception layer in the training process is the same as that in the actual application process. The perception task layer can include an extract layer (Extract layer) and a perception task head (3DTaskHead). For example, the extract layer can be Voxel, Pillar, etc. The deep fusion feature is extracted by the extract layer, and then the perception task head performs a preset perception task, i.e., an automatic driving task, according to the extracted feature to obtain a task execution result. For example, the preset perception task can be a 3D target detection task, a target segmentation task, etc.

[0104] The training optimization layer in the perception layer calculates a perception task loss value of the deep learning model according to the task execution result and the corresponding perception task loss function. When the preset perception task is different, the corresponding perception task loss function can also be different. For example, when the preset perception task is a 3D target detection task, its perception task loss function can include a position loss, a direction loss, and a category loss. When the preset perception task is a target segmentation task, its perception task loss function can include a category loss, a bounding box loss, a mask loss, and a region focal loss. Then, the training optimization layer calculates an overall loss value of the deep learning model according to the perception task loss value and the relationship loss value. The parameters of the deep learning model are trained by adjusting the overall loss value. The overall loss value can be minimized.

[0105] Further, the preset perception task is a target detection task; determining the overall loss value of the deep learning model according to the task execution result and the relationship loss value by the training optimization layer includes: determining a position loss value, a direction loss value, and a category loss value of the deep learning model according to the task execution result by the training optimization layer; determining the overall loss value of the deep learning model according to the position loss value, the direction loss value, the category loss value, and the relationship loss value by the training optimization layer.

[0106] Specifically, when the preset perception task is a target detection task, i.e., a 3D target detection task, the corresponding perception task loss function can include a position loss, a direction loss, and a category loss. Through training and optimization, the position loss value, the direction loss value, and the category loss value of the deep learning model are calculated according to the task execution result and the corresponding perception task loss function. The perception task loss function can include a position loss function, a direction loss function, and a category loss function. Among them, the position loss function can adopt a SmoothL1 function, the direction loss function can adopt a Softmax loss function, and the category loss can adopt a FocalLoss function. For example, the position loss function can be as follows:

[0107] L loc =∑ b∈(x,y,z,w,h,l,θ) SmoothL1(Δb) (3)

[0108] wherein L loc represents the position loss value, b represents the position of the target, which is represented by 7 degrees of freedom (target center spatial coordinates (x, y, z), target width, length, and height (w, l, h), and heading angle θ), and Δb represents the position deviation of the detection box of the target b predicted by the deep learning network from the real box.

[0109] Thus, through training and optimization, the overall loss value of the deep learning model is calculated according to the position loss value, the direction loss value, the category loss value, the relationship loss value, and the preset calculation relationship. The overall loss value can be minimized, and minimizing the overall loss value is the optimization goal. For example, the preset calculation relationship can be as follows:

[0110]

[0111] wherein L total represents the overall loss value, N positive represents the total number of positive samples, the determination of the positive sample is related to the threshold optimization, i.e., if the predicted score of the sample multi-modal data is greater than the positive sample threshold, then the sample is a positive sample, L loc represents the position loss value, β loc represents the position weight corresponding to the position loss value, L dir represents the direction loss value, β dir represents the direction weight corresponding to the direction loss value, L cls represents the category loss value, β cls represents the category weight corresponding to the category loss value, L relation represents the relationship loss value, and β relation represents the relationship weight corresponding to the relationship loss value.

[0112] By designing a new comprehensive optimization target, the performance of the autonomous driving perception task and the degree of multi-modal fusion can be comprehensively represented. Through this new optimization target, the deep learning network can learn the deep features and associations of multi-modal data, thereby improving the performance of autonomous driving perception.

[0113] Exemplarily, when the preset perception task is a target detection task, the network structure diagram of the deep learning model can be as shown in Figure 4 The deep learning model includes a feature extraction layer, a deep mining layer, a perception task layer, and a training optimization layer. During training of the deep learning model, the feature extraction layer, the deep mining layer, the perception task layer, and the training optimization layer are involved in the calculation of various loss values throughout the process, and the model parameters are updated and optimized to obtain a high-quality feature extraction model. In actual application, optimization and loss value calculation are not required, and all loss structures can be removed, leaving only the feature extraction layer and the perception task layer. The feature extraction layer and the deep mining layer each include a point cloud branch and an image branch. The point cloud branch in the feature extraction layer is connected to the point cloud branch in the deep mining layer, and the image branch in the feature extraction layer is connected to the image branch in the deep mining layer.

[0114] Specifically, the image branch in the feature extraction layer includes two Encoders. The first Encoder performs semantic segmentation on the image to be processed, and the clustered results are up-sampled by the second Encoder to obtain F_img (second deep feature data). The point cloud branch in the feature extraction layer includes an Encoder, which increases the feature dimension of the point cloud data to be processed to a preset dimension, thereby extracting high-dimensional and deep feature data, i.e., F_pt (first deep feature data). The feature extraction layer also includes a Fusion layer, which aligns each pixel point in F_img with the points in F_pt according to a preset extrinsic matrix to obtain aligned F_img, and then fuses the aligned F_img with F_pt to obtain F_fuse (deep fusion feature) corresponding to the multi-modal data to be processed.

[0115] The point cloud branch and the image branch of the deep mining layer each include an Encoder and a Projection (multi-layer perception machine). The Encoder in the image branch performs deep mining on the sample image features, and the mined image features are input into the Projection to obtain second mining features. The Encoder in the point cloud branch performs deep mining on the sample point cloud features, and the mined point cloud features are input into the Projection to obtain first mining features. The Relation (relationship loss value) of the deep learning model is calculated by the deep mining layer based on the first mining features and the second mining features.

[0116] The perception task layer includes an Extract layer and a 3D TaskHead, and the F_fuse is input into the Extract layer for feature extraction, and then the 3D TaskHead performs a target detection task according to the extracted features to obtain a task execution result, and Loc (position loss), Dir (direction loss) and Cls (category loss) are calculated according to the task execution result.

[0117] The training optimization layer calculates a TotalLoss (total loss value) of the deep learning model according to the Loc, Dir, Cls and Relation. The parameters of the deep learning model are optimized through the TotalLoss until the loss value of the deep learning model no longer decreases or reaches a preset iteration number, and the pre-constructed feature extraction model is obtained.

[0118] In this embodiment, the deep learning model to be trained includes a feature extraction layer, a deep mining layer and a perception layer. The feature extraction layer extracts sample point cloud features corresponding to sample point cloud data and sample image features corresponding to sample image data, fuses the sample point cloud features and the sample image features, and obtains sample fusion features. The deep mining layer in the deep learning model to be trained mines the sample point cloud features and the sample image features in depth to obtain a deep mining result. The perception layer trains the deep learning model according to the sample fusion features and the deep mining result, and obtains the pre-constructed feature extraction model. The feature extraction model obtained by training can fully extract the internal relationship between multi-modal data, fuse the features of the same target, extract more effective scene information, and provide more accurate information for planning and decision-making of autonomous driving.

[0119] In another embodiment, a multi-modal data fusion method is provided, which includes the following steps:

[0120] Obtaining sample multi-modal data; the sample multi-modal data includes sample point cloud data and sample image data.

[0121] Inputting the sample multi-modal data into a deep learning model to be trained; the deep learning model includes a feature extraction layer, a deep mining layer and a perception layer; the perception layer includes a perception task layer and a training optimization layer.

[0122] The feature extraction layer extracts sample point cloud features corresponding to sample point cloud data and sample image features corresponding to sample image data, and fuses the sample point cloud features and the sample image features to obtain sample fusion features.

[0123] The deep mining layer mines the sample point cloud features and the sample image features in depth to obtain mining feature data.

[0124] determine a relationship loss value of the deep learning model according to the mined feature data.

[0125] perform a preset perception task according to the sample fused feature through the perception task layer to obtain a task execution result.

[0126] When the preset perception task is a target detection task, determine a position loss value, a direction loss value and a category loss value of the deep learning model according to the task execution result through the training optimization layer.

[0127] determine a total loss value of the deep learning model according to the position loss value, the direction loss value, the category loss value and the relationship loss value through the training optimization layer, train the deep learning model according to the total loss value, and obtain the pre-constructed feature extraction model.

[0128] obtain to-be-processed multi-modal data; the to-be-processed multi-modal data includes to-be-processed point cloud data and to-be-processed image data.

[0129] invoke the pre-constructed feature extraction model.

[0130] extract features of the to-be-processed point cloud data through the feature extraction model to obtain first deep feature data corresponding to the to-be-processed point cloud data; the first deep feature data includes deep point cloud features and associated features between the to-be-processed point cloud data and the to-be-processed image data.

[0131] perform semantic segmentation on the to-be-processed image data through the feature extraction model to obtain a semantic segmentation result, and cluster the semantic segmentation result to obtain a clustering result.

[0132] In this embodiment, the to-be-processed multi-modal data is fused through the pre-constructed feature extraction model. Since the feature extraction model is trained according to deep mining results of sample multi-modal data, the feature extraction model can deeply mine deep semantic features of multi-modal data and internal association relationships between multi-modal data, so that the deep fused features can be closer to the real world, the fusion accuracy of multi-modal data is improved, and more accurate information can be provided for planning and decision-making of autonomous driving.

[0133] In one embodiment, the method is compared with the current mainstream methods PointPainting and RoarNet for vehicle classification on the KITTI dataset, as shown in Table 1 below. Among them, the positive and negative sample thresholds are 0.6 and 0.45 respectively. The experiment takes the evaluation standard AP (Average Precision, detection accuracy) of the KITTI dataset as the index. For vehicle targets of different difficulties, the detection accuracy of the method is higher than that of the current mainstream two multi-modal fusion methods. Especially for vehicle targets in difficult situations, there is a certain detection difficulty. However, the detection accuracy obtained by the method in such targets is 0.24% higher than that of PointPainting. The comparison test shows that the method has certain superiority in improving the perception ability compared with the current mainstream two methods. The improvement of the perception ability also depends on the learning of the model on the data, so the experiment also verifies that the method has a deeper mining ability for multi-modal data features.

[0134] Table 1 Comparison test

[0135]

[0136] It should be understood that although each step in the flowchart involved in each embodiment as described above is displayed in sequence according to the arrow, these steps are not necessarily executed in sequence according to the arrow. Unless otherwise stated herein, the execution of these steps is not strictly limited in sequence, and these steps can be executed in other sequences. Moreover, at least part of the steps in the flowchart involved in each embodiment as described above can include multiple steps or multiple stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution sequence of these steps or stages is not necessarily sequential, but can be executed alternately or alternately with at least part of other steps or steps or stages in other steps.

[0137] Based on the same inventive concept, the embodiments of the present application also provide a multi-modal data fusion device for implementing the multi-modal data fusion method described above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme described in the above method, so the specific limitations in one or more multi-modal data fusion device embodiments provided below can refer to the limitations of the multi-modal data fusion method described above, which will not be repeated here.

[0138] In one embodiment, as shown in Figure 5 a multi-modal data fusion device is provided, comprising: a data acquisition module 502, a model calling module 504, a feature extraction module 506 and a feature fusion module 508, wherein:

[0139] The processing data acquisition module 502 is configured to acquire the to-be-processed multi-modal data, wherein the to-be-processed multi-modal data comprises to-be-processed point cloud data and to-be-processed image data.

[0140] The pre-constructed model calling module 504 is configured to call a pre-constructed feature extraction model.

[0141] The feature extraction module 506 is configured to perform feature extraction on the to-be-processed point cloud data and the to-be-processed image data respectively by using the feature extraction model, to obtain first deep feature data corresponding to the to-be-processed point cloud data and second deep feature data corresponding to the to-be-processed image data.

[0142] The feature fusion module 508 is configured to fuse the first deep feature data and the second deep feature data by using the feature extraction model, to obtain deep fusion features corresponding to the to-be-processed multi-modal data.

[0143] In an embodiment, the device described above further comprises:

[0144] The sample data acquisition module is configured to acquire sample multi-modal data, wherein the sample multi-modal data comprises sample point cloud data and sample image data.

[0145] The to-be-trained model calling module is configured to input the sample multi-modal data into a to-be-trained deep learning model, wherein the deep learning model comprises a feature extraction layer, a deep mining layer and a perception layer.

[0146] The feature processing module is configured to extract sample point cloud features corresponding to the sample point cloud data and sample image features corresponding to the sample image data by using the feature extraction layer, and fuse the sample point cloud features and the sample image features to obtain sample fusion features.

[0147] The feature mining module is configured to perform deep mining on the sample point cloud features and the sample image features by using the deep mining layer, to obtain a deep mining result.

[0148] The model optimization module is configured to train the deep learning model according to the sample fusion features and the deep mining result by using the perception layer, to obtain the pre-constructed feature extraction model.

[0149] In an embodiment, the deep mining result is a relationship loss value; the feature mining module is further configured to perform deep mining on the sample point cloud features and the sample image features by using the deep mining layer, to obtain mining feature data; and the relationship loss value is determined according to the mining feature data.

[0150] In an embodiment, the perception layer includes a perception task layer and a training optimization layer; the model optimization module is further configured to perform a preset perception task on the sample fused feature through the perception task layer to obtain a task execution result; determine an overall loss value of the deep learning model according to the task execution result and the relationship loss value through the training optimization layer, and train the deep learning model according to the overall loss value.

[0151] In an embodiment, the preset perception task is a target detection task; the model optimization module is further configured to determine a position loss value, a direction loss value and a category loss value of the deep learning model according to the task execution result through the training optimization layer; and determine the overall loss value of the deep learning model according to the position loss value, the direction loss value, the category loss value and the relationship loss value through the training optimization layer.

[0152] In an embodiment, the feature extraction module 506 is further configured to perform feature extraction on the to-be-processed point cloud data through the feature extraction model to obtain first deep feature data corresponding to the to-be-processed point cloud data; the first deep feature data includes deep point cloud features and associated features between the to-be-processed point cloud data and the to-be-processed image data; perform semantic segmentation on the to-be-processed image data through the feature extraction model to obtain a semantic segmentation result, and perform clustering on the semantic segmentation result to obtain a clustering result; perform up-sampling on the clustering result through the feature extraction model to obtain second deep feature data corresponding to the to-be-processed image data; the second deep feature data includes deep image features and associated features between the to-be-processed image data and the to-be-processed point cloud data.

[0153] Each module in the above multi-modal data fusion device can be realized by software, hardware and a combination thereof in whole or in part. Each module can be embedded in or independent of a processor in a computer device in hardware form, or can be stored in a memory in a computer device in software form, so as to be called and executed by a processor to perform operations corresponding to each module.

[0154] In an embodiment, a computer device is provided, which can be a server, and an internal structure diagram of the computer device can be as shown in Figure 6As shown in the figure. The computer device includes a processor, a memory, an input / output interface (I / O for short) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through the system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capability. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store the multi-modal data to be processed and the like. The input / output interface of the computer device is used to exchange information between the processor and external devices. The communication interface of the computer device is used to communicate with the terminal outside through network connection. The computer program is executed by the processor to realize a multi-modal data fusion method.

[0155] Those skilled in the art can understand that, Figure 6 The structure shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different component arrangement.

[0156] In one embodiment, a computer device is provided, including a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps in each of the method embodiments.

[0157] In one embodiment, a computer readable storage medium is provided, which stores a computer program, and the computer program is executed by a processor to implement the steps in each of the method embodiments.

[0158] In one embodiment, a computer program product is provided, including a computer program, and the computer program is executed by a processor to implement the steps in each of the method embodiments.

[0159] It should be noted that the user information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of related data need to comply with relevant laws, regulations and standards of relevant countries and regions.

[0160] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when the computer program is executed, the processes of the above-mentioned embodiments of the methods can be included. Any reference to memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (Read-Only Memory, ROM), magnetic tape, floppy disk, flash memory, optical storage, high-density embedded non-volatile memory, resistive memory (ReRAM), magnetoresistive random access memory (Magnetoresistive Random Access Memory, MRAM), ferroelectric memory (Ferroelectric Random Access Memory, FRAM), phase change memory (Phase Change Memory, PCM), graphene memory, etc. Volatile memory can include random access memory (Random Access Memory, RAM) or external cache memory, etc. As an illustration but not limitation, RAM can be in various forms, such as static random access memory (Static Random Access Memory, SRAM) or dynamic random access memory (Dynamic Random Access Memory, DRAM), etc. The database involved in the embodiments provided in the present application can include at least one of a relational database and a non-relational database. The non-relational database can include a distributed database based on a block chain, etc., without being limited thereto. The processor involved in the embodiments provided in the present application can be a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., without being limited thereto.

[0161] Any combination of the technical features of the above embodiments can be made. In order to make the description simple, all possible combinations of the technical features in the above embodiments are not described, however, as long as the combination of the technical features does not exist contradictory, it should be considered as the scope of the present application.

[0162] The above embodiments only express several implementation manners of the present application, and the description is more specific and detailed, but it should not be understood as a limitation on the scope of the patent of the present application. It should be pointed out that for ordinary skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are within the scope of protection of the present application. Therefore, the protection scope of the present application should be subject to the appended claims.

Claims

1. A multi-modal data fusion method, characterized in that, The method comprises: acquiring to-be-processed multi-modal data; the to-be-processed multi-modal data comprises to-be-processed point cloud data and to-be-processed image data; calling a pre-constructed feature extraction model; the feature extraction model comprises a feature extraction layer, the feature extraction layer comprises a point cloud branch and an image branch, the point cloud branch comprises an encoder, and the image branch comprises two encoders; performing feature extraction on the to-be-processed point cloud data and the to-be-processed image data respectively through the feature extraction model to obtain first deep layer feature data corresponding to the to-be-processed point cloud data and second deep layer feature data corresponding to the to-be-processed image data, comprising: increasing the feature dimension of the to-be-processed point cloud data to a preset dimension through the encoder in the point cloud branch to obtain first deep layer feature data, the first deep layer feature data comprising deep layer point cloud features and associated features between the to-be-processed image data; performing semantic segmentation on the to-be-processed image data through the first encoder in the image branch to obtain a semantic segmentation result, performing clustering on the semantic segmentation result to obtain a clustering result, and performing up-sampling on the clustering result through the second encoder in the image branch to obtain second deep layer feature data, each pixel point in the second deep layer feature data having the same preset dimension as the point cloud in the first deep layer feature data, the second deep layer feature data comprising deep layer image features and associated features between the to-be-processed point cloud data; fusing the first deep layer feature data and the second deep layer feature data through the feature extraction model to obtain deep layer fusion features corresponding to the to-be-processed multi-modal data.

2. The method of claim 1, wherein, Before the acquiring to-be-processed multi-modal data, the method further comprises: acquiring sample multi-modal data; the sample multi-modal data comprises sample point cloud data and sample image data; inputting the sample multi-modal data into a to-be-trained deep learning model; the deep learning model comprises a feature extraction layer, a deep mining layer and a perception layer; extracting sample point cloud features corresponding to the sample point cloud data and sample image features corresponding to the sample image data through the feature extraction layer, and fusing the sample point cloud features and the sample image features to obtain sample fusion features; performing deep mining on the sample point cloud features and the sample image features through the deep mining layer to obtain a deep mining result; training the deep learning model according to the sample fusion features and the deep mining result through the perception layer to obtain the pre-constructed feature extraction model.

3. The method of claim 2, wherein, The deep mining result is a relationship loss value; the deep mining on the sample point cloud features and the sample image features through the deep mining layer to obtain a deep mining result comprises: performing deep mining on the sample point cloud features and the sample image features through the deep mining layer to obtain mining feature data; determining the relationship loss value according to the mining feature data.

4. The method of claim 3, wherein, The perception layer comprises a perception task layer and a training optimization layer; the training of the deep learning model according to the sample fusion feature and the deep mining result through the perception layer comprises: performing a preset perception task according to the sample fusion feature through the perception task layer to obtain a task execution result; determining a total loss value of the deep learning model according to the task execution result and the relationship loss value through the training optimization layer, and training the deep learning model according to the total loss value.

5. The method of claim 4, wherein, The preset perception task is a target detection task; the determination of the total loss value of the deep learning model according to the task execution result and the relationship loss value through the training optimization layer comprises: determining a position loss value, a direction loss value and a category loss value of the deep learning model according to the task execution result through the training optimization layer; determining the total loss value of the deep learning model according to the position loss value, the direction loss value, the category loss value and the relationship loss value through the training optimization layer.

6. A multi-modal data fusion apparatus, characterized by, The device comprises: a data acquisition module configured to acquire to-be-processed multi-modal data; the to-be-processed multi-modal data comprises to-be-processed point cloud data and to-be-processed image data; a model calling module configured to call a pre-constructed feature extraction model; the feature extraction model comprises a feature extraction layer, the feature extraction layer comprises a point cloud branch and an image branch, the point cloud branch comprises one encoder, and the image branch comprises two encoders; a feature extraction module configured to perform feature extraction on the to-be-processed point cloud data and the to-be-processed image data respectively through the feature extraction model, to obtain first deep feature data corresponding to the to-be-processed point cloud data and second deep feature data corresponding to the to-be-processed image data; a feature fusion module configured to fuse the first deep feature data and the second deep feature data through the feature extraction model, to obtain deep fusion features corresponding to the to-be-processed multi-modal data; the feature extraction module is further configured to increase the feature dimension of the to-be-processed point cloud data to a preset dimension through the encoder in the point cloud branch, to obtain first deep feature data, the first deep feature data comprising deep point cloud features and associated features between the to-be-processed image data; perform semantic segmentation on the to-be-processed image data through the first encoder in the image branch, to obtain a semantic segmentation result, perform clustering on the semantic segmentation result, to obtain a clustering result, and perform up-sampling on the clustering result through the second encoder in the image branch, to obtain second deep feature data, each pixel point in the second deep feature data having the same preset dimension as a point cloud in the first deep feature data, the second deep feature data comprising deep image features and associated features between the to-be-processed point cloud data.

7. The apparatus of claim 6, wherein, The device further comprises: a sample data acquisition module configured to acquire sample multi-modal data; the sample multi-modal data comprises sample point cloud data and sample image data; The to-be-trained model calling module is configured to input the sample multi-modal data into a deep learning model to be trained, and the deep learning model comprises a feature extraction layer, a deep mining layer, and a perception layer; The feature processing module is configured to extract sample point cloud features corresponding to the sample point cloud data and sample image features corresponding to the sample image data through the feature extraction layer, fuse the sample point cloud features and the sample image features, and obtain sample fusion features; The feature mining module is configured to perform deep mining on the sample point cloud features and the sample image features through the deep mining layer to obtain a deep mining result. The model optimization module is configured to train the deep learning model according to the sample fusion features and the deep mining result through the perception layer to obtain the pre-constructed feature extraction model.

8. The apparatus of claim 7, wherein, The deep mining result is a relationship loss value; the feature mining module is further configured to perform deep mining on the sample point cloud features and the sample image features through the deep mining layer to obtain mining feature data; and the relationship loss value is determined according to the mining feature data. 9.A computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the computer device is configured to perform the method according to any one of claims 1-8 when the computer program is executed by the processor. The processor executes the computer program to implement the steps of the method in any one of claims 1 to 5.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method in any one of claims 1 to 5.

Images