Data processing method and apparatus based on multi-modal fusion

Abstract

The present application belongs to the technical field of data processing. Provided are a data processing method and apparatus based on multi-modal fusion. The method comprises: acquiring one-dimensional data and image data; on the basis of the size of the image data, converting the one-dimensional data into two-dimensional data; performing zero padding processing on vacant positions in the two-dimensional data; performing stacking processing on the two-dimensional data which has been subjected to the zero padding processing, and the image data, so as to obtain a multi-layer stacked input feature map; by means of a neural network, performing fusion processing on the multi-layer stacked input feature map, so as to obtain a fused feature map; and on the basis of the fused feature map, performing data processing. In the present invention, one-dimensional data is converted into two-dimensional data which is of the same size as image data, and the two-dimensional data is stacked with the image data, such that data formats of different modalities can be unified, and the data can be processed within the same feature space, thereby greatly simplifying an alignment process between heterogeneous data.

Description

A data processing method and apparatus based on multimodal fusion

[0001] Cross-referencing related documents

[0002] This application claims priority to Chinese Patent Application No. 2024114374668, filed on October 15, 2024, entitled "A Data Processing Method and Apparatus Based on Multimodal Fusion", the entire contents of which are incorporated herein by reference. Technical Field

[0003] The embodiments of this application belong to the field of data processing technology, specifically relating to a data processing method and apparatus based on multimodal fusion. Background Technology

[0004] With the rapid development of computer technology, especially the widespread application of deep learning methods in the field of computer vision, multimodal fusion technology has become one of the important branches of current cutting-edge technology. Multimodal fusion technology mainly enhances the machine's information processing capabilities by integrating information from different data sources, such as images, text, audio, and sensor data.

[0005] Multimodal technology has a wide range of applications, covering fields such as intelligent transportation, security monitoring, and intelligent human-computer interaction. In these applications, multimodal scene classification technology can achieve accurate classification and understanding of scenes by comprehensively analyzing data from multiple modalities such as images, videos, and text.

[0006] Multimodal image classification further expands this scope, going beyond a single image type to combine various image modalities such as RGB, infrared, and depth images to obtain more detailed information about the observed object. Furthermore, multimodal target detection technology utilizes this comprehensive information to detect and locate specific targets in images or videos, such as in applications like autonomous driving and night vision surveillance, effectively improving recognition accuracy and system response speed.

[0007] However, current multimodal data fusion technologies suffer from low processing efficiency for heterogeneous data, difficulty in alignment during heterogeneous data conversion, easy loss of information, difficulty in effective feature extraction, and low data fusion quality. Technical solutions

[0008] To address the technical problems of low processing efficiency of heterogeneous data, difficulty in alignment during heterogeneous data conversion, easy loss of information, difficulty in effective feature extraction, and low data fusion quality in existing technologies, this invention provides a data processing method and apparatus based on multimodal fusion.

[0009] In a first aspect, the present invention provides a data processing method based on multimodal fusion, comprising:

[0010] Acquire one-dimensional data and image data;

[0011] Based on the dimensions of the image data, the one-dimensional data is converted into two-dimensional data;

[0012] Zero-padding is performed on the empty positions in the two-dimensional data;

[0013] The zero-padding two-dimensional data is stacked with the image data to obtain a multi-layer stacked input feature map;

[0014] The input feature maps, which are stacked in multiple layers, are fused using a neural network to obtain a fused feature map.

[0015] Data processing is performed based on the fused feature map.

[0016] Secondly, the present invention provides a data processing apparatus based on multimodal fusion, comprising:

[0017] The acquisition module is used to acquire one-dimensional data and image data;

[0018] A conversion module is used to convert the one-dimensional data into two-dimensional data according to the size of the image data;

[0019] The zero-padding module is used to fill empty positions in the two-dimensional data with zeros;

[0020] The stacking module is used to stack the zero-padded 2D data with the image data to obtain a multi-layer stacked input feature map;

[0021] The fusion module is used to fuse the multi-layered stacked input feature maps through a neural network to obtain a fused feature map;

[0022] The processing module is used to perform data processing based on the fused feature map.

[0023] Compared with the prior art, the present invention has at least the following beneficial effects:

[0024] In this invention, by converting one-dimensional data into two-dimensional data of the same size as image data and stacking it with image data, the data formats of different modalities can be unified, enabling them to be processed in the same feature space. This greatly simplifies the alignment process between heterogeneous data and ensures data integrity. By processing multi-layer stacked input feature maps through neural networks and utilizing the powerful automatic feature extraction capabilities of deep learning models, features of different modalities can be efficiently fused, improving the quality of data fusion. Attached Figure Description

[0025] Figure 1 is a flowchart illustrating a data processing method based on multimodal fusion provided by the present invention.

[0026] Figure 2 is a schematic diagram of the overall network structure of a data processing method based on multimodal fusion provided by the present invention.

[0027] Figure 3 is a schematic diagram of a data conversion structure provided by the present invention.

[0028] Figure 4 is a schematic diagram of a feature filtering structure provided by the present invention.

[0029] Figure 5 is a schematic diagram of the structure of a data processing device based on multimodal fusion provided by the present invention.

[0030] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and their descriptions, serving to explain this application and not constituting an undue limitation thereof. Some specific embodiments of this application will be described in detail below with reference to the accompanying drawings in an exemplary and non-limiting manner. The best embodiment of the present invention

[0031] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are merely some, not all, of the embodiments of the present application. All other embodiments obtained by those skilled in the art based on the embodiments of the present application without creative effort should fall within the scope of protection of the present application.

[0032] Firstly, referring to Figure 1 of the specification, a flowchart illustrating a data processing method based on multimodal fusion provided by an embodiment of the present invention is shown. Referring to Figure 2 of the specification, a schematic diagram of the overall network structure of a data processing method based on multimodal fusion provided by an embodiment of the present invention is shown.

[0033] This invention provides a data processing method based on multimodal fusion, comprising:

[0034] S1: Obtain one-dimensional data and image data.

[0035] Optionally, one-dimensional data can specifically be sound data, text data, current data, etc.

[0036] S2: Convert one-dimensional data into two-dimensional data based on the size of the image data.

[0037] Referring to Figure 3 in the specification, a schematic diagram of a data conversion structure provided by an embodiment of the present invention is shown.

[0038] In one possible implementation, S2 specifically includes sub-steps S201 to S203:

[0039] S201: Determine multiple panes of the same size based on the dimensions of the image data.

[0040] Optionally, when the image data size is H×W, it can be processed according to... Identify 25 panes of the same size.

[0041] S202: Divide one-dimensional data into multiple one-dimensional sub-data according to a preset size.

[0042] Optionally, those skilled in the art can set specific values ​​for the preset size according to actual conditions; this invention does not impose any limitations.

[0043] S203: Fill the individual panes with one-dimensional sub-data to form two-dimensional data.

[0044] Optionally, one-dimensional sub-data is filled into each pane in a top-to-bottom, left-to-right order, and within each pane, each data point is filled into each pixel in the same top-to-bottom, left-to-right order.

[0045] In this invention, panes are used to divide one-dimensional data into blocks, ensuring that adjacent or related one-dimensional data are filled into the same pane. This method preserves the local correlations in one-dimensional data, allowing the data to maintain a certain continuity and relevance in the two-dimensional structure, which helps the model better capture local features and patterns. Simultaneously, pane filling makes the distribution of one-dimensional data in two-dimensional space more structured and ordered. Within each pane, data is filled in a specific order; this structured representation enhances the spatial interpretability of the data, enabling the model to more intuitively understand and utilize the distribution characteristics of this data.

[0046] S3: Fill empty positions in the two-dimensional data with zeros.

[0047] In this invention, zero-padding ensures that each position has a value, thus avoiding information loss due to empty positions and ensuring the complete representation of data in two-dimensional space. Furthermore, during neural network training, empty positions may lead to the generation of invalid gradients, affecting the convergence of the model. Zero-padding avoids this problem and ensures the validity of gradient calculation.

[0048] S4: Stack the zero-padding 2D data with the image data to obtain a multi-layered stacked input feature map.

[0049] S5: The input feature maps of multiple stacks are fused using a neural network to obtain a fused feature map.

[0050] It should be noted that neural networks possess powerful nonlinear feature extraction capabilities. By processing multi-layered stacked input feature maps, neural networks can automatically extract and fuse higher-level, more abstract features, fully leveraging the advantages of multimodal data to automatically extract and fuse valuable features, enhancing the model's expressive power, robustness, and generalization ability, while simultaneously improving decision-making accuracy and model flexibility.

[0051] In one possible implementation, S5 specifically includes sub-steps S501 to S503:

[0052] S501: Perform feature filtering on the multi-layer stacked input feature maps to obtain the filtered feature maps.

[0053] Referring to Figure 4 in the specification, a schematic diagram of a feature filtering structure provided by an embodiment of the present invention is shown.

[0054] Optionally, S501 specifically includes:

[0055] S5011: The first convolutional unit extracts and activates features from the input feature map to obtain the activated feature map.

[0056] Optionally, the activation feature map is specifically as follows:

[0057]

[0058] Where represents the activation feature map, ReLU represents the ReLU activation function, Conv1 represents the first convolutional unit, and X represents the input feature map.

[0059] Optionally, the first convolutional unit employs a 3×3 convolution and the ReLU activation function.

[0060] Among them, the ReLU (Rectified Linear Unit) activation function is one of the most commonly used activation functions in deep learning. Its main function is to introduce non-linearity, enabling the model to learn more complex features. Linear transformations are inherently unable to handle non-linear problems, while ReLU introduces non-linearity by setting the less-than-zero parts of the input values ​​to zero, thus allowing the model to fit and handle complex non-linear data distributions.

[0061] It should be noted that feature extraction and activation are performed through the first convolutional unit. The combination of a 3×3 convolutional kernel and the ReLU activation function can effectively enhance the feature extraction capability, introduce nonlinearity, improve computational efficiency, and increase the sparsity and robustness of the model.

[0062] S5012: Copy the activation feature map into two copies, namely the first activation feature map and the second activation feature map.

[0063] S5013: The first activated feature map is symmetrically flipped along the central axis to obtain a symmetrical feature map.

[0064] Optionally, the symmetric feature map is specifically:

[0065]

[0066] Among them, X f _ represents a symmetrical feature map, and Flip represents a symmetrical flip.

[0067] It should be noted that by symmetrically flipping the first activation feature map, the model can be helped to identify symmetrical structures in the image, improving the sensitivity and robustness of the feature map to symmetrical shapes. Symmetry is an important geometric property in many vision tasks, and this operation enables the model to better understand and utilize this property.

[0068] S5014: The second activation feature map is convolved by the second convolution unit to obtain the deformed feature map.

[0069] Optionally, the second convolutional unit may employ a 7×7 convolution connected in series, a 5×5 deformable dilated convolution with a dilation rate of 1, or a 3×3 convolution.

[0070] Optionally, the deformation feature map is specifically as follows:

[0071] Among them, X d The denots represent deformable feature maps, Conv2 represents 3×3 convolution, DConv represents deformable dilated convolution, and DDConv represents 7×7 convolution.

[0072] It's worth noting that using deformable dilated convolutions can enhance the model's adaptability to deformable objects. Dilated convolutions (with a dilation rate) can expand the receptive field of the convolution, while deformable convolutions can dynamically adjust the position of the convolution kernel, thus better capturing features with significant deformation or scale changes. This makes the model more accurate and flexible when dealing with objects with complex geometries.

[0073] Furthermore, by using multiple convolutional layers (such as 3×3 convolution, 7×7 convolution, and deformable dilated convolution), features can be extracted from different scales and receptive fields. This diverse range of convolutional operations can capture richer and more complex features in images, improving the model's understanding of details and global information.

[0074] S5015: Multiply the deformed feature map with the symmetric feature map and perform convolution processing through the third convolution unit.

[0075] Optionally, the third convolutional unit uses a 3×3 convolution.

[0076] S5016: Add the feature map processed by the third convolution unit to the input feature map to obtain the filtered feature map.

[0077] Optionally, the feature map selection process specifically involves:

[0078]

[0079] Where Y represents the filtered feature map, Conv3 represents 3×3 convolution, and Conv4 represents 3×3 convolution.

[0080] It should be noted that adding the feature map processed by the third convolutional unit to the original input feature map enables feature selection and optimization. This residual connection method can incorporate newly extracted feature information while preserving the original features, thereby reducing information loss and enhancing the overall expressive power of the feature map. This operation can avoid the gradient vanishing problem, further improving the model's stability and training efficiency.

[0081] S502: Perform feature enhancement on the selected feature map to obtain an enhanced feature map.

[0082] Optionally, S502 specifically includes:

[0083] S5021: The selected feature map is convolved by the fourth convolutional unit and the channel attention unit to obtain a large receptive field feature map.

[0084] Optionally, the fourth convolutional unit uses a 9×9 convolution.

[0085] Optionally, the large receptive field feature map is specifically as follows:

[0086]

[0087] Where represents the large receptive field feature map, SE represents the channel attention unit, represents the 9×9 convolution, and Y represents the filtered feature map.

[0088] It should be noted that large kernel convolution can obtain a larger receptive field, can consider the relationships between a wider range of features, and capture more semantic information.

[0089] Furthermore, the channel attention mechanism (SE module) assigns weights to channels of different modalities, enabling the model to adaptively focus on more important feature channels. This weighting method effectively enhances the model's attention to key features and reduces interference from irrelevant or secondary information, thereby improving the quality of feature extraction. For multimodal fusion, channel attention can also ensure that useful information from each modality is fully utilized, enhancing the fusion effect.

[0090] S5022: Deforms large receptive field feature maps using a linear transformation unit.

[0091] Optionally, the linear transformation unit includes a first linear layer, a GULE activation function, and a second linear layer connected in series.

[0092] It should be noted that, through concatenated linear layers and the GULE activation function, the linear transformation unit can effectively connect the relationships between various modalities, enabling the interaction of multimodal semantic information. The linear layers perform transformations and mappings in the multimodal feature space, allowing features from different modalities to be effectively fused and interacted within the same feature space. This interaction helps the model better understand and utilize the comprehensive information from different modalities, thereby improving the final decision-making ability.

[0093] S5023: Add the large receptive field feature map after deformation processing to the selected feature map to obtain the enhanced feature map.

[0094] Optionally, the enhanced feature map is specifically:

[0095]

[0096] Where Z represents the enhanced feature map, linear1 represents the first linear layer, GULE represents the GULE activation function, and linear2 represents the second linear layer.

[0097] It should be noted that the deformed large receptive field feature map is added to the selected feature map to form a residual connection. This operation not only preserves the original feature information but also integrates the features enhanced by large kernel convolution and channel attention. This multi-level fusion can improve the expressive power of the feature map, ensuring that while capturing more semantic information, the integrity and detail of the original features are maintained.

[0098] S503: Decode the enhanced feature map to obtain the fused feature map.

[0099] Optionally, S503 specifically includes:

[0100] S5031: Normalize the enhanced feature map.

[0101] It should be noted that normalization can reduce the scale difference of input data, prevent the model's training effect from being unstable due to data of different scales, and help to learn features better.

[0102] S5032: The normalized enhanced feature map is convolved through the fifth convolutional unit.

[0103] Optionally, the fifth convolutional unit employs depthwise separable convolution.

[0104] Optionally, depthwise separable convolution specifically refers to:

[0105]

[0106] Wherein, represents the feature map after depthwise separable convolution, DWC represents depthwise separable convolution, GN represents normalization, and Z represents enhanced feature map.

[0107] It's important to note that most commonly used attention decoders consist of multiple linear layers. Compared to conventional attention decoders and ordinary convolutions, depthwise separable convolutions decompose the convolution operation into two steps: depthwise convolution and pointwise convolution. This significantly reduces computational cost while preserving or even enhancing feature extraction capabilities. Due to this increased computational efficiency, the model can handle more complex tasks with the same computing resources, and its representational power is also enhanced, resulting in richer and more effective feature extraction.

[0108] S5033: Perform channel scaling on the enhanced feature map after convolution.

[0109] It should be noted that by performing channel scaling on the enhanced feature map after convolution, the weight of each channel can be dynamically adjusted, allowing the model to better focus on the importance of different feature channels.

[0110] S5034: Add the enhanced feature map after channel scaling to the enhanced feature map to obtain the intermediate feature map.

[0111] It should be noted that adding the original enhanced feature map to form a residual connection not only preserves the original feature information but also integrates the new features after convolution, further enhancing the expressiveness and complexity of the features. This multi-level fusion strategy can improve the model's ability to integrate data from different modalities, ensuring that important information is effectively utilized.

[0112] S5035: The intermediate feature map is decoded by group normalization and channel multilayer perceptron to obtain the fused feature map.

[0113] Optionally, the fused feature maps are specifically as follows:

[0114] Where W represents the fused feature map and CMLP represents the multilayer perceptron.

[0115] It should be noted that processing the intermediate feature maps using a channel-multilayer perceptron allows for complex nonlinear transformations along the channel dimension. This operation further enhances the model's feature decoding capabilities, enabling the final output fused feature map to better capture the interaction relationships and semantic structures of multimodal information, thus providing stronger expressiveness.

[0116] S6: Perform data processing based on the fused feature map.

[0117] In one possible implementation, S6 specifically includes: performing target detection based on the fused feature map.

[0118] It's important to note that object detection is a crucial task in computer vision. It requires not only identifying the categories of objects appearing in an image (such as cats, cars, and people) but also determining the locations of these objects within the image. The final output of object detection typically includes the object's category label and its corresponding bounding box, i.e., the coordinates of the region where the object is located.

[0119] Alternatively, the feature maps obtained from the multi-layer fusion can be input separately into the object detection network for object detection:

[0120]

[0121] Where CBS represents a combination of convolution, batch normalization, and SILU activation function, C2F is a multi-branch structure in YOLOv8 used for feature extraction, and Upsample represents the upsampling operation. P5, P4, and P3 are the inputs to the neck network, and T5, T4, and T3 are the outputs of the neck network.

[0122] Furthermore, the feature map obtained from the neck network is input into the detection head to obtain the prediction result:

[0123]

[0124] The `Split` table represents the splitting operation, with one part of the split result used for classification prediction and the other for bounding box prediction. The superscripts `B` and `C` in `T5`, `T4`, and `T3` ​​represent the prediction of the bounding box and the class, respectively. The input feature maps `T5`, `T4`, and `T3` ​​are used to predict small target `S`, medium target `M`, and large target `L`, respectively. `Bbox` represents the predicted bounding box, and `Cls` represents the predicted class.

[0125] In one possible implementation, the loss function for training the neural network in object detection is specifically:

[0126]

[0127] Among them, L a L represents the loss function in the object detection process. C Let L represent the cross-entropy loss, λ1 represent the weights of the cross-entropy loss, and L represent the weights of the cross-entropy loss. CIoU Let L represent the target box loss, λ2 represent the weights of the target box loss, and L represent the weights of the target box loss. conf λ represents the confidence loss, and λ3 represents the weight of the confidence loss.

[0128] Those skilled in the art can set the weights λ1 of the cross-entropy loss, λ2 of the target box loss, and λ3 of the confidence loss according to the actual situation; this invention does not impose any limitations.

[0129] It should be noted that combining these different loss functions and using a weighted loss function for training can enable the model to learn different object detection tasks more comprehensively, thereby improving the final detection accuracy and generalization ability.

[0130] Optionally, the cross-entropy loss is as follows:

[0131] Where i represents each sample, p i The model predicts that the i-th sample is 1, and the closer it is to 1, the more likely it is to belong to that category. i Indicate whether the actual sample belongs to this category; if so, y i If it is 1, then y i If 0, then N represents the total number of predictions.

[0132] Optionally, the target box loss is specifically as follows:

[0133] Where IoU represents the intersection-union ratio of the predicted bounding box A and the ground truth bounding box B, and b represents the center point of the predicted bounding box. gt w represents the center point of the actual bounding box, and w represents the width of the predicted bounding box. gt h represents the width of the actual bounding box, and h represents the height of the predicted bounding box. gt ρ represents the height of the actual bounding box. 2 represents the Euclidean distance between the two boxes, and c represents the diagonal distance of the smallest rectangle containing both boxes.

[0134] Optionally, the confidence loss is as follows:

[0135]

[0136] Where obj represents a sample that actually contains a target, i corresponds to each sample containing a target, no represents a sample that does not actually contain a target, and j corresponds to each sample that does not contain a target. obj The weight used to adjust the presence of the target is generally set to 4. 1 represents the value corresponding to the sample that actually has the target; 0 represents the predicted probability corresponding to the sample that actually has the target; 1 represents the value corresponding to the sample that actually does not have the target; 0 represents the predicted probability corresponding to the sample that actually does not have the target.

[0137] In one possible implementation, S6 specifically includes: performing classification processing based on the fused feature map.

[0138] In one possible implementation, the loss function for training the neural network in the classification process is specifically:

[0139]

[0140] Among them, L b Let y represent the loss function in the classification process. c p represents the actual label value of the c-th category. c This represents the predicted classification value for the c-th category, where C represents the total number of categories.

[0141] It's worth noting that cross-entropy loss is effective for multi-class classification problems, assigning appropriate loss weights to each class even when there are many classes. It helps the model optimize for the correct class by measuring the difference between the predicted probability distribution and the true label distribution.

[0142] Compared with the prior art, the present invention has at least the following beneficial effects:

[0143] In this invention, by converting one-dimensional data into two-dimensional data of the same size as image data and stacking it with image data, the data formats of different modalities can be unified, enabling them to be processed in the same feature space. This greatly simplifies the alignment process between heterogeneous data and ensures data integrity. By processing multi-layer stacked input feature maps through neural networks and utilizing the powerful automatic feature extraction capabilities of deep learning models, features of different modalities can be efficiently fused, improving the quality of data fusion.

[0144] Secondly, referring to Figure 5 in the specification, a schematic diagram of the structure of a data processing device based on multimodal fusion provided by an embodiment of the present invention is shown.

[0145] This invention provides a data processing device 20 based on multimodal fusion, comprising:

[0146] The acquisition module 201 is used to acquire one-dimensional data and image data.

[0147] The conversion module 202 is used to convert one-dimensional data into two-dimensional data according to the size of the image data.

[0148] The zero-padding module 203 is used to fill empty positions in two-dimensional data with zeros.

[0149] The stacking module 204 is used to stack the zero-padding two-dimensional data with the image data to obtain a multi-layer stacked input feature map.

[0150] The fusion module 205 is used to fuse the multi-layer stacked input feature maps through a neural network to obtain a fused feature map.

[0151] The processing module 206 is used to perform data processing based on the fused feature map.

[0152] In one possible implementation, the conversion module 202 is specifically used for:

[0153] Based on the dimensions of the image data, determine multiple panes of the same size.

[0154] One-dimensional data is divided into multiple one-dimensional sub-data according to a preset size.

[0155] One-dimensional sub-data is filled into each pane to form two-dimensional data.

[0156] In one possible implementation, the fusion module 205 is specifically used for:

[0157] Feature filtering is performed on the multi-layered stacked input feature maps to obtain filtered feature maps.

[0158] Feature enhancement is performed on the selected feature map to obtain the enhanced feature map.

[0159] The enhanced feature map is decoded to obtain the fused feature map.

[0160] In one possible implementation, the fusion module 205 is specifically used for:

[0161] The first convolutional unit extracts and activates features from the input feature map to obtain the activated feature map.

[0162] The activation feature map is copied into two copies, namely the first activation feature map and the second activation feature map.

[0163] The first activation feature map is symmetrically flipped along the central axis to obtain a symmetrical feature map.

[0164] The second convolutional unit performs convolution processing on the second activation feature map to obtain the deformed feature map.

[0165] The deformed feature map is multiplied by the symmetric feature map, and then convolution is performed through the third convolution unit.

[0166] The feature map processed by the third convolution unit is added to the input feature map to obtain the filtered feature map.

[0167] In one possible implementation, the fusion module 205 is specifically used for:

[0168] The selected feature map is convolved using the fourth convolutional unit and the channel attention unit to obtain a large receptive field feature map.

[0169] The feature map with a large receptive field is deformed by a linear transformation unit.

[0170] The enhanced feature map is obtained by adding the large receptive field feature map after deformation processing to the selected feature map.

[0171] In one possible implementation, the fusion module 205 is specifically used for:

[0172] The enhanced feature map is normalized.

[0173] The normalized enhanced feature map is convolved using the fifth convolutional unit.

[0174] Channel scaling is applied to the enhanced feature map after convolution.

[0175] The enhanced feature map after channel scaling is added to the enhanced feature map to obtain the intermediate feature map.

[0176] By using group normalization and channel multilayer perceptron, the intermediate feature map is decoded to obtain the fused feature map.

[0177] In one possible implementation, the processing module 206 is specifically used for:

[0178] Target detection is performed based on the fused feature map.

[0179] In one possible implementation, the loss function for training the neural network in object detection is specifically:

[0180]

[0181] Among them, L a L represents the loss function in the object detection process. C Let L represent the cross-entropy loss, λ1 represent the weights of the cross-entropy loss, and L represent the weights of the cross-entropy loss. CIoU Let L represent the target box loss, λ2 represent the weights of the target box loss, and L represent the weights of the target box loss. confλ represents the confidence loss, and λ3 represents the weight of the confidence loss.

[0182] In one possible implementation, the processing module 206 is specifically used for:

[0183] Classification is performed based on the fused feature map.

[0184] In one possible implementation, the loss function for training the neural network in the classification process is specifically:

[0185]

[0186] Among them, L b Let y represent the loss function in the classification process. c p represents the actual label value of the c-th category. c This represents the predicted classification value for the c-th category, where C represents the total number of categories.

[0187] The data processing device 20 based on multimodal fusion provided by the present invention can implement the various processes implemented in the method embodiment of the first aspect above. To avoid repetition, it will not be described again here.

[0188] The virtual device provided by this invention can be a device, or a component, integrated circuit, or chip in a terminal.

[0189] Compared with the prior art, the present invention has at least the following beneficial effects:

[0190] In this invention, by converting one-dimensional data into two-dimensional data of the same size as image data and stacking it with image data, the data formats of different modalities can be unified, enabling them to be processed in the same feature space. This greatly simplifies the alignment process between heterogeneous data and ensures data integrity. By processing multi-layer stacked input feature maps through neural networks and utilizing the powerful automatic feature extraction capabilities of deep learning models, features of different modalities can be efficiently fused, improving the quality of data fusion.

[0191] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. A data processing method based on multi-modal fusion, characterized in that, include: Acquire one-dimensional data and image data; Based on the dimensions of the image data, the one-dimensional data is converted into two-dimensional data; Zero-padding is performed on the empty positions in the two-dimensional data; The zero-padding two-dimensional data is stacked with the image data to obtain a multi-layer stacked input feature map; The input feature maps, which are stacked in multiple layers, are fused using a neural network to obtain a fused feature map. Data processing is performed based on the fused feature map.

2. The data processing method based on multi-modal fusion according to claim 1, characterized in that, The step of converting the one-dimensional data into two-dimensional data according to the size of the image data specifically includes: Based on the dimensions of the image data, determine multiple panes of the same size; The one-dimensional data is divided into multiple one-dimensional sub-data according to a preset size; The one-dimensional sub-data is filled into each of the panes to form the two-dimensional data. 3.The data processing method based on multi-modal fusion according to claim 1, characterized in that, The process of fusing the multi-layered stacked input feature maps using a neural network to obtain a fused feature map specifically includes: The input feature maps, which are stacked in multiple layers, are subjected to feature filtering to obtain filtered feature maps; The selected feature map is enhanced to obtain an enhanced feature map; The enhanced feature map is decoded to obtain the fused feature map.

4. The data processing method based on multi-modal fusion according to claim 3, characterized in that, The step of performing feature filtering on the multi-layered stacked input feature maps to obtain filtered feature maps specifically includes: The input feature map is processed by the first convolutional unit to extract and activate features, resulting in an activated feature map. The activation feature map is copied into two copies, namely a first activation feature map and a second activation feature map; The first activation feature map is symmetrically flipped along the central axis to obtain a symmetrical feature map; The second activation feature map is convolved by the second convolution unit to obtain the deformed feature map; The deformed feature map is multiplied by the symmetrical feature map, and then convolution is performed through the third convolution unit; The feature map processed by the third convolution unit is added to the input feature map to obtain the filtered feature map.

5. The data processing method based on multi-modal fusion according to claim 3, characterized in that, The step of enhancing the selected feature map to obtain an enhanced feature map specifically includes: The selected feature map is convolved using the fourth convolutional unit and the channel attention unit to obtain a large receptive field feature map. The large receptive field feature map is deformed by a linear transformation unit; The enhanced feature map is obtained by adding the large receptive field feature map after deformation processing to the selected feature map.

6. The data processing method based on multi-modal fusion according to claim 3, characterized in that, Decoding the enhanced feature map to obtain the fused feature map specifically includes: The enhanced feature map is normalized. The normalized enhanced feature map is convolved through the fifth convolutional unit. Channel scaling is applied to the enhanced feature map after convolution. The enhanced feature map after channel scaling is added to the enhanced feature map to obtain the intermediate feature map; The intermediate feature map is decoded by group normalization and channel multilayer perceptron to obtain the fused feature map.

7. The data processing method based on multi-modal fusion according to claim 1, characterized in that, The data processing based on the fused feature map specifically includes: Target detection is performed based on the fused feature map.

8. The data processing method based on multi-modal fusion according to claim 7, characterized in that, The loss function used to train the neural network in the object detection is specifically as follows: wherein, L a represents a loss function in the target detection process, L C represents a cross-entropy loss, λ1 represents a weight of the cross-entropy loss, L CIoU represents a target frame loss, λ2 represents a weight of the target frame loss, L conf represents a confidence loss, and λ3 represents a weight of the confidence loss. 9.The data processing method based on multi-modal fusion according to claim 1, characterized in that, The data processing based on the fused feature map specifically includes: Based on the fused feature map, classification processing is performed; The loss function used to train the neural network in the classification process is specifically as follows: where L b represents a loss function in the classification process, y c represents an actual label value of the cth class, p c represents a classification prediction value of the cth class, and C represents the total number of classes.

10. A data processing device based on multi-modal fusion, characterized in that, include: The acquisition module is used to acquire one-dimensional data and image data; A conversion module is used to convert the one-dimensional data into two-dimensional data according to the size of the image data; The zero-padding module is used to fill empty positions in the two-dimensional data with zeros; The stacking module is used to stack the zero-padded 2D data with the image data to obtain a multi-layer stacked input feature map; The fusion module is used to fuse the multi-layered stacked input feature maps through a neural network to obtain a fused feature map; The processing module is used to perform data processing based on the fused feature map.

Images