A highly fault-tolerant multimodal data fusion method based on paired feature synthesis network

Abstract

The present invention discloses a highly fault-tolerant multimodal data fusion method based on a paired feature synthesis network. The method specifically comprises: dynamically assigning weights to the input raw data and preprocessing it to obtain initial refined feature pairs; performing scale transformation on the initial refined feature pairs to obtain input features; and classifying and predicting the input features to obtain multimodal data fusion classification results. By dynamically assigning weights to the input raw data and preprocessing it, the present invention ensures the system's fault tolerance and reliability when a single data input fails. Furthermore, through the scale transformation operation, spatial information can be highlighted, the feature representation of edge regions can be enhanced, the classification accuracy of the algorithm can be improved, and the waste of model parameters can be reduced. The method can be widely applied to the field of computer vision multimodal data fusion technology.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision multimodal data fusion, and in particular to a high-fault-tolerant multimodal data fusion method based on a paired feature synthesis network. Background Art

[0002] In recent years, multimodal data fusion, particularly image and LiDAR data fusion, has played a crucial role in scene understanding and object recognition. However, because images are susceptible to varying weather conditions and lighting, and lack accurate depth information, they perform poorly in three-dimensional tasks and suffer from insufficient robustness. Multimodal data fusion using LiDAR point clouds and images can improve the integrity, accuracy, and reliability of perception tasks, achieving greater redundancy and fault tolerance.

[0003] The existing multimodal data fusion methods have the following problems:

[0004] 1. Feature extraction methods based on deep learning. Considering the particularity of different modalities, an intuitive approach is to use a disjoint stream fusion strategy to extract different features and then connect them. However, disjoint stream feature fusion methods require more parameters and model memory, which is poor in terms of mobility and portability.

[0005] 2. Feature extraction methods based on convolutional neural networks often enhance feature expression capabilities by increasing the number of training samples or convolutional layers, resulting in waste of model parameters.

[0006] 3. The context-aware feature extraction method uses sample weight sharing or indirect feature interaction strategies, which has low transmission efficiency during the feature fusion process. Summary of the Invention

[0007] In view of this, an embodiment of the present invention provides a highly fault-tolerant and high-precision multimodal data fusion method based on a paired feature synthesis network.

[0008] On the one hand, an embodiment of the present invention provides a highly fault-tolerant multimodal data fusion method based on a paired feature synthesis network, comprising:

[0009] Perform dynamic weight assignment and preprocessing on the input raw data to obtain the initial refined feature pairs;

[0010] Performing scale transformation on the refined feature pairs to obtain input features;

[0011] The input features are classified and predicted to obtain a multimodal data fusion classification result.

[0012] Optionally, the step of dynamically assigning weights and preprocessing the input raw data to obtain initial refined feature pairs includes:

[0013] The original data includes original image data and original lidar data;

[0014] Performing weight distribution and principal component analysis dimensionality reduction on the original image data to obtain first image data;

[0015] Performing weight distribution and convolution dimension conduction on the original laser radar data to obtain first laser radar data;

[0016] Performing block learning and channel segmentation on the first image data and the first lidar data to obtain original features of the image data and original features of the lidar data;

[0017] Performing a cross affine transformation on the original features of the image data and the original features of the radar data to obtain refined features of the image data and refined features of the radar data;

[0018] The image data refined features and radar data refined features are connected to obtain the initial refined feature pair.

[0019] Optionally, performing scale transformation on the refined feature pairs to obtain input features includes:

[0020] Converting the image data refined features and the radar data refined features into a 2D matrix;

[0021] Obtaining a feature pair confidence map from the 2D matrix using a first calculation formula;

[0022] Multiplying the feature pair confidence map and the initial refined feature pair to obtain a target refined feature pair;

[0023] The input features are obtained by concatenating all the grouped target refinement feature pairs.

[0024] Optionally, performing average pooling and convolution operations on the input features to obtain semantic features;

[0025] The semantic features are converted into prediction vectors through global average pooling, fully connected layers and Softmax function for feature classification to obtain multimodal data fusion classification results;

[0026] The true value and error of the classification result are detected by the cross entropy loss function.

[0027] Optionally, performing a cross affine transformation on the original features of the image data and the original features of the radar data to obtain refined features of the image data and refined features of the radar data includes:

[0028] The image data original features and the radar data original features are connected through complementary grouping to obtain complementary original features of the image data and complementary original features of the radar data;

[0029] The complementary original features of the image data are used to generate a first transformation factor and a second change factor through a second calculation formula;

[0030] The complementary original features of the radar data are used to generate a third transformation factor and a fourth transformation factor through a third calculation formula;

[0031] Obtain the radar data refined feature by applying a fourth calculation formula to the radar data original feature using the first transformation factor and the second transformation factor;

[0032] The refined features of the image data are obtained by applying a fifth calculation formula to the original features of the image data using the third transformation factor and the fourth transformation factor.

[0033] Optionally, in the step of generating the first transformation factor and the second change factor by using a second calculation formula from the complementary original features of the image data, the expression of the second calculation formula is:

[0034]

[0035] Among them, β hsi,g is the first transformation factor, γ hsi,g is the second transformation factor, split(.) is the channel splitting operation, It consists of two convolution operations with a kernel size of 1×1 and a ReLU function. Represents the complementary original features of the g-th group of image data.

[0036] Optionally, in the step of obtaining the radar data refined feature by using a fourth calculation formula on the original feature of the radar data using the first transformation factor and the second transformation factor, the fourth calculation formula is expressed as:

[0037]

[0038] Among them, X l l id,g represents the radar data refinement feature, β hsi,g is the first transformation factor, Represents element-wise multiplication operation, X lid,g represents the original features of the g-th group of radar data, γ hsi,g represents the second transformation factor.

[0039] On the other hand, an embodiment of the present invention further provides a highly fault-tolerant multimodal data fusion device based on a paired feature synthesis network, comprising:

[0040] The data pair affine representation module is used to dynamically assign weights and preprocess the input raw data to obtain initial refined feature pairs;

[0041] A feature pair scaling attention module is used to scale the initial refined feature pairs to obtain input features;

[0042] The multimodal data classification module is used to classify and predict the input features to obtain a multimodal data fusion classification result.

[0043] On the other hand, an embodiment of the present invention further provides an electronic device, including a processor and a memory, wherein the memory is used to store a program, and the processor executes the program to implement the above-mentioned high-fault-tolerant multimodal data fusion method based on the paired feature synthesis network.

[0044] On the other hand, an embodiment of the present invention further provides a computer-readable storage medium, which stores a program, and the program is executed by a processor to implement the above-mentioned high-fault-tolerant multimodal data fusion method based on the paired feature synthesis network.

[0045] The present invention also discloses a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium and execute the computer instructions, causing the computer device to perform the above method.

[0046] The embodiments of the present invention include at least the following beneficial results: the embodiments of the present invention ensure the system fault tolerance and reliability when a single data input fails by dynamically assigning weights and preprocessing the input raw data, and the dynamic weight assignment process can also improve transmission efficiency; the embodiments of the present invention obtain input features by performing a scale transformation on the refined feature pairs, wherein the scale transformation operation can highlight spatial information, enhance the feature representation of edge areas, improve the classification accuracy of the algorithm, reduce the waste of model parameters and model memory, and improve mobility and portability. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0048] Figure 1 Flowchart of a high-fault-tolerant multimodal data fusion method based on a paired feature synthesis network provided by an embodiment of the present invention;

[0049] Figure 22 is a schematic diagram of a high-fault-tolerant multimodal data fusion device based on a paired feature synthesis network provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solutions and advantages of this application more clearly understood, the present application is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0051] In view of the problems existing in the prior art, the embodiment of the present invention provides a high fault-tolerant multimodal data fusion method based on a paired feature synthesis network. Figure 1 As shown, the method includes steps 101 to 103:

[0052] Step 101: Perform dynamic weight assignment and preprocessing on the input raw data to obtain initial refined feature pairs.

[0053] The original data includes original image data and original lidar data; the original lidar data is lidar point cloud data, and each lidar point cloud data contains three-dimensional coordinate information.

[0054] According to the actual weather and light scene conditions, dynamic weight allocation is performed on the input multimodal data to analyze the interdependence of each branch and perform direct feature interaction and transmission.

[0055] The above step 101 performs dynamic weight distribution and preprocessing on the input raw data to obtain the initial refined feature pair, which may specifically include steps 201 to 205.

[0056] Step 201: performing weight distribution and principal component analysis dimensionality reduction on the original image data to obtain first image data.

[0057] Among them, the original image data is X HSI ∈R M×N×D , M is the width of the original image data, N is the height of the original image data, D is the number of channels of the original image data, R is the data set; the first image data is X' HSI =α hsi X HSI ∈R M×N×d , α hsi is the weight of the first data branch, α hsi ∈(0,1), R is the data set, M is the width of the data, N is the height of the data, and d is the number of channels after dimensionality reduction.

[0058] Step 202: Perform weight assignment and convolution dimension conduction on the original lidar data to obtain first lidar data.

[0059] Among them, the original lidar data is X LiD ∈R M×N×D , M is the width of the data, N is the height of the data, D is the number of channels of the data, R is the data set; the first lidar data is X′ LiD =α LiD X LiD ∈R M×N×d ,α LiD is the weight of the second data branch, α LiD ∈(0,1), R is the data set, M is the width of the data, N is the height of the data, and d is the number of channels after dimensionality reduction.

[0060] Step 203: Perform block learning and channel segmentation on the first image data and the first lidar data to obtain original features of the image data and original features of the lidar data.

[0061] Among them, block learning is to divide the M*N input data into multiple m*n block data and then input them into the network.

[0062] The first image data X' HSI and the first lidar data X′ LiD Perform block learning to obtain the original image feature X hsi and radar data original features X lid ; The training set for block learning is Among them, X is the training set of block learning, T is the total number of samples in training, i is the i-th value of the training set, and the corresponding true value label is y i ∈{1,2,…,C}, C is the largest class, the data block size is m*n, the convolution kernel size is K*K, and the image data block is The lidar data block is m represents the width of the block data, n represents the height of the block data, and d is the number of channels after dimensionality reduction.

[0063] Further X hsi and X lid Group along the channel domain. The expression of the grouping operation is:

[0064] {X hsi,1 ,…,X hsi,g ,…,X hsi,G}=split(X hsi )

[0065] {X lid,1 ,…,X lid,g ,…,X lid,G}=split(X lid )

[0066] Among them, split(·) is the channel splitting operation, represents the original features of the g-th group of image data, represents the original features of the g-th group of radar data, G represents the total number of groups, m represents the width of the block data, n represents the height of the block data, and d is the number of channels after dimensionality reduction.

[0067] Step 204: performing a cross affine transformation on the original features of the image data and the original features of the radar data to obtain refined features of the image data and refined features of the radar data.

[0068] The first data pair is connected by a specific group channel and an opposite residual channel and the second data pair in, is the complementary original feature of the g-th group of image data, Represents X hsi，g Connection of complementary groups; is the complementary original feature of the g-th group of radar data, Represents X lid,g Connection of complementary groups; the first data pair generates image data refinement feature X′ based on the original feature of the image data hsi,g The second data pair generates radar data refined features X′ based on the original features of the radar data lid,g .

[0069] The above step 204 performs a cross affine transformation on the original features of the image data and the original features of the radar data to obtain the refined features of the image data and the refined features of the radar data. This step specifically includes steps 301 to 305.

[0070] Step 301: The image data original features and the radar data original features are connected by complementary grouping to obtain image data complementary original features and radar data complementary original features.

[0071] in, is the complementary original feature of the g-th group of image data, indicating that X hsi,g Connection of complementary groups; is the complementary original feature of the g-th group of radar data, indicating X lid,g Connection of complementary groups.

[0072] Step 302: The complementary original features of the image data are used to generate a first transformation factor and a second change factor through a second calculation formula.

[0073] The expression of the second calculation formula is:

[0074]

[0075] Among them, βhsi,g is the first transformation factor, γ hsi,g is the second transformation factor, split(.) is the channel splitting operation, It consists of two convolution operations with a kernel size of 1×1 and a ReLU function. Represents the complementary original features of the g-th group of image data; the ReLU function is a commonly used activation function in neural networks, usually referring to a nonlinear function represented by a ramp function and its variants.

[0076] Step 303: The complementary original features of the radar data are used to generate a third transformation factor and a fourth transformation factor through a third calculation formula.

[0077] Wherein, the expression of the third calculation formula is:

[0078]

[0079] Among them, β lid,g is the third transformation factor, γ lid,g is the fourth transformation factor, split(.) is the channel splitting operation, It consists of two convolution operations with a kernel size of 1×1 and a ReLU function. represents the complementary original features of the g-th group of radar data.

[0080] Step 304: Obtain the radar data refined features by applying a fourth calculation formula to the original features of the radar data using the first transformation factor and the second transformation factor.

[0081] Wherein, the expression of the fourth calculation formula is:

[0082]

[0083] Among them, X′ lid,g represents the radar data refinement feature, β hsi,g is the first transformation factor, Represents element-wise multiplication operation, X lid,g represents the original features of the g-th group of radar data, γ hsi,g represents the second transformation factor.

[0084] Step 305: Obtain the refined features of the image data by applying a fifth calculation formula to the original features of the image data using the third transformation factor and the fourth transformation factor.

[0085] Wherein, the expression of the fifth calculation formula is:

[0086]

[0087] Among them, X′ hsi,grepresents the image data refinement feature, β lid,g is the third transformation factor, Represents element-wise multiplication operation, X hsi,g represents the original features of the g-th group of image data, γ lid,g represents the fourth transformation factor.

[0088] Step 205: Connect the image data refined features and the radar data refined features to obtain the initial refined feature pair.

[0089] The original feature pair X can be obtained by connecting the original feature of the image data and the original feature of the radar data. pair,g =(X hsi,g ,X lid,g ), X pair,g is the original feature pair, X hsi,g represents the original features of the g-th group of image data, X lid,g Represents the original features of the g-th group of radar data.

[0090] Perform cross affine transformation on the original feature pair to generate the initial refined feature pair, that is, the two refined features X′ hsi,g and X′ lid,g Connect and get the initial refined feature pair. The connection expression is as follows:

[0091] X′ pair,g =Concat(X′ lid,g ,X′ hsi,g )

[0092] Among them, X′ pair,g is the initial refined feature pair of the g-th group, and Concat(.) represents the connection operation.

[0093] Step 102: Performing scale transformation on the initial refined feature pair to obtain input features.

[0094] The above step 102 of performing scale transformation on the initial refined feature pair to obtain input features specifically includes steps 401 to 404 .

[0095] Step 401: Convert both the image data refined features and the radar data refined features into 2D matrices.

[0096] First, the scale transformation of the pooling operation is performed on the radar data refinement features to highlight the edge details. The expression of the pooling operation is:

[0097] pooling(X′ lid,g )

[0098] Among them, pooling(·) is the pooling operation.

[0099] Then refine the feature X′ of the image data hsi,g Perform convolution operation with a convolution kernel size of 32 to extract detail features, and perform pooling (X′ lid,g ) performs convolution operation with a convolution kernel size of 64 to extract detail features.

[0100] Finally, the 3D tensor X′ hsi,g and X′ lid,g Converted to a 2D matrix, the conversion expression is:

[0101]

[0102]

[0103] Among them, X″ hsi,g is the 2D matrix of image data refinement features, X″ lid,g is the 2D matrix of radar data refinement features, m×n is the size of the data block, and K×K is the size of the convolution kernel.

[0104] Step 402: Obtain a feature pair confidence map from the 2D matrix using a first calculation formula.

[0105] The first calculation formula is expressed as follows:

[0106] M g =Reshape(Max(X″) hsi,g ⊙(X″ lid,g ) T ))

[0107] Among them, M g is the confidence map of feature pairs, Reshape(.) is the matrix shape transformation operation function, ⊙ represents the dot product operation, Max(.) represents the maximum value of the result, (X″ lid,g ) T The transpose of the 2D matrix representing the refined features of the radar data.

[0108] Step 403: obtaining a target refined feature pair by multiplying the feature pair confidence map and the initial refined feature pair.

[0109] The expression of the multiplication operation is:

[0110]

[0111] Among them, X″ pair, Refine the feature pairs for the target, M g is the feature pair confidence map, Represents element-wise multiplication operation, X′ pair, is the initial refined feature pair.

[0112] Step 404: Connect all the grouped target refined feature pairs to obtain input features.

[0113] The expression of the connection is:

[0114] X″ pair, =Concat(X″) pair, ,…,X″ pair, ,…,X″ pair, )

[0115] Among them, X″ pair,g is the input feature, Concat(.) is the connection operation, and G represents the total number of groups.

[0116] Step 103: performing classification prediction on the input features to obtain a multimodal data fusion classification result.

[0117] The above step 103 of performing classification prediction on the input features to obtain a multimodal data fusion classification result specifically includes steps 501 to 503.

[0118] Step 501: Perform average pooling and convolution operations on the input features to obtain semantic features.

[0119] Among them, average pooling is to average the feature points in the field, which can well preserve the background; the convolution operation is performed through two convolution layers, the convolution kernel size of one convolution layer is 128, and the convolution kernel size of the other convolution layer is 256.

[0120] After performing average pooling and convolution operations on the input features, deeper semantic features are obtained.

[0121] Step 502: The semantic features are converted into prediction vectors through global average pooling, a fully connected layer, and a Softmax function for feature classification to obtain a multimodal data fusion classification result.

[0122] Among them, global average pooling is to output a value by globally averaging the feature map.

[0123] The fully connected layer plays the role of a "classifier" in the convolutional neural network, mapping the "distributed feature representation" learned by the neural network to the sample label space.

[0124] The Softmax function is an activation function used for multi-class classification problems where class membership is required for more than two class labels. The Softmax function compresses any real vector of length K into a real vector of length K with values ​​in the range [0, 1] and where the sum of the elements in the vector is 1.

[0125] Prediction vector For classification, C is the maximum category of data classification.

[0126] Step 503: Detect the true value and error of the classification result through the cross entropy loss function.

[0127] Among them, the cross entropy loss function is used to detect the true value and error of the classification result; the cross entropy can measure the degree of difference between two different probability distributions in the same random variable, which is expressed as the difference between the true probability distribution and the predicted probability distribution in machine learning; the smaller the cross entropy value, the better the model prediction effect.

[0128] The cross entropy loss function is used to measure the error between the true value and the predicted classification result.

[0129] The following describes an application scenario of the high-fault-tolerant multimodal data fusion method based on a paired feature synthesis network provided by an embodiment of the present invention.

[0130] 1. First, dynamically assign weights and preprocess the input raw data to obtain an initial refined feature pair. Based on the actual weather and lighting conditions, dynamically assign weights to the input multimodal data, namely the raw image data and raw lidar data. Perform weight assignment and principal component analysis dimensionality reduction on the raw image data to obtain first image data, and perform weight assignment and convolution dimensionality transfer on the raw lidar data to obtain first lidar data. Block learning and channel segmentation are performed on the first image data and the first lidar data to obtain original image data features and original radar data features. Cross-affine transformation is performed on the original image data features and the original radar data features to obtain refined image data features and refined radar data features. The refined image data features and the refined radar data features are concatenated to obtain the initial refined feature pair.

[0131] 2. Then, the initial refined feature pairs are scaled to obtain input features. The image data refined features and the radar data refined features are converted into 2D matrices. A feature pair confidence map is obtained from the 2D matrices using the first calculation formula. The feature pair confidence map and the initial refined feature pairs are multiplied to obtain target refined feature pairs. All grouped target refined feature pairs are concatenated to obtain input features.

[0132] 3. Finally, the input features are classified and predicted to obtain the multimodal data fusion classification results. The input features are average pooled and convolved to obtain semantic features. The semantic features are converted into prediction vectors through global average pooling, a fully connected layer, and a softmax function for feature classification to obtain the multimodal data fusion classification results. The true value and error of the classification results are detected using the cross entropy loss function.

[0133] In summary, the highly fault-tolerant multimodal data fusion method based on the paired feature synthesis network according to the embodiment of the present invention has the following advantages:

[0134] 1. The embodiment of the present invention ensures system fault tolerance and reliability when a single data input fails by dynamically assigning weights and preprocessing the input raw data. The dynamic weight assignment process can also improve transmission efficiency. Moreover, without increasing parameters and computational complexity, the feature characterization capability is enhanced by learning and refining features from specific channels and relative residual channels. The embodiment of the present invention performs direct feature interaction and transmission, thereby improving transmission efficiency during the feature fusion process.

[0135] 2. The embodiment of the present invention obtains input features by performing a scale transformation on the refined feature pairs, wherein the scale transformation operation can highlight spatial information, enhance the feature representation of edge areas, improve the classification accuracy of the algorithm, reduce the waste of model parameters, and improve mobility and portability.

[0136] Reference Figure 2 The embodiment of the present invention further provides a highly fault-tolerant multimodal data fusion device based on a paired feature synthesis network, comprising:

[0137] A data pair affine representation module 601 is used to dynamically assign weights and preprocess the input raw data to obtain initial refined feature pairs;

[0138] A feature pair scaling attention module 602 is configured to scale the initial refined feature pairs to obtain input features;

[0139] The multimodal data classification module 603 is used to perform classification prediction on the input features to obtain a multimodal data fusion classification result.

[0140] An embodiment of the present invention also provides an electronic device that can perform high-fault-tolerant multimodal data fusion based on a paired feature synthesis network. First, the input raw data is dynamically weighted and preprocessed to obtain an initial refined feature pair. Then, the initial refined feature pair is scaled to obtain input features. Finally, the input features are classified and predicted to obtain a multimodal data fusion classification result. The embodiment of the present invention ensures the system's fault tolerance and reliability when a single data input fails by dynamically weighting and preprocessing the input raw data. The dynamic weight allocation process can also improve transmission efficiency. Moreover, without increasing parameters and computational complexity, the feature characterization capability is enhanced by learning refined features from specific channels and relative residual channels. The embodiment of the present invention performs direct feature interaction and transmission, thereby improving transmission efficiency during the feature fusion process. The embodiment of the present invention obtains input features by scale-transforming the refined feature pair. The scale-transformation operation can highlight spatial information, enhance the feature representation of edge areas, improve the classification accuracy of the algorithm, reduce the waste of model parameters, and improve mobility and portability.

[0141] An embodiment of the present invention further provides a readable storage medium, which stores a program, and the program is executed by a processor to implement the above-mentioned high-fault-tolerant multimodal data fusion method based on the paired feature synthesis network.

[0142] The embodiment of the present invention also discloses a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs Figure 1 The method shown.

[0143] In some optional embodiments, the function / operation mentioned in the block diagram may not occur in the order mentioned in the operation diagram. For example, depending on the function / operation involved, the two boxes shown in succession can actually be executed substantially simultaneously or the boxes can sometimes be executed in reverse order. In addition, the embodiment presented and described in the flow chart of the present invention is provided in an exemplary manner for the purpose of providing a more comprehensive understanding of the technology. The disclosed method is not limited to the operation and logic flow presented herein. Optional embodiments are contemplated in which the order of the various operations is changed and the sub-operations described as a part of a larger operation are performed independently.

[0144] Furthermore, although the present invention is described in the context of functional modules, it should be understood that, unless otherwise indicated, one or more of the functions and / or features described may be integrated into a single physical device and / or software module, or one or more functions and / or features may be implemented in separate physical devices or software modules. It will also be understood that a detailed discussion of the actual implementation of each module is not necessary for understanding the present invention. More specifically, given the properties, functions, and internal relationships of the various functional modules in the devices disclosed herein, the actual implementation of the module will be understood within the ordinary skill of an engineer. Therefore, a person skilled in the art using ordinary skill will be able to implement the present invention set forth in the claims without undue experimentation. It will also be understood that the specific concepts disclosed are merely illustrative and are not intended to limit the scope of the present invention, which is determined by the full scope of the appended claims and their equivalents.

[0145] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0146] The logic and / or steps represented in the flowcharts or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing the logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (e.g., a computer-based system, a system including a processor, or other system that can fetch and execute instructions from an instruction execution system, apparatus, or device). For purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0147] More specific examples (a non-exhaustive list) of computer-readable media include the following: an electrical connection with one or more wires (electronic devices), a portable computer disk cartridge (magnetic devices), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disc read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering, or processing in another suitable manner as necessary, and then stored in a computer memory.

[0148] It should be understood that various parts of the present invention can be implemented using hardware, software, firmware, or a combination thereof. In the above-described embodiments, multiple steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof can be used: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application-specific integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0149] Throughout this specification, reference to terms such as "one embodiment," "some embodiments," "examples," "specific examples," or "some examples" means that a specific feature, structure, material, or characteristic described in conjunction with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

[0150] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to the embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the claims and their equivalents.

[0151] The above is a specific description of the preferred implementation of the present invention, but the present invention is not limited to the embodiments. Those skilled in the art can make various equivalent modifications or substitutions without violating the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of this application.

Claims

1. A highly fault-tolerant multimodal data fusion method based on a paired feature synthesis network, characterized in that: include: Perform dynamic weight assignment and preprocessing on the input raw data to obtain the initial refined feature pairs; Performing a scale transformation on the initial refined feature pair to obtain an input feature; Performing classification prediction on the input features to obtain a multimodal data fusion classification result; The dynamic weight assignment and preprocessing of the input raw data to obtain the initial refined feature pairs include: The original data includes original image data and original lidar data; Performing weight distribution and principal component analysis dimensionality reduction on the original image data to obtain first image data; Performing weight distribution and convolution dimension conduction on the original laser radar data to obtain first laser radar data; Performing block learning and channel segmentation on the first image data and the first lidar data to obtain original features of the image data and original features of the lidar data; Performing a cross affine transformation on the original features of the image data and the original features of the radar data to obtain refined features of the image data and refined features of the radar data; Connecting the image data refined features and the radar data refined features to obtain the initial refined feature pair; The classifying and predicting the input features to obtain a multimodal data fusion result includes: Performing average pooling and convolution operations on the input features to obtain semantic features; The semantic features are converted into prediction vectors through global average pooling, fully connected layers and Softmax function for feature classification to obtain multimodal data fusion classification results; The true value and error of the classification results are detected through the cross entropy loss function; The performing a cross affine transformation on the original features of the image data and the original features of the radar data to obtain the refined features of the image data and the refined features of the radar data includes: The image data original features and the radar data original features are connected through complementary grouping to obtain complementary original features of the image data and complementary original features of the radar data; The complementary original features of the image data are used to generate a first transformation factor and a second change factor through a second calculation formula; The complementary original features of the radar data are used to generate a third transformation factor and a fourth transformation factor through a third calculation formula; Obtain the radar data refined feature by applying a fourth calculation formula to the radar data original feature using the first transformation factor and the second transformation factor; The refined features of the image data are obtained by applying a fifth calculation formula to the original features of the image data using the third transformation factor and the fourth transformation factor.

2. The method of claim 1, wherein the method comprises: The step of performing scale transformation on the refined feature pairs to obtain input features includes: Converting the image data refined features and the radar data refined features into a 2D matrix; Obtaining a feature pair confidence map from the 2D matrix using a first calculation formula; Multiplying the feature pair confidence map and the initial refined feature pair to obtain a target refined feature pair; The target refined feature pairs of all groups are connected to obtain input features.

3. The method for high fault-tolerant multimodal data fusion based on paired feature synthesis network according to claim 1, characterized in that: In the step of generating the first transformation factor and the second change factor by using the second calculation formula for the complementary original features of the image data, the expression of the second calculation formula is: in, is the first transformation factor, is the second transformation factor, is the channel splitting operation, The two kernels are of size The convolution operation and a ReLU function are composed. Represents the complementary original features of the g-th group of image data.

4. The method of claim 3 for high fault tolerance multimodal data fusion based on paired feature synthesis network, characterized in that: In the step of obtaining the radar data refined features by using a fourth calculation formula on the original features of the radar data using the first transformation factor and the second transformation factor, the fourth calculation formula is expressed as: in, Represents the radar data refinement features, is the first transformation factor, represents element-wise multiplication operation, represents the original features of the g-th group of radar data, represents the second transformation factor.

5. A device for implementing the high-fault-tolerant multimodal data fusion method based on a paired feature synthesis network as described in any one of claims 1 to 4, characterized in that: include: The data pair affine representation module is used to dynamically assign weights and preprocess the input raw data to obtain initial refined feature pairs; A feature pair scaling attention module is used to scale the initial refined feature pairs to obtain input features; The multimodal data classification module is used to classify and predict the input features to obtain a multimodal data fusion classification result.

6. An electronic device, characterized in that: including a processor and a memory; The memory is used to store programs; The processor executes the program to implement the method according to any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that The storage medium stores a program, and the program is executed by a processor to implement the method according to any one of claims 1 to 4.

Images