Improved multi-modal information fusion extraction method and system

Abstract

The invention discloses an improved multi-modal information fusion extraction method and system, and relates to the technical field of multi-modal information processing, and the system comprises the following components: S1, data collection and preprocessing, S2, construction of a multi-task collaborative learning framework, S3, multi-modal information fusion and multi-task training, and S4, information extraction and application. According to the method, a multi-task collaborative learning framework and a shared information fusion module are constructed in a multi-modal information fusion and multi-task training stage, so that effective fusion and collaborative training of multi-modal data are realized, and rich and universal feature representation in the data is further mined; the improved method not only improves the accuracy of data processing, but also accelerates the processing speed, and provides powerful support for practical application.

Description

Technical Field

[0001] The present invention relates to the technical field of multimodal information processing, and in particular to an improved multimodal information fusion extraction method and system. Background Art

[0002] With the rapid development of information technology, multimodal information processing technology has shown broad application prospects in many fields, such as intelligent security, human-computer interaction, and medical diagnosis. Multimodal information fusion extraction, as the core link of this technology, aims to integrate and integrate data from different modalities to mine more comprehensive and rich information. However, how to achieve efficient and accurate multimodal information fusion extraction is an important challenge currently faced.

[0003] Traditional technologies have shortcomings. Traditional technologies cannot effectively fuse and collaboratively train multimodal data, and cannot further mine the rich and universal feature representations in the data. Therefore, it is particularly important to develop an improved multimodal information fusion extraction method and system. Summary of the invention

[0004] The purpose of the present invention is to make up for the deficiencies of the prior art and to provide an improved multimodal information fusion extraction method and system, which realizes the effective fusion and collaborative training of multimodal data through innovative algorithm and framework design.

[0005] In order to solve the above technical problems, the present invention provides the following technical solutions: an improved multimodal information fusion extraction method, characterized in that the specific steps of the method are:

[0006] S1, data acquisition and preprocessing: collect data of multiple modalities, and perform preprocessing operations on the data of each modality to obtain the initial feature representation of each modality. For image modal data, a feature extraction algorithm combining multi-scale texture structure tensor and deep convolutional neural network feature extraction layer is used. The multi-scale texture structure tensor calculates the texture direction and intensity information of the image at different scales. The deep convolutional neural network feature extraction layer adopts convolution kernel layout, and its convolution kernel is non-uniformly distributed in the spatial dimension and the distribution density is determined according to the prior statistical law of image texture. For text modal data, The feature extraction method of semantic role labeling and word vector dynamic weighting is used to determine the semantic role weight of each word in the text through semantic role labeling. The dynamic weighting of word vectors is based on the syntactic structure and semantic dependency of the text, and a dynamic weighting matrix is ​​constructed to weight the word vectors. For audio modal data, the feature extraction operation of harmonic structure analysis and time-frequency domain feature fusion is performed. The audio signal is first subjected to harmonic structure analysis to extract the frequency and amplitude parameters of the fundamental wave and harmonic wave, and then the parameters are fused with the short-time energy and zero-crossing rate characteristics of the audio signal in the time-frequency domain to obtain a comprehensive feature representation of the audio modality.

[0007] S2, build a multi-task collaborative learning framework: establish a multi-task collaborative learning framework including a classification task module, a regression task module and a generation task module, and the framework is provided with a shared information fusion module for receiving the initial feature representation of each modality. The shared information fusion module adopts a fusion algorithm driven by modality correlation. The feature vectors from different modalities are F1, F2, ..., F n , calculate the correlation matrix C between the modes, where the element c ij represents the degree of association between the i-th mode and the j-th mode, which calculates the joint measure of mutual information and covariance between modal eigenvectors, i.e. c ij =α×MI(F i ,F j )+(1-α)×Cov(F i ,F j (, where MI(F i ,F j ) is F i With F j The mutual information of Cov(F i ,F j ) is their covariance, α is the adjustment coefficient, and the fusion weight vector w is calculated according to the correlation matrix. The calculation formula is: The fused feature vector

[0008] S3, multimodal information fusion and multi-task training: Use the shared information fusion module to fuse the multimodal data to obtain the fused feature representation, write the fused features into different task modules for training, and optimize the parameters of the shared information fusion module and each task module through multi-task collaborative training to explore richer and more universal feature representations in multimodal data. During the training process, the classification task module adopts the classification algorithm of deep forest and multimodal feature hierarchical aggregation to construct multiple random forest classifiers. In each classifier, the multimodal features are hierarchically aggregated according to their semantic hierarchy and feature importance. The distance between the cluster center and each modal feature is used as the basis for weight allocation. The output of the classifier is combined by weighted voting to obtain the final classification result. The regression task module uses a regression model that combines kernel ridge regression with multimodal features adaptively. The kernel function uses the Laplace kernel function of the geometric structure of the multimodal feature space. where f i (x) and f i (y) are the values ​​of samples x and y on the i-th modal feature, is the variance of the i-th modal feature, β and γ are adjustable parameters, and the weight of the model is calculated by minimizing the regularized loss function of the residual OK, where y i is the true value, is the predicted value, m is the number of samples, λ is the regularization coefficient, and the generation task module adopts the generation model guided by conditional variational autoencoder and multimodal semantics. The encoder part maps the fusion feature to the base variable space through the transformation formula: Where μ and σ are the mean and standard deviation vectors learned through the neural network, ⊙ represents element-by-element multiplication, ∈ is a standard normal distribution random vector, and g i (F fusion ) is the feature transformation function guided by the i-th modal semantics, θ i is the corresponding weight coefficient, which is determined by analyzing the semantic structure of the multimodal training data. The decoder part generates new sample features according to the latent variable z, and each task module shares some neural network layer parameters during the training process;

[0009] S4, Information Extraction and Application: Use the trained system to fuse and extract new multimodal data and apply it to classification, regression or generation tasks.

[0010] Furthermore, in the data acquisition and preprocessing steps, for the calculation of the multi-scale texture structure tensor of the image modal data, the selection of different scales is adaptively determined based on the resolution and texture complexity of the image. Specifically, the image is decomposed into a multi-scale pyramid, the entropy and contrast of the texture at each scale are calculated, and the three scales with the largest product of entropy and contrast are selected as feature extraction scales. The non-uniform distribution parameters of the convolution kernel of the deep convolutional neural network feature extraction layer are determined according to the texture energy distribution of the image in different directions. The distribution density ratio of the convolution kernel in different directions is determined by statistical analysis of the texture direction histogram of a large number of image samples.

[0011] Furthermore, in the step of constructing a multi-task collaborative learning framework, when the shared information fusion module calculates the inter-modal correlation matrix, the mutual information MI (F i ,F j ) is calculated using an approximate calculation method based on information entropy. The modal eigenvector is first discretized, and the discrete interval is adaptively determined according to the distribution range of the eigenvalue and the amount of data. Then, the information entropy and joint information entropy are calculated based on the discretized eigenvector, and then the mutual information and covariance Cov(F i ,F j ) is calculated using a covariance calculation method based on sample weighting. The weight is determined according to the distance of the samples in the multimodal feature space. The closer the distance, the greater the weight, so as to improve the accuracy of the correlation matrix calculation and its adaptability to the characteristics of multimodal data.

[0012] Furthermore, in the multimodal information fusion and multi-task training steps, in the classification algorithm based on deep forest and multimodal feature hierarchical aggregation of the classification task module, the number of random forest classifiers is determined according to the number of categories and feature dimensions of the multimodal data, and an initial number range is determined by analyzing the category distribution of the training data and evaluating the feature importance, and then the number of random forests that gives the highest classification accuracy is selected within the range through cross-validation. When the multimodal features are hierarchically aggregated, the division of the semantic hierarchy is based on the domain knowledge and semantic annotation information of the multimodal data, and the feature importance is jointly determined by the information gain ratio and the correlation coefficient between the feature and the category label.

[0013] Furthermore, in the multimodal information fusion and multi-task training steps, in the regression model of the regression task module based on kernel ridge regression and adaptive combination of multimodal features, β and γ in the Laplace kernel function are determined by a method combining grid search and Bayesian optimization. A better parameter range is first determined by grid search, and then Bayesian optimization is used to find a parameter combination within the range that minimizes the mean square error of the model on the validation set. The regularization coefficient λ determined by the model weight is determined according to the noise level and feature redundancy of the data. An initial value range is determined by noise analysis and feature correlation analysis of the multimodal data and is dynamically adjusted according to the performance of the model on the validation set during the training process.

[0014] Furthermore, in the multimodal information fusion and multi-task training steps, in the generative model based on conditional variational autoencoder and multimodal semantic guidance of the generated task module, the feature transformation function g based on multimodal semantic guidance i (F fusion ) is realized by constructing a semantic neural network. The structure of the neural network is determined according to the complexity of the semantic structure of multimodal data, including a semantic encoding layer, a semantic conversion layer, and a semantic decoding layer. The semantic encoding layer maps the fusion features to the semantic space. The semantic conversion layer converts the semantic representation according to the semantic relationship of different modalities. The semantic decoding layer maps the converted semantic representation back to the feature space. Its weight coefficient θ i It is determined by analyzing the semantic similarity of multimodal training data and the target requirements of the generation task.

[0015] Furthermore, in the multi-task collaborative learning framework, the neural network layer shared between the task modules adopts a sharing method based on multimodal feature dynamic routing, that is, the shared layer dynamically determines the information transmission path according to the input multimodal fusion features. Suppose the input feature of the shared layer is F in , the output feature is F out , by constructing a routing decision network R, whose output is the routing vector r, then in The elements of the routing vector r are determined by the routing decision network according to the feature distribution of the multimodal fusion features, task requirements and current training status, so as to further improve the synergy between the task modules.

[0016] Furthermore, in the information extraction and application steps, for the fusion extraction of new multimodal data, when some modalities are missing in the data, a fusion strategy based on adversarial generative network and multimodal knowledge transfer is adopted. First, a multimodal knowledge extraction network is trained using existing modal data. The network extracts the multimodal knowledge representation in the existing modal data through an autoencoder structure, and then inputs it into the adversarial generative network as prior knowledge. The generator of the adversarial generative network generates approximate data of the missing modality based on the existing modality and multimodal knowledge. The discriminator network is used to distinguish the difference between the generated data and the real data. The generator network is continuously optimized through adversarial training to achieve the best compatibility and consistency between the reconstructed modal data and the original multimodal data in the fusion process, thereby achieving effective fusion extraction of incomplete multimodal data.

[0017] Furthermore, in the step of constructing a multi-task collaborative learning framework, the shared information fusion module performs feature purification on each modal feature vector before performing modal correlation-driven fusion, and adopts a joint processing method based on principal component analysis and outlier detection. The principal component analysis is first used to remove redundant components in each modal feature vector, and then the outlier detection algorithm based on local outlier factors is used to remove outliers in the feature vector, so as to improve the quality and stability of the fused data.

[0018] On the other hand, an improved multimodal information fusion extraction system is characterized in that the system includes a data acquisition and preprocessing module, a multi-task collaborative learning framework construction module, a multimodal information fusion and multi-task training module, and an information extraction and application module;

[0019] Data acquisition and preprocessing module: collects data of multiple modalities, and preprocesses the data of each modality to obtain the initial feature representation of each modality. For image modal data, a feature extraction algorithm based on the combination of multi-scale texture structure tensor and deep convolutional neural network feature extraction layer is used. The multi-scale texture structure tensor calculates the texture direction and intensity information of the image at different scales. The deep convolutional neural network feature extraction layer adopts a specially designed convolution kernel layout. The convolution kernel is non-uniformly distributed in the spatial dimension and the distribution density is determined according to the prior statistical law of the image texture. For text modal data, A feature extraction method based on semantic role labeling and dynamic weighting of word vectors is adopted. The semantic role weights of each word in the text are determined by semantic role labeling. The dynamic weighting of word vectors is based on the syntactic structure and semantic dependency of the text, and a dynamic weighting matrix is ​​constructed to weight the word vectors. For audio modal data, a feature extraction operation based on harmonic structure analysis and time-frequency domain feature fusion is performed. The audio signal is first subjected to harmonic structure analysis to extract the frequency and amplitude parameters of the fundamental wave and harmonic wave, and then the parameters are fused with the short-time energy and zero-crossing rate characteristics of the audio signal in the time-frequency domain to obtain a comprehensive feature representation of the audio modality.

[0020] Construct a multi-task collaborative learning framework module: Establish a multi-task collaborative learning framework including a classification task module, a regression task module and a generation task module, and the framework is provided with a shared information fusion module for receiving the initial feature representation of each modality. The shared information fusion module adopts a fusion algorithm driven by modality correlation. The feature vectors from different modalities are F1, F2, …, F n , calculate the correlation matrix C between the modes, where the element c ij It represents the degree of association between the i-th mode and the j-th mode, which is calculated based on the joint measurement of mutual information and covariance between modal eigenvectors, that is, c ij =α×MI(F i ,F j )+(1-α)×Cov(F i ,F j ), where MI(F i ,F j ) is F i With F j The mutual information of Cov(F i ,F j ) is their covariance, α is the adjustment coefficient, and the fusion weight vector w is calculated according to the correlation matrix. The calculation formula is: The fused feature vector

[0021] Multimodal information fusion and multi-task training module: Use the shared information fusion module to fuse multimodal data to obtain fused feature representation, write the fused features into different task modules for training, and optimize the parameters of the shared information fusion module and each task module through multi-task collaborative training to explore richer and more universal feature representations in multimodal data. During the training process, the classification task module adopts a classification algorithm based on deep forest and multimodal feature hierarchical aggregation to construct multiple random forest classifiers. In each classifier, multimodal features are hierarchically aggregated according to their semantic hierarchy and feature importance. The distance between the cluster center and each modal feature is used as the basis for weight allocation. The output of the classifier is combined by weighted voting to obtain the final classification result. The regression task module uses a regression model based on kernel ridge regression and adaptive combination of multimodal features. The kernel function uses the Laplace kernel function based on the geometric structure of the multimodal feature space. where f i (x) and f i (y) are the values ​​of samples x and y on the i-th modal feature, is the variance of the i-th modal feature, β and γ are adjustable parameters, and the weight of the model is obtained by minimizing the regularized loss function based on the residual OK, where y i is the true value, is the predicted value, m is the number of samples, λ is the regularization coefficient, and the generation task module adopts a generation model based on conditional variational autoencoder and multimodal semantic guidance. The encoder part maps the fusion feature to the base variable space through the transformation formula: Where μ and σ are the mean and standard deviation vectors learned through the neural network, ⊙ represents element-by-element multiplication, ∈ is a standard normal distribution random vector, and g i (F fusion ) is the feature transformation function guided by the i-th modal semantics, θ i is the corresponding weight coefficient, which is determined by analyzing the semantic structure of the multimodal training data. The decoder part generates new sample features according to the latent variable z, and each task module shares some neural network layer parameters during the training process;

[0022] Information extraction and application module: Use the trained system to fuse and extract new multimodal data and apply it to classification, regression or generation tasks.

[0023] Compared with the prior art, an improved multimodal information fusion extraction method and system has the following beneficial effects:

[0024] 1. The present invention realizes the effective fusion and collaborative training of multimodal data by constructing a multi-task collaborative learning framework and a shared information fusion module in the multimodal information fusion and multi-task training stages, and further mines the rich and universal feature representations in the data. This improved method not only improves the accuracy of data processing, but also speeds up the processing speed, providing strong support for practical applications.

[0025] 2. The present invention further improves the performance and stability of the system through a fusion algorithm driven by modal correlation and a sharing technology based on dynamic routing of multimodal features. This improved system is not only suitable for current multimodal information processing tasks, but can also be expanded and optimized according to actual needs, laying a solid foundation for the future development of multimodal information processing technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0027] Figure 1 An improved multimodal information fusion extraction method and system operation flow chart. DETAILED DESCRIPTION

[0028] The technical solutions in the embodiments of the present invention are described clearly and completely below. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0029] Embodiment 1

[0030] This embodiment describes that in the imaging diagnosis department of a hospital, doctors need to conduct a comprehensive analysis of the patient's various imaging data, medical record text data, and the doctor's voice recording to make accurate diagnostic decisions, such as determining the type of disease, predicting the development trend of the disease, and generating simulated imaging effects after treatment.

[0031] For image modality data, medical imaging equipment is used to collect patient images, and a feature extraction algorithm based on a combination of a multi-scale texture structure tensor and a deep convolutional neural network feature extraction layer is used. The multi-scale texture structure tensor adaptively determines different scales according to the resolution and texture complexity of the image. For example, when analyzing lung CT images, for areas with high resolution and complex textures, a finer scale is selected to accurately capture the lung texture features. The convolution kernel of the deep convolutional neural network feature extraction layer determines the non-uniform distribution in the spatial dimension based on the energy distribution of the lung texture in different directions, so as to more effectively extract lung disease-related features. For text modality data, a semantic role labeling and word recognition based algorithm is used. The feature extraction method of dynamic vector weighting determines the semantic role weight of each word of symptoms and examination results in the medical record through semantic role labeling, and then constructs a dynamic weighting matrix according to the syntactic structure and semantic dependency of the medical record to perform weighted combination of word vectors. For example, higher weights are given to words describing key symptoms. For audio modal data, feature extraction operations based on harmonic structure analysis and time-frequency domain feature fusion are performed. First, the speech signal is subjected to harmonic structure analysis to extract the frequency and amplitude parameters of the fundamental wave and harmonics, which are then fused with the short-time energy and zero-crossing rate features of the speech signal in the time-frequency domain to obtain a comprehensive feature representation of the audio modality. For example, diagnosis can be aided by analyzing the intonation and speaking speed features in the doctor's voice.

[0032] A multi-task collaborative learning framework is established, which includes a classification task module (determining the disease type), a regression task module (predicting the disease development) and a generation task module (simulating the treatment effect). The shared information fusion module in the framework receives the initial feature representation of each modality and calculates the correlation matrix between the modalities. For example, when calculating the correlation between images and medical records, the mutual information and covariance are used to measure the correlation between the images and the medical records. ij =α×MI(F i ,F j )+(1-α)×Cov(F i ,F j ), where MI(F i ,F j ) is F i With F j The mutual information of Cov(F i ,F j ) is their covariance, α is the adjustment coefficient, and the fusion weight vector w is calculated according to the correlation matrix. The calculation formula is: The fused feature vector Then, the fusion weight vector is calculated according to the correlation matrix to achieve modal fusion.

[0033] The shared information fusion module is used to fuse the multimodal data to obtain the fused feature representation, and the fused features are input into different task modules for training. The classification task module adopts a classification algorithm based on deep forest and multimodal feature hierarchical aggregation. The number of random forest classifiers is determined according to the number of disease types and feature dimensions. For example, 10 random forest classifiers are constructed. The multimodal features are semantically hierarchically divided according to medical domain knowledge and semantic annotation information. The feature importance is jointly determined by the information gain ratio and the correlation coefficient between the feature and the disease category label. The classifier output is combined by weighted voting to obtain the final disease type classification result.

[0034] The regression task module uses a regression model based on kernel ridge regression and adaptive combination of multimodal features. The kernel function uses the Laplace kernel function based on the geometric structure of the multimodal feature space. where f i (x) and f i (y) are the values ​​of samples x and y on the i-th modal feature, is the variance of the i-th modal feature, β and γ are adjustable parameters, and the weight of the model is obtained by minimizing the regularized loss function based on the residual OK, where y i is the true value, is the predicted value, m is the number of samples, λ is the regularization coefficient, and the generation task module adopts a generation model based on conditional variational autoencoder and multimodal semantic guidance. The encoder part maps the fusion feature to the base variable space through the transformation formula: Where μ and σ are the mean and standard deviation vectors learned through the neural network, ⊙ represents element-by-element multiplication, ∈ is a standard normal distribution random vector, and g i (F fusion ) is the feature transformation function guided by the i-th modal semantics, θ i is the corresponding weight coefficient. The sum in the Laplace kernel function is determined by combining grid search with Bayesian optimization. First, a better parameter range is determined by grid search, and then Bayesian optimization is used to find the parameter combination that minimizes the mean square error of the model on the validation set (part of the historical case data) within this range. The regularization coefficient is determined according to the noise level in the medical data (such as the noise of the imaging equipment, the inaccuracy in the medical records) and the feature redundancy (such as some repeated information in the image, and insignificant descriptions in the medical records). The initial value range is determined by noise analysis and feature correlation analysis of multimodal data. During the training process, the model is dynamically adjusted according to its performance on the validation set. The model predicts the development trend of the disease, such as predicting the time point when the disease worsens.

[0035] The generation task module adopts a generation model based on conditional variational autoencoder and multimodal semantic guidance. The feature transformation function based on multimodal semantic guidance is realized by constructing a semantic neural network. Its structure is determined according to the complexity of the semantic structure of medical data (such as the complex semantic relationship between image features and pathological symptoms and treatment plans), including a semantic encoding layer, a semantic conversion layer and a semantic decoding layer. The semantic encoding layer maps the fused features to the semantic space. The semantic conversion layer converts the semantic representation according to the semantic relationship between different modalities of image, pathology and speech. The semantic decoding layer maps the converted semantic representation back to the feature space. The weight coefficient is determined by the semantic similarity analysis of the multimodal training data (such as the similarity of images, medical records and speech features of similar cases) and the target requirements of the generation task (such as generating image effects that meet the expectations of treatment) to generate simulated image effects after treatment. During the training process, each task module shares some neural network layer parameters, such as sharing the neural network layer for preliminary processing of multimodal features.

[0036] The trained system is used to fuse and extract the multimodal data of new patients. For example, when the voice record of a new patient is missing, a fusion strategy based on a generative adversarial network and multimodal knowledge transfer is adopted. First, a multimodal knowledge extraction network is trained using the existing patient's images and medical records. The network extracts the multimodal knowledge representation in the existing data through an autoencoder structure, and then inputs it into the generative adversarial network as prior knowledge. The generator of the generative adversarial network generates approximate data of the missing voice modality based on the existing images, medical records and multimodal knowledge. The discriminator network is used to distinguish the difference between the generated data and the real voice data. The generator network is continuously optimized through adversarial training to achieve the best compatibility and consistency between the reconstructed voice data and the original multimodal data during the fusion process. Finally, the fused multimodal data is applied to disease diagnosis, disease progression prediction and treatment effect simulation tasks.

[0037] Embodiment 2

[0038] This embodiment describes that in a smart driving car, the vehicle is equipped with a camera to obtain road scene images, on-board sensors to collect vehicle driving status data, and road text information provided by a navigation system. The system needs to integrate these multimodal information to realize traffic sign recognition, vehicle driving trajectory prediction, and generate simulated driving scenes for simulation training.

[0039] For image modal data, a feature extraction algorithm based on the combination of a multi-scale texture structure tensor and a deep convolutional neural network feature extraction layer is used. The multi-scale texture structure tensor adaptively determines the scale according to the resolution and texture complexity of the image. For example, in urban road scenes, a finer scale is selected to capture the detailed texture of traffic signs and road markings. The convolution kernel of the deep convolutional neural network feature extraction layer determines the non-uniform distribution according to the texture energy distribution of common texture directions in road images, such as setting a higher convolution kernel distribution density in the direction of the lane line.

[0040] For text modal data, a feature extraction method based on semantic role labeling and dynamic weighting of word vectors is adopted. The semantic role weights of each word in road names and intersection information are determined through semantic role labeling. A dynamic weighting matrix is ​​constructed according to the syntactic structure and semantic dependency of the text to perform weighted combination of word vectors. For example, higher weights are given to words related to key intersections and destinations. For audio modal data, a feature extraction operation based on harmonic structure analysis and time-frequency domain feature fusion is performed. The numerical changes in sensor data are regarded as fluctuations similar to audio signals. Harmonic structure analysis is performed to extract the frequency and amplitude parameters of the fundamental and harmonic waves, which are then fused with the short-time energy and zero-crossing rate characteristics of the sensor data in the time-frequency domain to obtain a comprehensive feature representation. For example, the frequency change characteristics of speed sensor data can be analyzed to assist in judging the driving status of a vehicle.

[0041] A multi-task collaborative learning framework including classification task module, regression task module and generation task module is established. The shared information fusion module calculates the correlation matrix between modalities. The feature vectors from different modalities are assumed to be F1, F2, …, F n , calculate the correlation matrix C between the modes, where the element c ij It represents the degree of association between the i-th mode and the j-th mode, which is calculated based on the joint measurement of mutual information and covariance between modal eigenvectors, that is, c ij =α×MI(F i ,F j )+(1-α)×Cov(F i ,F j ), where MI(F i ,F j ) is F i With F j The mutual information of Cov(F i ,F j ) is their covariance, α is the adjustment coefficient, and the fusion weight vector w is calculated according to the correlation matrix. The calculation formula is: The fused feature vector For example, when calculating the correlation between the camera image and the navigation text, the mutual information adopts an approximate calculation method based on information entropy, and the covariance adopts a covariance calculation method based on sample weighting. Then, the fusion weight vector is calculated according to the correlation matrix to fuse the features of each modality.

[0042] The shared information fusion module is used to fuse the multimodal data to obtain the fused feature representation, and the fused features are input into different task modules for training. The classification task module adopts a classification algorithm based on deep forest and multimodal feature hierarchical aggregation. The number of random forest classifiers is determined according to the number of traffic sign types and feature dimensions. For example, 8 random forest classifiers are constructed. The multimodal features are semantically hierarchically divided according to traffic domain knowledge and semantic annotation information. The feature importance is jointly determined by the information gain ratio and the correlation coefficient between the feature and the traffic sign category label. The classifier output is combined by weighted voting to obtain the final traffic sign recognition result.

[0043] The regression task module uses a regression model based on kernel ridge regression and adaptive combination of multimodal features. The kernel function uses the Laplace kernel function based on the geometric structure of the multimodal feature space. where f i (x) and f i (y) are the values ​​of samples x and y on the i-th modal feature, is the variance of the i-th modal feature, β and γ are adjustable parameters, and the weight of the model is obtained by minimizing the regularized loss function based on the residual OK, where y i is the true value, is the predicted value, m is the number of samples, λ is the regularization coefficient, and the generation task module adopts a generation model based on conditional variational autoencoder and multimodal semantic guidance. The encoder part maps the fusion feature to the base variable space through the transformation formula: Where μ and σ are the mean and standard deviation vectors learned through the neural network, ⊙ represents element-by-element multiplication, ∈ is a standard normal distribution random vector, and g i (F fusion ) is the feature transformation function guided by the i-th modal semantics, θ i is the corresponding weight coefficient. The sum in the Laplace kernel function is determined by combining grid search and Bayesian optimization. First, a better parameter range is determined by grid search, and then Bayesian optimization is used to find the parameter combination that minimizes the mean square error of the model on the validation set within this range. The regularization coefficient is determined according to the noise level and feature redundancy in the vehicle data. The initial value range is determined by noise analysis and feature correlation analysis of multimodal data. During the training process, the model is dynamically adjusted according to its performance on the validation set. The model predicts the vehicle's trajectory, such as predicting the vehicle's position and direction in the next few seconds.

[0044] The generation task module adopts a generation model based on conditional variational autoencoder and multimodal semantic guidance. The feature transformation function based on multimodal semantic guidance is realized by constructing a semantic neural network. Its structure is determined according to the complexity of the semantic structure of the driving scene data, including a semantic encoding layer, a semantic conversion layer and a semantic decoding layer. The semantic encoding layer maps the fused features to the semantic space. The semantic conversion layer converts the semantic representation according to the semantic relationship of different modalities of images, texts and sensors. The semantic decoding layer maps the converted semantic representation back to the feature space. The weight coefficient is determined by the semantic similarity analysis of the multimodal training data and the target requirements of the generation task. A simulated driving scene is generated for simulation training. Each task module shares some neural network layer parameters during the training process, such as sharing the neural network layer for basic processing of multimodal features.

[0045] The trained system is used to fuse and extract multimodal data of new driving scenarios. For example, when the text information of the vehicle navigation system is missing, a fusion strategy based on adversarial generative network and multimodal knowledge transfer is adopted. First, a multimodal knowledge extraction network is trained using the images and sensor data of the existing driving scenes. The network extracts the multimodal knowledge representation in the existing data through the autoencoder structure, and then inputs it into the adversarial generative network as prior knowledge. The generator of the adversarial generative network generates approximate data of the missing navigation text modality based on the existing images, sensor data and multimodal knowledge. The discriminator network is used to distinguish the difference between the generated data and the real navigation text data. The generator network is continuously optimized through adversarial training to achieve the best compatibility and consistency between the reconstructed navigation text data and the original multimodal data in the fusion process. Finally, the fused multimodal data is applied to traffic sign recognition, vehicle trajectory prediction and simulated driving scene generation tasks to assist intelligent driving cars in making safe and reasonable driving decisions.

[0046] It will be apparent to those skilled in the art that the invention is not limited to the details of the exemplary embodiments described above and that the invention can be implemented in other specific forms without departing from the spirit or essential features of the invention. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description, and it is intended that all variations falling within the meaning and scope of the equivalent elements of the claims be included in the invention. Any reference numeral in a claim should not be considered as limiting the claim to which it relates.

Claims

1. An improved multimodal information fusion extraction method, characterized in that: The specific steps of this method are: S1, data acquisition and preprocessing: collect data of multiple modalities, and perform preprocessing operations on the data of each modality to obtain the initial feature representation of each modality. For image modal data, a feature extraction algorithm combining multi-scale texture structure tensor and deep convolutional neural network feature extraction layer is used. The multi-scale texture structure tensor calculates the texture direction and intensity information of the image at different scales. The deep convolutional neural network feature extraction layer adopts convolution kernel layout, and its convolution kernel is non-uniformly distributed in the spatial dimension and the distribution density is determined according to the prior statistical law of image texture. For text modal data, The feature extraction method of semantic role labeling and word vector dynamic weighting is used to determine the semantic role weight of each word in the text through semantic role labeling. The dynamic weighting of word vectors is based on the syntactic structure and semantic dependency of the text, and a dynamic weighting matrix is ​​constructed to weight the word vectors. For audio modal data, the feature extraction operation of harmonic structure analysis and time-frequency domain feature fusion is performed. The audio signal is first subjected to harmonic structure analysis to extract the frequency and amplitude parameters of the fundamental wave and harmonic wave, and then the parameters are fused with the short-time energy and zero-crossing rate characteristics of the audio signal in the time-frequency domain to obtain a comprehensive feature representation of the audio modality. S2, build a multi-task collaborative learning framework: establish a multi-task collaborative learning framework including a classification task module, a regression task module and a generation task module, and the framework is provided with a shared information fusion module for receiving the initial feature representation of each modality. The shared information fusion module adopts a fusion algorithm driven by modality correlation. The feature vectors from different modalities are F1, F2, ..., F n , calculate the correlation matrix C between the modes, where the element c ij represents the degree of association between the i-th mode and the j-th mode, which calculates the joint measure of mutual information and covariance between modal eigenvectors, i.e. c ij =α×MI(F i ,F j )+(1-α)×Cov(F i ,F j ), where MI(F i ,F j ) is F i With F j The mutual information of Cov(F i ,F j ) is their covariance, α is the adjustment coefficient, and the fusion weight vector w is calculated according to the correlation matrix. The calculation formula is: The fused feature vector S3, multimodal information fusion and multi-task training: Use the shared information fusion module to fuse the multimodal data to obtain the fused feature representation, write the fused features into different task modules for training, and optimize the parameters of the shared information fusion module and each task module through multi-task collaborative training to explore richer and more universal feature representations in multimodal data. During the training process, the classification task module adopts the classification algorithm of deep forest and multimodal feature hierarchical aggregation to construct multiple random forest classifiers. In each classifier, the multimodal features are hierarchically aggregated according to their semantic hierarchy and feature importance. The distance between the cluster center and each modal feature is used as the basis for weight allocation. The output of the classifier is combined by weighted voting to obtain the final classification result. The regression task module uses a regression model that combines kernel ridge regression with multimodal features adaptively. The kernel function uses the Laplace kernel function of the geometric structure of the multimodal feature space. where f i (x) and f i (y) are the values ​​of samples x and y on the i-th modal feature, is the variance of the i-th modal feature, β and γ are adjustable parameters, and the weight of the model is calculated by minimizing the regularized loss function of the residual OK, where y i is the true value, is the predicted value, m is the number of samples, λ is the regularization coefficient, and the generation task module adopts the generation model guided by conditional variational autoencoder and multimodal semantics. The encoder part maps the fusion feature to the base variable space through the transformation formula: Where μ and σ are the mean and standard deviation vectors learned through the neural network, ⊙ represents element-by-element multiplication, ∈ is a standard normal distribution random vector, and g i (F fusion ) is the feature transformation function guided by the i-th modal semantics, θ i is the corresponding weight coefficient, which is determined by analyzing the semantic structure of the multimodal training data. The decoder part generates new sample features according to the latent variable z, and each task module shares some neural network layer parameters during the training process; S4, Information Extraction and Application: Use the trained system to fuse and extract new multimodal data and apply it to classification, regression or generation tasks.

2. According to claim 1, an improved multimodal information fusion extraction method is characterized in that: In the data acquisition and preprocessing steps, for the calculation of the multi-scale texture structure tensor of the image modal data, the selection of different scales is adaptively determined based on the resolution and texture complexity of the image. Specifically, the image is decomposed into a multi-scale pyramid, the entropy and contrast of the texture at each scale are calculated, and the three scales with the largest product of the entropy and the contrast are selected as the feature extraction scales. The non-uniform distribution parameters of the convolution kernel of the deep convolutional neural network feature extraction layer are determined according to the texture energy distribution of the image in different directions. The distribution density ratio of the convolution kernel in different directions is determined by statistically analyzing the texture direction histograms of a large number of image samples.

3. The improved multimodal information fusion extraction method according to claim 1, characterized in that: In the step of constructing a multi-task collaborative learning framework, when the shared information fusion module calculates the inter-modal correlation matrix, the mutual information MI (F i ,F j ) is calculated by using the approximate calculation method of information entropy. The modal eigenvector is first discretized, and the discrete interval is determined adaptively according to the distribution range of the eigenvalue and the amount of data. Then, the information entropy and joint information entropy are calculated based on the discretized eigenvector, and then the mutual information and covariance Cov(F i ,F j ) is calculated using the sample weighted covariance calculation method, where the weight is determined based on the distance of the samples in the multimodal feature space. The closer the distance, the greater the weight.

4. The improved multimodal information fusion extraction method according to claim 1, characterized in that: In the multimodal information fusion and multi-task training steps, in the classification algorithm of the deep forest and multimodal feature hierarchical aggregation of the classification task module, the number of random forest classifiers is determined according to the number of categories and feature dimensions of the multimodal data, an initial number range is determined by analyzing the category distribution of the training data and evaluating the importance of features, and then the number of random forests that makes the classification accuracy highest is selected within the range through cross-validation, and when the multimodal features are hierarchically aggregated, the division of semantic levels is based on the domain knowledge and semantic annotation information of the multimodal data, and the feature importance is jointly determined by the information gain ratio and the correlation coefficient between the feature and the category label.

5. The improved multimodal information fusion extraction method according to claim 1, characterized in that: In the multimodal information fusion and multi-task training steps, in the regression model of the kernel ridge regression and multimodal feature adaptive combination of the regression task module, the determination of β and γ in the Laplace kernel function adopts a method combining grid search and Bayesian optimization. First, a better parameter range is determined by grid search, and then Bayesian optimization is used to find a parameter combination within the range that minimizes the mean square error of the model on the validation set. The regularization coefficient λ determined by the model weight is determined according to the noise level and feature redundancy of the data. An initial value range is determined by noise analysis and feature correlation analysis of the multimodal data, and is dynamically adjusted according to the performance of the model on the validation set during the training process.

6. The improved multimodal information fusion extraction method according to claim 1, characterized in that: In the multimodal information fusion and multi-task training steps, in the conditional variational autoencoder of the generation task module and the generation model guided by multimodal semantics, the feature transformation function g guided by multimodal semantics i (F fusion ) is realized by constructing a semantic neural network. The structure of the neural network is determined according to the complexity of the semantic structure of multimodal data, including a semantic encoding layer, a semantic conversion layer, and a semantic decoding layer. The semantic encoding layer maps the fusion features to the semantic space. The semantic conversion layer converts the semantic representation according to the semantic relationship of different modalities. The semantic decoding layer maps the converted semantic representation back to the feature space. Its weight coefficient θ i It is determined by analyzing the semantic similarity of multimodal training data and the target requirements of the generation task.

7. The improved multimodal information fusion extraction method according to claim 1, characterized in that: In the multi-task collaborative learning framework, the neural network layer shared between task modules adopts a sharing method of multimodal feature dynamic routing, that is, the shared layer dynamically determines the information transmission path according to the input multimodal fusion features. Suppose the input feature of the shared layer is F in , the output feature is F out , by constructing a routing decision network R, whose output is the routing vector r, then in is the input feature after feature transformation. The elements of the routing vector r are determined by the routing decision network according to the feature distribution of the multimodal fusion features, task requirements and current training status.

8. The improved multimodal information fusion extraction method according to claim 1, characterized in that: In the information extraction and application step, for the fusion extraction of new multimodal data, when some modalities are missing in the data, a fusion strategy of adversarial generative network and multimodal knowledge transfer is adopted. First, a multimodal knowledge extraction network is trained using existing modal data. The network extracts the multimodal knowledge representation in the existing modal data through an autoencoder structure, and then inputs it into the adversarial generative network as prior knowledge. The generator of the adversarial generative network generates approximate data of the missing modality based on the existing modality and multimodal knowledge. The discriminator network is used to discriminate the difference between the generated data and the real data. The generator network is continuously optimized through adversarial training to achieve the best compatibility and consistency between the reconstructed modal data and the original multimodal data in the fusion process.

9. The improved multimodal information fusion extraction method according to claim 1, characterized in that: In the step of constructing a multi-task collaborative learning framework, the shared information fusion module performs feature purification on each modal feature vector before performing modal correlation-driven fusion, and adopts a joint processing method of principal component analysis and outlier detection. The principal component analysis is first used to remove redundant components in each modal feature vector, and then the outlier detection algorithm of the local outlier factor is used to remove the outliers in the feature vector.

10. An improved multimodal information fusion extraction system according to any one of claims 1 to 8, characterized in that: The system includes a data acquisition and preprocessing module, a multi-task collaborative learning framework building module, a multi-modal information fusion and multi-task training module, and an information extraction and application module; Data acquisition and preprocessing module: collects data of multiple modalities, and preprocesses each modal data to obtain the initial feature representation of each modality. For image modal data, a feature extraction algorithm combining multi-scale texture structure tensor and deep convolutional neural network feature extraction layer is used. The multi-scale texture structure tensor calculates the texture direction and intensity information of the image at different scales. The deep convolutional neural network feature extraction layer adopts a specially designed convolution kernel layout. The convolution kernel is non-uniformly distributed in the spatial dimension and the distribution density is determined according to the prior statistical law of the image texture. For text modal data, , a feature extraction method based on semantic role labeling and dynamic weighting of word vectors is adopted. The semantic role weight of each word in the text is determined by semantic role labeling. The dynamic weighting of word vectors is based on the syntactic structure and semantic dependency of the text, and a dynamic weighting matrix is ​​constructed to weight the word vectors. For audio modal data, a feature extraction operation of harmonic structure analysis and time-frequency domain feature fusion is performed. The audio signal is first subjected to harmonic structure analysis to extract the frequency and amplitude parameters of the fundamental wave and harmonic wave, and then the frequency and amplitude parameters are fused with the short-time energy and zero-crossing rate characteristics of the audio signal in the time-frequency domain to obtain a comprehensive feature representation of the audio modality; Construct a multi-task collaborative learning framework module: Establish a multi-task collaborative learning framework including a classification task module, a regression task module and a generation task module, and the framework is provided with a shared information fusion module for receiving the initial feature representation of each modality. The shared information fusion module adopts a fusion algorithm driven by modality correlation. The feature vectors from different modalities are F1, F2, …, F n , calculate the correlation matrix C between the modes, where the element c ij represents the degree of association between the i-th mode and the j-th mode, which calculates the joint measure of mutual information and covariance between modal eigenvectors, i.e. c ij =α×MI(F i ,F j )+(1-α)×Cov(F i ,F j ), where MI(F i ,F j ) is F i With F j The mutual information of Cov(F i ,F j 0 is their covariance, α is the adjustment coefficient, and the fusion weight vector w is calculated according to the correlation matrix. The calculation formula is: The fused feature vector Multimodal information fusion and multi-task training module: Use the shared information fusion module to fuse multimodal data to obtain fused feature representation, write the fused features into different task modules for training, and optimize the parameters of the shared information fusion module and each task module through multi-task collaborative training to explore richer and more universal feature representations in multimodal data. During the training process, the classification task module adopts a classification algorithm that combines deep forest and multimodal feature hierarchical aggregation to construct multiple random forest classifiers. In each classifier, multimodal features are hierarchically aggregated according to their semantic hierarchy and feature importance. The distance between the cluster center and each modal feature is used as the basis for weight allocation. The output of the classifier is combined by weighted voting to obtain the final classification result. The regression task module uses a regression model that combines kernel ridge regression with multimodal feature adaptive combination. The kernel function uses the Laplace kernel function of the geometric structure of the multimodal feature space. where f i (x) and f i (y) are the values ​​of samples x and y on the i-th modal feature, is the variance of the i-th modal feature, β and γ are adjustable parameters, and the weight of the model is calculated by minimizing the regularized loss function of the residual OK, where y i is the true value, is the predicted value, m is the number of samples, λ is the regularization coefficient, and the generation task module adopts the generation model guided by conditional variational autoencoder and multimodal semantics. The encoder part maps the fusion feature to the base variable space through the transformation formula: Where μ and σ are the mean and standard deviation vectors learned through the neural network, ⊙ represents element-by-element multiplication, ∈ is a standard normal distribution random vector, and g i (F fusion ) is the feature transformation function guided by the i-th modal semantics, θ i is the corresponding weight coefficient, which is determined by analyzing the semantic structure of the multimodal training data. The decoder part generates new sample features according to the latent variable z, and each task module shares some neural network layer parameters during the training process; Information extraction and application module: Use the trained system to fuse and extract new multimodal data and apply it to classification, regression or generation tasks.

Images