A multi-modal fusion video classification method and system based on brain-like feedback interaction

Abstract

The application discloses a kind of multi-modal fusion video classification method and system based on brain-like feedback interaction, method includes: video pre-processing, obtain visual information and auditory information in video;After feature extraction, input into the multi-modal fusion framework based on neural network architecture search to obtain fusion information representation after fusion;Fusion information representation, visual information and auditory information are input into feedback module based on feedback modulation effect generated by single sensory cortex after auditory and visual information integration of human brain in superior temporal sulcus, and the output multi-modal fusion visual information and multi-modal fusion auditory information are respectively through fully connected layer to obtain the confidence of each classification;The confidence of each classification is input into DS decision fusion module, and the final classification result is obtained.The application learns from the processing mode of human brain to perceive external environment, integrates multiple senses from various modal perception information, and effectively improves the accuracy of recognizing and classifying the expressions of people in the video.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology, specifically relating to a multimodal fusion video classification method and system based on brain-like feedback interaction. Background Technology

[0002] When faced with complex interactive scenarios, it is difficult to make accurate judgments based on information from only one modality (such as auditory or visual). For example, in the process of recognizing and classifying facial expressions in videos, information from a single modality will lead to problems such as low classification accuracy. Summary of the Invention

[0003] To address the shortcomings of existing technologies, this invention provides a multimodal fusion video classification method and system based on brain-like feedback interaction. By drawing on the way the human brain perceives the external environment, it perceives information from various modalities and achieves multisensory integration, effectively improving the accuracy of recognizing and classifying facial expressions in videos.

[0004] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0005] Firstly, a multimodal fusion video classification method based on brain-like feedback interaction is provided, comprising: receiving an input video and preprocessing it to obtain visual and auditory information from the video; extracting features from the visual and auditory information respectively, and then inputting them into a multimodal fusion framework based on neural network architecture search for fusion to obtain a fused information representation; inputting the fused information representation, visual information, and auditory information into a feedback module constructed based on the feedback modulation effect generated on a single sensory cortex after the human brain integrates visual and auditory information in the superior temporal sulcus, and outputting multimodal fused visual information and multimodal fused auditory information; passing the multimodal fused visual information and multimodal fused auditory information through fully connected layers to obtain the confidence of each classification; and inputting the confidence of each classification into a DS decision fusion module to obtain the final classification result.

[0006] Furthermore, the input video is received and preprocessed to obtain visual and auditory information from the video, including: segmenting the video, using the FFmpeg video processing package to extract the segmented video into image frames, using the face detection interface provided by Face++ to perform face detection on the image frames, locating facial key points, and cropping the images based on internal key points and contour key points to obtain visual information; performing short-time Fourier transform on the speech in the video, and adding information from the preceding and following frames through first-order and second-order difference operations before and after the features extracted from the frames to obtain auditory information with temporal continuity.

[0007] Furthermore, first-order and second-order difference operations are used to add information from the preceding and following frames before and after the features extracted from each frame. This is achieved through the following method:

[0008]

[0009] Where Q represents the order of the cepstral coefficients, K represents the time difference of the first derivative, and d t Let C represent the first-order difference value of the t-th frame. t The Mel-spectral coefficients representing the t-th frame are d t Substituting this into the equation yields the second-order difference value.

[0010] Furthermore, feature extraction is performed on visual and auditory information separately, including: visual information is extracted using ResNet18, and auditory information is extracted using ConvLSTM. The extracted features are then convolved through a one-dimensional temporal convolutional layer.

[0011]

[0012] in, This represents the input feature sequence, where k is the mode k. {V,A,F} The size of the convolution kernel, where d is the common dimension.

[0013] Furthermore, the basic search unit for audiovisual fusion in the multimodal fusion framework based on neural network architecture search includes five visual video modal output features (v1, v2, v3, v4, v5), three auditory audio modal output features (a1, a2, a3), and three nonlinear activation functions; among which, the three nonlinear activation functions are: Sigmoid, ReLU, and LeakyReLU.

[0014] Furthermore, a feedback module constructed based on the feedback modulation effect on a single sensory cortex generated by the integration of visual and auditory information in the superior temporal sulcus of the human brain includes two identical cross-modal interactors. The method for constructing the cross-modal interactors includes: defining a query matrix. Key matrix Value matrix in These are all weights; thus, the potential cross-modal adaptation from β to α

[0015]

[0016] Here, Y and Q have the same sequence length T, and it also represents the feature space of V; the scaling of softmax calculates a fractional matrix. Its (i,j)th term calculates the attention from the i-th time step in modality α to the j-th time step in modality β; therefore, the i-th time step of Y is a weighted summation of V, with the weights determined by the i-th row in softmax(); in the cross-modal attention calculation, a residual connection is added, and then injected into the feedforward sub-layer at another location; each cross-modal attention block is directly from the low-level feature sequence Z. β [0] The adaptation involves a location-fully connected feedforward network operating on the channel dimension, mapping the attention results at each location to a larger-dimensional feature space, then introducing non-linearity for filtering, and finally restoring it back to the original dimension.

[0017] Furthermore, all dimensions of each cross-modal attention block are fixed to d. {α,β,k,v} The cross-modal interactor includes a D-layer cross-modal attention block. Formally, it performs feedforward computation on layers i=1,...,D, and the specific computation method is as follows:

[0018]

[0019] Among them, f θ It is a position-feedforward sublayer parameterized by θ. Let LN denote the multi-head form of CM in the i-th layer, i.e., multi-head attention, and LN denotes layer normalization.

[0020] Furthermore, the cross-modal interactor is operated using a gene expression programming algorithm based on semi-supervised learning.

[0021] Furthermore, the final classification result is as follows:

[0022]

[0023] Where, m1(θ) i Evidence 1 is for proposition θ i The degree of trust, m1(θ) i )·m2(θ j () represents the evidence pair after evidence fusion for the proposition θ=θ i ∩θ j The level of trust, where θ is the sentiment category. It is an empty set; n is the total number of categories. K is the normalization factor, and K is the conflict factor, representing the overall conflict of the evidence, as shown in the following formula:

[0024]

[0025] Secondly, a multimodal fusion video classification system based on brain-like feedback interaction is provided, comprising: a data preprocessing module for receiving and preprocessing input videos to obtain visual and auditory information from the videos; a multimodal fusion module for extracting features from the visual and auditory information respectively, and then inputting them into a multimodal fusion framework based on neural network architecture search for fusion to obtain a fused information representation; a feedback module for inputting the fused information representation, visual information, and auditory information into a feedback module constructed based on the feedback modulation effect generated on the single sensory cortex after the human brain integrates visual and auditory information in the superior temporal sulcus, outputting multimodal fused visual information and multimodal fused auditory information; and a classification module for passing the multimodal fused visual information and multimodal fused auditory information through fully connected layers to obtain the confidence score of each classification; and a classification module for inputting the confidence score of each classification into a DS decision fusion module to obtain the final classification result.

[0026] Compared with the prior art, the beneficial effects achieved by the present invention are as follows:

[0027] (1) This invention extracts features from visual and auditory information separately, and then inputs them into a multimodal fusion framework based on neural network architecture search to obtain a fused information representation; the fused information representation, visual information, and auditory information are input into a feedback module constructed based on the feedback modulation effect generated on the single sensory cortex after the human brain integrates visual and auditory information in the superior temporal sulcus, and outputs multimodal fused visual information and multimodal fused auditory information; the multimodal fused visual information and multimodal fused auditory information are respectively passed through a fully connected layer to obtain the confidence of each classification; the confidence of each classification is input into the DS decision fusion module to obtain the final classification result, which effectively improves the accuracy of recognizing and classifying human expressions in videos;

[0028] (2) The multimodal fusion method at the decision level of this invention adopts the fusion method of uncertain reasoning algorithm - DS evidence theory. In view of the problem that this algorithm is prone to data conflict, which leads to conclusions that are completely contrary to common sense, a method based on Pearson correlation coefficient is proposed to solve the conflict paradox problem by obtaining the correlation between evidence bodies and correcting the evidence bodies. Attached Figure Description

[0029] Figure 1 This is a schematic diagram of the main process of a multimodal fusion video classification method based on brain-like feedback interaction provided in an embodiment of the present invention;

[0030] Figure 2 This is a schematic diagram of cross-modal attention in an embodiment of the present invention. Figure 1 ;

[0031] Figure 3 This is a schematic diagram of cross-modal attention in an embodiment of the present invention. Figure 2 ;

[0032] Figure 4 This is a schematic diagram of the feedback interaction model in an embodiment of the present invention. Detailed Implementation

[0033] The present invention will be further described below with reference to the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and should not be used to limit the scope of protection of the present invention.

[0034] Example 1:

[0035] like Figures 1-4 As shown, a multimodal fusion video classification method based on brain-like feedback interaction includes: receiving and preprocessing an input video to obtain visual and auditory information from the video; extracting features from the visual and auditory information respectively, and then inputting them into a multimodal fusion framework based on neural network architecture search to obtain a fused information representation; inputting the fused information representation, visual information, and auditory information into a feedback module constructed based on the feedback modulation effect generated on a single sensory cortex after the human brain integrates visual and auditory information in the superior temporal sulcus, and outputting multimodal fused visual information and multimodal fused auditory information; passing the multimodal fused visual information and multimodal fused auditory information through fully connected layers to obtain the confidence score of each classification; and inputting the confidence score of each classification into a DS decision fusion module to obtain the final classification result.

[0036] like Figure 1 As shown, for the input video, the visual image and auditory speech are first processed to adapt to the model's input. Then, the input is fed into a multimodal fusion framework based on neural network architecture search to obtain the optimal fusion architecture. Fusion is performed according to this architecture to obtain a fused information representation. Next, the fused information and the audiovisual unimodal information are respectively input into two identical cross-modal interactors. By constructing cross-modal interactors, the fused information is fed back to the unimodal information, thereby strengthening the unimodal information representation and increasing the model's coherence. Then, each is passed through a fully connected layer to obtain the confidence score for each classification. This output serves as the input for the DS evidence theory fusion algorithm, which, after passing through the DS decision fusion algorithm, yields the final classification result. Specifically, the process includes the following steps.

[0037] S1. Receive the input video and preprocess it to obtain visual and auditory information from the video.

[0038] S11. Image preprocessing: Due to the varying lengths and sizes of the original videos, video segmentation is necessary to standardize the network input size. Furthermore, the training process of deep neural networks requires a large amount of input data, necessitating data augmentation. Considering both points, this embodiment employs a sliding window method to segment the video, achieving both size standardization and dataset expansion. The FFmpeg video processing package is used to extract image frames from the segmented video. Face detection is performed on the images using the face detection interface provided by Face++, locating facial key points. Based on internal and contour key points, the images are cropped into 224×224 pixel images.

[0039] S12. Preprocess the speech. To preserve temporal information, a short-time Fourier transform is first performed on the speech features. To preserve temporal continuity, first-order and second-order difference operations are used to add information from the preceding and following frames before and after the features extracted in each frame. This is achieved through the following method:

[0040]

[0041] Where Q represents the order of the cepstral coefficients, K represents the time difference of the first derivative, and d t Let d represent the first-order difference value of the t-th frame. t Let C represent the first-order difference value of the t-th frame. t The Mel-spectral coefficients representing the t-th frame are d t Substituting this into the equation yields the second-order difference value.

[0042] S2. Feature extraction is performed on visual and auditory information respectively, and then input into a multimodal fusion framework based on neural network architecture search to obtain a fused information representation; the fused information representation, visual information, and auditory information are input into a feedback module constructed based on the feedback modulation effect generated on the single sensory cortex after the human brain integrates visual and auditory information in the superior temporal sulcus, and the multimodal fused visual information and multimodal fused auditory information are output; the multimodal fused visual information and multimodal fused auditory information are respectively passed through a fully connected layer to obtain the confidence of each classification.

[0043] S21. Feature extraction is performed on visual and auditory information separately, and then the data is input into a multimodal fusion framework based on neural network architecture search for fusion to obtain a fused information representation. The preprocessed data is output, where visual information is extracted using ResNet18 and auditory information is extracted using ConvLSTM. To ensure that each element in the input sequence has sufficient knowledge of its neighboring elements, the extracted features are convolved through a one-dimensional temporal convolutional layer.

[0044]

[0045] in, This represents the input feature sequence, where k is the mode k. {V,A,F} The size of the convolution kernel is denoted by d, where d is the common dimension. The sequence after convolution not only contains local information of the sequence, but also projects features from different modalities onto the same dimension d, which is beneficial for subsequent dot product operations.

[0046] The basic search unit for audiovisual fusion in the neural network-based multimodal fusion framework includes five visual video modal output features (v1, v2, v3, v4, v5), three auditory audio modal output features (a1, a2, a3), and three nonlinear activation functions: Sigmoid, ReLU, and LeakyReLU. These nonlinear activation functions are used to perform nonlinear processing on the fusion features and connect the audiovisual information output features.

[0047] S22. In order to make the sequence output from S21 carry time information, the position information is then embedded into sequence X. {V,A,F} middle:

[0048] Z [0] {V,A,F} =X {V,A,F} +PE(T {V,A,F} ,d) (3)

[0049] in, Calculate the embedding for each position index, Z [0] {V,A,F} These are low-level position-aware features of different modalities.

[0050] This embodiment encodes the positional information of a sequence of length T output from S21 using sin and cos functions, with the encoding frequency determined by the feature index. The sequence... The position embedding (PE) is defined as a matrix, and its calculation formula is as follows:

[0051]

[0052] in, d is the common dimension, and T is the sequence length.

[0053] Therefore, each feature dimension of the PE is a position value that presents a sinusoidal pattern. After calculation, the position embedding is directly added to the sequence so that X+PE encodes the position information of the elements at each time step.

[0054] S23. A feedback module, constructed based on the feedback modulation effect on a single sensory cortex after the human brain integrates visual and auditory information in the superior temporal sulcus, inputs fused information, visual information, and auditory information. It outputs multimodal fused visual information and multimodal fused auditory information. This includes two identical cross-modal interactors, which enable one modality to receive information from another. The input is the sequence output from S22, and its specific description is provided by receiving fused information (F) using instance visual information (V), i.e., F→V.

[0055] The construction of a cross-modal interactor includes the following process:

[0056] like Figure 3 As shown, define the query matrix. Key matrix Value matrix in, These are all weights. Thus, the potential cross-modal adaptation from β to α can be represented as a cross-modal...

[0057]

[0058] Here, Y and Q have the same sequence length T, and it also represents the feature space of V; the scaling of softmax calculates a fractional matrix. Its (i,j)th term calculates the attention from the i-th time step in mode α to the j-th time step in mode β; therefore, the i-th time step size of Y is a weighted summation of V, with the weights determined by the i-th row in softmax();

[0059] like Figure 4 As shown, to complete the construction of the cross-modal interactor, a residual connection is added to the cross-modal attention computation, and then injected into a feedforward sub-layer at another location. In this way, each cross-modal attention block directly receives input from the low-level feature sequence Z. β [0] Adaptation, independent of self-attention, allows the model to better preserve low-level information for each modality. The positionally fully connected feedforward network operates along the channel dimension, functioning similarly to a 1x1 convolution in a convolutional network. This maps the attention results at each location to a larger-dimensional feature space, introduces non-linear filtering, and finally restores the original dimension.

[0060] Fix all dimensions of each cross-modal attention block to d. {α,β,k,v} Cross-modal interactors are all composed of D-layer cross-modal attention blocks. Formally, they perform feedforward calculations on layers i=1,...,D, as shown in the following formula:

[0061]

[0062] Among them, f θ It is a position-feedforward sublayer parameterized by θ. Let LN denote the multi-head form of CM in the i-th layer, i.e., multi-head attention, and LN denotes layer normalization.

[0063] In this process, each modality can update its sequence using external information learned from the multi-head cross-modal attention block. Within the cross-modal attention block, information from the source modality is transformed into a distinct set of key-value pairs to facilitate interaction with the target modality. Therefore, the cross-modal interactor can learn meaningful elements associated across modalities, and the final model relies on feedback interaction based on the cross-modal interactor. For both audiovisual and visual modalities, two cross-modal interactors are required.

[0064] S24. The feedback reinforcement information obtained from audiovisual information through a cross-modal interactor can be represented as: Then, a self-attention converter is used to collect temporal information for prediction. Finally, the last element in the sequence model is extracted and the classification task is completed through a fully connected layer.

[0065] To reduce resource consumption and improve model accuracy, a gene expression programming algorithm based on semi-supervised learning is adopted, including the following process:

[0066] (1) Initialize the population: Randomly generate a set of individuals with and without labels as the initial population P.

[0067] (2) Calculating individual fitness: First, a labeled dataset is generated, i.e., a subset of individuals from the initial population P are selected for training to calculate their accuracy, and these individuals are labeled with their accuracy. To facilitate prediction, the controller is pre-trained using the generated new dataset. After pre-training, the accuracy of the remaining individuals is calculated using the controller, generating an unlabeled dataset. Subsequently, all datasets are merged and trained together using the controller to obtain the accuracy of all individuals. The accuracy obtained for each individual is its fitness in the fitness space.

[0068] (3) Elite Selection: Generally, individuals with high fitness are selected to remain. However, due to the use of a semi-supervised learning method, the fitness of individuals in the population depends entirely on the performance of the controller. If the controller cannot be fully trained, the prediction performance will decrease, leading to a reduction in the reliability of the obtained accuracy. If only individuals with high fitness are retained, the subsequent reproduction process may not proceed smoothly, easily resulting in new individuals with lower performance. In order to reduce the dependence of individual fitness on the controller and generate new individuals with better performance, this chapter uses a confidence-driven approach for elite selection. The calculation formula is shown below:

[0069] F p(i) =y i T / ∑ j y j T (7)

[0070] Among them, F p This is the fitness function, where y represents the controller's prediction accuracy and T represents the confidence level. The formula for calculating T is as follows:

[0071]

[0072] Where T0 represents the initial confidence level, T max denoted by , where i represents the number of iterations for model optimization, and a represents the confidence increase rate.

[0073] (4) Generating new population individuals: Due to the complexity of the gene propagation process, this embodiment uses a simpler controller to generate a new population. The controller consists of an encoder / decoder and a performance predictor. The specific process is as follows: First, the parent elites are encoded into a vector expression by the encoder. This expression is then used as the input of the performance predictor to calculate the predicted fitness of the parent elites. Finally, population individuals with better performance are generated along the gradient ascent direction.

[0074] As can be seen from the algorithm flow steps, gene expression programming based on semi-supervised learning simplifies the complex gene propagation process into a controller evolution process, reducing algorithm complexity and architecture search time, while still ensuring the accuracy of the algorithm.

[0075] S3. Input the confidence level of each classification into the DS decision fusion module to obtain the final classification result.

[0076] S31. Input the confidence scores for each category obtained from S2. Optimal initial weights are obtained through training, and different priority levels are assigned to audiovisual evidence based on different datasets. By default, the weights of the evidence bodies are the same; the initial weights are used to refine the evidence bodies.

[0077] m i ′=ω i ·m i (i = 1, 2, ..., n) (9)

[0078] Where, ω i It is the weight of evidence i, m i For the basic probability assignment of evidence i, m i ω' is the modified basic probability assignment function, and n is the number of evidence bodies. By default, ω = 1 / n.

[0079] S32. Input the corrected evidence obtained in S31, use the Pearson correlation coefficient to calculate the correlation between the audiovisual information and the evidence, and obtain the confidence level cred(m) between them. i ′).

[0080]

[0081] Where E is the mathematical expectation, s ij The correlation between evidence bodies i and j.

[0082] In the case of multiple evidence bodies, the correlation of the evidence bodies is represented by a correlation matrix composed of correlation coefficients.

[0083]

[0084] The Pearson correlation coefficient ranges from -1 to 1. However, the evidence theory requires non-negativity, so negative values ​​in the correlation coefficient are corrected to 0.0001 in this chapter.

[0085] Define evidence m i The support rate is The confidence level is cred(m) i ′).

[0086]

[0087] Among them, sup(m i ′) is evidence m i Support level, cred(m) i ) is evidence m i The confidence level.

[0088] S34. Obtain intermediate evidence of visual and auditory evidence based on the confidence level obtained in S33. Calculate the distance from the audiovisual evidence to the intermediate evidence, and obtain the credibility F[dm] of the audiovisual evidence. i ′).

[0089] (1) Calculate intermediate evidence

[0090]

[0091] Where, m i ' represents the corrected basic probability assignment, cred(m) i ′) is evidence m i The confidence level.

[0092] (2) Calculate the credibility of the basic probability assignments of the evidence. The distance dm between each evidence's basic probability assignment and the intermediate evidence's basic probability assignment is calculated using Euclidean distance. i ′:

[0093]

[0094] Where, m i ′(I) represents the degree of confidence that the revised evidence i has in proposition I. This is the intermediate evidence for proposition I, and N is the number of propositions.

[0095] (3) Using the distance dm of the basic probability assignment i 'Calculate the confidence level F[dm] of the basic probability assignment i ′]:

[0096]

[0097] in, This serves as intermediate evidence.

[0098] S35. Using the credibility obtained in S34, allocate the conflicts and revise the audiovisual evidence accordingly:

[0099]

[0100] Where, ω i It is the weight of evidence i, F[dm i ′] represents the confidence level of the basic probability assignment.

[0101] S36. The modified audiovisual information obtained in S35 is fused using the synthesis formula to obtain the optimal fusion result, which is the final classification result:

[0102]

[0103] Where, m1(θ) i Evidence 1 is for proposition θ i The degree of trust, m1(θ) i )·m2(θ j () represents the evidence pair after evidence fusion for the proposition θ=θ i ∩θ j The level of trust, where θ is the sentiment category. It is an empty set; n is the total number of categories. K is the normalization factor, and K is the conflict factor, representing the overall conflict of the evidence, as shown in the following formula:

[0104]

[0105] This invention proposes a feedback fusion framework based on neural network architecture search for a multimodal fusion method at the data level, which can automatically generate fusion architectures based on different datasets. To better balance search time, fusion layer combination methods, and performance, a gene expression programming algorithm based on semi-supervised learning is proposed. Inspired by the integration effect of the superior temporal sulcus in the human brain, a feedback model based on attention mechanisms is proposed. By constructing cross-modal attention blocks and designing cross-modal interactors, single-modal information can receive multimodal information, and the fused information is fed back to the single-modal information to strengthen the representation of single-modal information and enhance the connection between modalities. Experiments on the audiovisual datasets eNTERFACE'05 and AFEW9.0 show that the final recognition accuracy can reach 92.30% and 59.97%, respectively.

[0106] This invention employs the fusion method of the uncertain reasoning algorithm—DS evidence theory—in its multimodal fusion method at the decision-making level. To address the problem that this algorithm is prone to data conflicts, leading to conclusions that completely contradict common sense, a method based on Pearson correlation coefficient is proposed. This method resolves the conflict paradox by obtaining the correlation between evidence bodies and correcting the evidence bodies.

[0107] Inspired by the parallel integration model of human brain's visual and auditory perception, this invention proposes a hybrid multimodal fusion model. It utilizes feedback-enhanced single-modal information for decision-making, while visual and auditory decision information is fused using an improved DS evidence theory fusion method. Ultimately, the model achieves accuracies of 94.55% and 61.15% on the eNTERFACE'05 and AFEW9.0 datasets, respectively, demonstrating excellent recognition performance.

[0108] Example 2:

[0109] Based on the multimodal fusion video classification method based on brain-like feedback interaction described in Embodiment 1, this embodiment provides a multimodal fusion video classification system based on brain-like feedback interaction, including:

[0110] The data preprocessing module is used to receive the input video and perform preprocessing to obtain visual and auditory information from the video;

[0111] The multimodal fusion module is used to extract features from visual and auditory information separately, and then input them into a multimodal fusion framework based on neural network architecture search to fuse them and obtain a fused information representation;

[0112] It is also used to input fused information representation, visual information and auditory information into a feedback module based on the feedback modulation effect generated on a single sensory cortex after the human brain integrates visual and auditory information in the superior temporal sulcus, and output multimodal fused visual information and multimodal fused auditory information.

[0113] It is also used to pass the visual information and auditory information fused from multimodal fusion through fully connected layers to obtain the confidence scores for each classification.

[0114] The classification module is used to input the confidence level of each classification into the DS decision fusion module to obtain the final classification result.

[0115] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the technical principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A multimodal fusion video classification method based on brain-like feedback interaction, characterized in that, include: Receive the input video and preprocess it to obtain visual and auditory information from the video; Visual and auditory information are extracted separately, and then input into a multimodal fusion framework based on neural network architecture search to obtain a fused information representation; The feedback module, which integrates visual and auditory information, is constructed based on the feedback modulation effect on the single sensory cortex after the human brain integrates visual and auditory information in the superior temporal sulcus. The output is multimodal fused visual information and multimodal fused auditory information. The visual information and auditory information fused from multimodal fusion are respectively passed through a fully connected layer to obtain the confidence score for each classification. The confidence score for each category is input into the DS decision fusion module to obtain the final classification result; The feedback module, constructed based on the feedback modulation effect on a single sensory cortex generated by the integration of visual and auditory information in the superior temporal sulcus of the human brain, includes two identical cross-modal interactors. The method for constructing the cross-modal interactors includes: Define query matrix Key matrix Value matrix ,in , , All are weights; thus, from arrive Potential cross-modal adaptation , （5） in, and Having the same sequence length It also indicates that in In the feature space; The scaling computed a fractional matrix , its first The term calculated the modality The i-th time step to the mode Attention at the j-th time step; therefore, The i-th time step is The weighted summation, with weights from The i-th row in the equation determines the result; In cross-modal attention computation, residual connections are added, and then injected into a feedforward sub-layer at another location; each cross-modal attention block is directly fed from the low-level feature sequence. The adaptation involves a location-fully connected feedforward network operating on the channel dimension, mapping the attention results at each location to a larger-dimensional feature space, then introducing non-linearity for filtering, and finally restoring it back to the original dimension.

2. The multimodal fusion video classification method based on brain-like feedback interaction according to claim 1, characterized in that, Receive the input video and preprocess it to obtain visual and auditory information from the video, including: The video is segmented, and the FFmpeg video processing package is used to extract the segmented video into image frames. Face detection is performed on the image frames using the face detection interface provided by Face++ to locate facial key points. The images are then cropped based on internal key points and contour key points to obtain visual information. Short-time Fourier transform is performed on the speech in the video, and first-order and second-order difference operations are used to add information from the preceding and following frames before and after the features extracted in each frame, so as to obtain auditory information with temporal continuity.

3. The multimodal fusion video classification method based on brain-like feedback interaction according to claim 2, characterized in that, Information from the preceding and following frames is added before and after the features extracted by frame segmentation through first-order and second-order difference operations, which is achieved through the following method: （1） Where Q represents the order of the cepstral coefficients, and K represents the time difference of the first derivative. Indicates the first The first-order difference of the frame, The Mel-spectral coefficients representing the t-th frame will be... Substituting this into the equation yields the second-order difference value.

4. The multimodal fusion video classification method based on brain-like feedback interaction according to claim 1, characterized in that, Feature extraction is performed separately for visual and auditory information, including: visual information is extracted using ResNet18, and auditory information is extracted using ConvLSTM. The extracted features are then convolved through a one-dimensional temporal convolutional layer. （2） in, This represents the input feature sequence, where k is the mode. The size of the convolution kernel, where d is the common dimension.

5. The multimodal fusion video classification method based on brain-like feedback interaction according to claim 1, characterized in that, In a multimodal fusion framework based on neural network architecture, the basic search unit for audiovisual fusion includes five visual and video modal output features. Three auditory audio modal output features And three non-linear activation functions; the three non-linear activation functions are: Sigmoid, ReLU, and LeakyReLU.

6. The multimodal fusion video classification method based on brain-like feedback interaction according to claim 1, characterized in that, Fix all dimensions of each cross-modal attention block as follows: ; The cross-modal interactor includes a D-layer cross-modal attention block; formally, it is a... The layer performs feedforward calculations, and the specific calculation method is as follows: （6） in, It is by Parameterized position feedforward sublayer Let LN denote the multi-head form of CM in the i-th layer, i.e., multi-head attention, and LN denotes layer normalization.

7. The multimodal fusion video classification method based on brain-like feedback interaction according to claim 6, characterized in that, The cross-modal interactor is operated using a gene expression programming algorithm based on semi-supervised learning.

8. The multimodal fusion video classification method based on brain-like feedback interaction according to claim 1, characterized in that, The final classification result is: （17） in, For evidence 1, the proposition The level of trust, For the evidence-based proposition after evidence fusion The level of trust, For sentiment classification, It is an empty set; The total number of categories, As the normalization factor, The conflict factor represents the overall conflict within the body of evidence, as shown in the following formula: （18）。 9. A multimodal fusion video classification system based on brain-like feedback interaction, characterized in that, include: The data preprocessing module is used to receive the input video and perform preprocessing to obtain visual and auditory information from the video; The multimodal fusion module is used to extract features from visual and auditory information separately, and then input them into a multimodal fusion framework based on neural network architecture search to fuse them and obtain a fused information representation; It is also used to input fused information representation, visual information and auditory information into a feedback module based on the feedback modulation effect generated on a single sensory cortex after the human brain integrates visual and auditory information in the superior temporal sulcus, and output multimodal fused visual information and multimodal fused auditory information. It is also used to pass the visual information and auditory information fused from multimodal fusion through fully connected layers to obtain the confidence scores for each classification. The classification module is used to input the confidence score of each classification into the DS decision fusion module to obtain the final classification result; The feedback module, constructed based on the feedback modulation effect on a single sensory cortex generated by the integration of visual and auditory information in the superior temporal sulcus of the human brain, includes two identical cross-modal interactors. The method for constructing the cross-modal interactors includes: Define query matrix Key matrix Value matrix ,in , , All are weights; thus, from arrive Potential cross-modal adaptation , （5） in, and Having the same sequence length It also indicates that in In the feature space; The scaling computed a fractional matrix , its first The term calculated the modality The i-th time step to the mode Attention at the j-th time step; therefore, The i-th time step is The weighted summation, with weights from The i-th row in the equation determines the result; In cross-modal attention computation, residual connections are added, and then injected into a feedforward sub-layer at another location; each cross-modal attention block is directly fed from the low-level feature sequence. The adaptation involves a location-fully connected feedforward network operating on the channel dimension, mapping the attention results at each location to a larger-dimensional feature space, then introducing non-linearity for filtering, and finally restoring it back to the original dimension.

Images