Audio-visual emotion generation method based on neural anatomy alignment

Abstract

The invention belongs to the field of emotion generation simulation, and provides an audiovisual emotion generation method for neuroanatomy alignment, which comprises the following steps: step 1, a visual module simulates a brain visual cortex function and processes visual emotion data; 2, simulating a brain auditory cortex function by an auditory module, and processing audio emotion data; step 3, the fusion module simulates an audio-visual fusion function of anterior temporal superior gyrus areas on both sides of the brain, integrates output signals of the visual module and the auditory module, and generates a comprehensive feature vector; 4, the BEL module simulates an emotional interaction mechanism of the brain amygdala kernel and orbital frontal cortex, and bionic emotion learning and generation are completed based on the comprehensive feature vector; 5, comparing the EPP value of the generated data with the EPP value of the original data, and verifying the interpretability of emotion generation; according to the method, brain emotion learning and generation bionic mechanisms are improved through an integrated modular component of neuroanatomy alignment, and the interpretability of an audiovisual emotion generation model is improved.

Description

Technical Field

[0001] The present invention belongs to the field of emotion generation simulation, and in particular relates to a neuroanatomically aligned audiovisual emotion generation method. Background Art

[0002] Emotion is one of the brain's higher-order functions and fundamental to our ability to communicate, adapt, and survive with the world and others. Emotion plays an indispensable role in human communication. Emotional capacity is a key hallmark of human intelligence. Affective computing is a hot research topic in the field of artificial intelligence, encompassing interdisciplinary research across computer science, neuroscience, psychology, and mathematics. Leveraging the unique storage, analysis, and computational capabilities of computers, it explores the characteristics and secrets associated with physiological responses to human emotions, including facial expressions, heart rate, galvanic skin response, and temperature. Traditional emotion generation methods in the field of affective computing rely critically on deep learning technology and large-scale emotion datasets. Standardizing large-scale emotion datasets is difficult and expensive. Traditional deep learning models often employ fixed, complex network architectures. This is particularly true when processing large amounts of video and audio data. Due to the complex structure of deep learning models, their operational mechanisms are often difficult to understand in a black box fashion. This makes replication difficult and limits the interpretability of this type of deep learning in specific application contexts.

[0003] Research methods based on neuroanatomical alignment offer new opportunities for the development of affective computing. By functionally aligning different neural regions and networks in the brain, we can decompose and refine the brain's different neural pathways, layers, and structures, enabling the brain to combine and fuse information from both visual and audio modalities. Brain-inspired affective computing, from a comprehensive research perspective, combines brain neuroanatomy, artificial intelligence, and affective computing, allowing affective computing to develop towards more interpretable brain-inspired affective computing.

[0004] The present invention aims to solve the problem that the reliability of the deep neural network model is difficult to explain in the current deep neural network generation process of emotions. Summary of the Invention

[0005] To solve the above technical problems, the present invention provides a neuroanatomically aligned audiovisual emotion generation method to solve the problems in the prior art. The technical solution adopted by the present invention is: A neuroanatomically aligned audiovisual emotion generation method, comprising the following steps: Step 1: extract visual emotion features from the original emotional video. The visual module simulates the function of the brain's visual cortex to process the visual emotion data. Step 2: Extract acoustic emotion features from the original emotional audio. The auditory module simulates the function of the brain's auditory cortex to process the audio emotion data. Step 3: The fusion module simulates the audiovisual fusion function of the bilateral anterior superior temporal gyrus of the brain, integrates the output signals of the visual module and the auditory module, and generates a comprehensive feature vector; Step 4: The BEL module simulates the emotional interaction mechanism between the amygdala and orbitofrontal cortex of the brain, and completes bionic emotion learning and generation based on the comprehensive feature vector; Step 5: Design the emotional positivity parameter EPP and verify the interpretability of emotion generation by comparing the EPP values ​​of generated data with the original data.

[0006] Furthermore, in step 1, processing the visual emotion data includes: The visual module is implemented using the improved CORnet-Z model. The improved CORnet-Z model uses convolutional blocks to build the main structure. Each convolutional block includes convolution, activation, and pooling for feature extraction and downsampling. Global average pooling is used to pool the entire feature map into one value to output the visual emotion feature as a sequence of feature vectors, which is used as the input signal of the fusion module in step 3.

[0007] Furthermore, in step 2, processing the audio emotion data includes: The auditory module is implemented through a spiking neural network, which includes an excitatory neural population and two inhibitory neural subpopulations. Different types of neuron models are defined, namely excitatory neurons, inhibitory neurons, and SOM inhibitory neurons. Provide different input currents to each neuron cluster to simulate the activity of neurons in the primary auditory cortex; construct a feature vector sequence based on the number of pulses of the three neurons, and output the acoustic emotion features as a feature vector sequence, which is used as the input signal of the fusion module in step 3.

[0008] Furthermore, step 3 includes the following steps: In step 3.1, the fusion module is implemented using a multimodal multilayer perceptron. The visual and audio feature vector sequences obtained by the visual module and the auditory module are combined and processed through a fully connected layer and a ReLU activation function to obtain the hidden layer feature vector. In step 3.2, the two features are fused and connected separately according to the attention weight after the attention mechanism is processed, and then the feature vectors of vision and audio are spliced ​​together to form a comprehensive feature vector.

[0009] Furthermore, step 4 includes the following steps: Step 4.1: The BEL module is based on the human brain emotion learning model and imitates the emotional communication process between the amygdala and the frontal cortex. By constructing a recursive neural network based on the human brain emotion learning model, the BEL module is used to process the comprehensive feature vector generated by the fusion module. ; Step 4.2: A sliding window time series construction method is used to capture dynamic emotion changes to ensure that the BEL module can adapt to the time-varying characteristics of emotions; In step 4.3, the activation layer uses reinforcement learning rules to learn the connection weight V of the perception node. The metric of the perception node A in the activation layer is: , is the adjustable connection weight value; the reinforcement learning rules of the activation layer are as follows: ; in, Represents the connection weight The adjustment amount, is the learning rate, Represents the comprehensive feature vector from the fusion module, Rew is the reinforcement signal, Represents the total output of all sensor nodes; To find the upper limit of the sum, Represents the summed output of all sensor nodes A; In step 4.4, the suppression layer receives the input from the fusion module and the actual and expected reinforcement information from the activation layer. The suppression node O of the suppression layer is calculated as follows: , where U is an adjustable connection weight, and the measure of suppressing node O is as follows: ; in, Represents the adjustment amount of the connection weight U, is the learning rate, Represents the comprehensive feature vector from the fusion module, Rew is the reinforcement signal, is the current output The total difference between the expected reinforcement signal Rew; For each output node, the connection weight U is adjusted based on the current sensor input. and the total difference between all output nodes and the expected reinforcement signal Rew; ; Where A is the output of the perception node and O is the output of the inhibition node.

[0010] Furthermore, step 5 includes the following steps: Step 5.1: Design the emotional positivity parameter EPP of the emotional evaluation index, which is expressed by the following formula: ; Among them, the variable 、 、 、 and Representing the weights of fear, sadness, anger, calmness and happiness respectively. 、 、 、 and is the corresponding emotion score; when the value of the emotional positivity parameter EPP is 0, it corresponds to the negative side of emotion, and when the value of the emotional positivity parameter EPP is 1, it corresponds to the positive side of emotion; Step 5.2, calculate the emotional positivity parameter EPP of the true emotion of the sample data; Step 5.3, generate the EPP corresponding to the sample data through the trained emotion generation model; In step 5.4, the performance and interpretability of the emotion generation model are verified by comparing the average Euclidean distance between the real emotion EPP and the generated EPP.

[0011] The present invention has the following beneficial effects: This invention aims to address a major issue with current deep neural network emotion generation processes: the difficulty in interpreting the reliability of deep neural network models. Therefore, this invention proposes a neuroanatomically aligned audiovisual emotion generation method that improves upon the traditional brain emotion learning (BEL) model, thereby enabling a more interpretable emotion learning and generative biomimetic simulation process. The visual module simulates the visual cortex of the brain to extract visual emotion features; the auditory module simulates the auditory cortex of the brain to extract auditory emotion features; the fusion module simulates the audiovisual fusion mechanism to optimize the fusion of visual and auditory modal emotion features; and the BEL module simulates the emotion mechanism in the brain's short emotion pathway. This approach improves the brain's emotion learning and generative biomimetic mechanisms through neuroanatomically aligned integrated modular components, enhancing the interpretability of the audiovisual emotion generation model. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 It is a schematic diagram of the process of the present invention; Figure 2 Schematic diagram of the interpretable emotion generation model framework of the present invention; Figure 3 This is a similarity comparison diagram between the EPP generated by the present invention and the real EPP. In the figure, (a) is a comparison between the real EPP and the EPP generated by the present invention containing Euclidean distance, and (b) is a comparison between the real EPP and the EPP generated by the present invention. DETAILED DESCRIPTION

[0013] The following is a combination of the embodiments of the present invention Figure 1-Figure 3 , the technical solutions in the embodiments of the present invention are described clearly and completely. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. Unless otherwise specified, the technical means used in the embodiments are conventional means well known to those skilled in the art.

[0014] A neuroanatomically aligned audiovisual emotion generation method, comprising the following steps: Step 1: extract visual emotion features from the original emotional video. The visual module simulates the function of the brain's visual cortex to process the visual emotion data. Step 2: Extract acoustic emotion features from the original emotional audio. The auditory module simulates the function of the brain's auditory cortex to process the audio emotion data. Step 3: The fusion module simulates the audiovisual fusion function of the bilateral anterior superior temporal gyrus of the brain, integrates the output signals of the visual module and the auditory module, and generates a comprehensive feature vector; Step 4: The BEL module simulates the emotional interaction mechanism between the amygdala and orbitofrontal cortex of the brain, and completes bionic emotion learning and generation based on the comprehensive feature vector; Step 5: Design the Emotional Positivity Parameter (EPP). Comparing the EPP values ​​of the generated data with the original data verifies the interpretability of the generated emotion. The EPP (Emotional Positivity Parameter) quantifies the strength of the "positive tendency" of emotion.

[0015] Specifically, the specific method steps of the present invention are as follows: Step 1: Extract visual emotion features from the original emotional video. The visual module simulates the function of the brain's visual cortex and processes the visual emotion data: like Figure 2 As shown, the convolutional layer extracts multiple layers of emotional feature maps from the input image. Convolution uses a large 7x7 convolution kernel to quickly reduce the input dimensionality, ensuring better capture of emotional features. Subsequent convolutional blocks use smaller 3x3 convolution kernels to refine the layers, extracting emotional features at a finer granularity and forming a hierarchical feature representation. This approach enables the model to effectively capture complex emotional patterns and fully leverage the advantages of convolutional neural networks. A max pooling layer is introduced to optimize computational efficiency. This further reduces the spatial size of the feature map, preserving important emotional information.

[0016] After convolution and pooling, the model further reduces the feature map using an adaptive average pooling layer, compressing each channel in the feature map into a single eigenvalue. The Flatten layer compresses these features into a one-dimensional vector to facilitate subsequent sentiment analysis and generation tasks. The Identity layer allows these features to be directly utilized, facilitating subsequent operations. During training, model weights are uniformly initialized using XAVIer to ensure training stability. Bias is initialized to zero to prevent imbalances in signal propagation.

[0017] Through effective dimensionality reduction, the visual processing module is able to better extract emotional features from visual data. This module feeds emotional videos into the improved CORnet-Z model. This model comprises four computational regions conceptually similar to dorsal visual areas V1, V2, V4, and IT, along with a linear classification decoder. The resulting neural populations in these visual areas are mapped to behavioral choices, enabling more precise emotion generation and expression.

[0018] Through this design, the CORnet series of models provides a strong technical foundation for affective computing and promotes further research and application in this field. p is a four-dimensional tensor with dimensions (n, c, h, w), where n is the batch size, c is the number of channels corresponding to the number of features, h is the height of inp, and w is the width of inp. A convolution operation is performed on the input feature tensor inp, processing each channel with kernel k and adding a bias term b. The ReLU activation function is then applied: ; Afterwards, the convolution i p Perform a maximum pooling operation and select the maximum value from each region as the pooling result: ; Tensors that will be flattened and combined Convert to a one-dimensional vector Then perform a linear transformation on the flattened eigenvector X to obtain the final eigenvector : ; Feature vector of the vision module It will be used as the input signal of the fusion module in step 3.

[0019] Step 2: Extract acoustic emotion features from the original emotional audio. The auditory module simulates the function of the brain's auditory cortex and processes the audio emotion data: like Figure 2As shown in the figure, in the model setting of the auditory cortex module, the neurons in the three types of neuronal groups are set as excitatory neurons (G-PYR) and inhibitory neurons (G-PV, GSOM). The neural dynamics of each neuronal group are represented by equations to simulate the electrical activity of various neurons. The NeuronGroup function is used to establish 400 pyramidal neurons, 200 PV neurons, and 200 SOM neurons in the model. Each neuron is set with a threshold condition (membrane potential exceeds 0.5V) and a reset condition (membrane potential is set to 0). The input voltage value is set to 0.6V for pyramidal neurons and PV neurons, and 0.65V for SOM neurons. By simulating the stimulation levels of various neurons under real physiological conditions, it is shown that the model meets the requirements under physiological conditions.

[0020] SpikeMonitor is used to monitor neuronal discharge activity, which can observe the discharge of each neuron group at different times, including M-PYR (PYR neuron monitoring), M-PV (PV neuron monitoring), and M-SOM (SOM neuron monitoring); run(1000*ms) is used to perform a 1000-millisecond simulation to observe the operating effects of different neuron groups when processing audio stimuli. By building a basic framework to conduct in-depth research on the mechanism of auditory information processing, after observing and studying the discharge activities of each neuron, the functional characteristics of different neurons in auditory signal processing are obtained. The function of this module is to input the acoustic features (such as tone, pitch, volume, beat and duration) obtained from the original emotional audio into the modified Col1-spikingSNN model, and then study the stimulation effects of these different acoustic features on neurons and the processing of neuronal discharges.

[0021] The model consists of an excitatory G-PYR (Generate Pyramid) cluster and two inhibitory neuron clusters, G-PV (Generate Parvalbumin) and G-OM (Generate Somatostatin). These neuron groups are clearly defined using different neuronal models. Each neuron can simulate the activity of neurons in the primary auditory cortex, which helps us understand the processing of emotional audio and analyze it from a new perspective. These modules in the model are the basis for describing the relationship between emotional audio and auditory neural activity. The dynamic model of the G-PYR in the model is described as follows: ; represents the membrane potential of G-PYR in excitatory neurons, is the input current, is the time constant. The dynamic model of G-PV can be described as: ; represents the membrane potential of the inhibitory neuron G-PV, is the input current, is a time constant. The dynamic model of G-SOM can be described as: ; represents the membrane potential of the inhibitory neuron G-SOM, is the input current, is the time constant. The emotional auditory features (pitch, pitch, volume, beat and duration) are mapped to the input currents of the three neurons.

[0022] ; The dynamic models of these neurons can simulate changes in neuronal membrane potential and record their respective activity states.

[0023] (4) By monitoring the activity of these neurons, the function of the auditory cortex can be feature extracted and analyzed. The spike activity of all neurons is monitored and the conversion process during the simulation time is ; Nea refers to the process of simulating and observing neuronal activity. It is a function that maps emotional auditory features to neuronal input current. This is an equation from emotional auditory features to neuronal activity, reflecting the complex response of the auditory cortex to musical features. The eigenvector of the auditory module It will serve as the input signal for the subsequent fusion module.

[0024] Step 3: The fusion module simulates the audiovisual fusion function of the bilateral anterior superior temporal gyrus of the brain, integrates the output signals of the visual module and the auditory module, and generates a comprehensive feature vector: like Figure 2 As shown in the figure, the fusion module is a multi-layer perceptron (MLP) architecture design that serves as a framework for effectively processing multimodal datasets that combine audio and visual features. This module primarily uses multimodal fusion to train the MLP model and achieve audio and visual emotion feature fusion.

[0025] Step 3.1, input feature vector and And after the operation of the fully connected layer and the ReLU activation function, the hidden layer feature vector is obtained and : ;

[0026] Where W music is the weight matrix b of the audio feature hidden layer music is the bias term of the audio feature hidden layer.

[0027] ; Where W video is the weight matrix b of the hidden layer of video features video is the bias term of the hidden layer of video features.

[0028] Step 3.2: After the attention mechanism is processed, the two features are fused and connected according to the attention weights. ; Where W music is the weight matrix b of the audio feature attention mechanism music is the bias term of the audio feature attention mechanism.

[0029] ; Where W video is the weight matrix b of the video feature attention mechanism video is the bias term of the video feature attention mechanism.

[0030] Then, the two features are fused and connected according to the attention weight: ; The feature vectors X of audio and video m1 and X a1 Combined into a comprehensive feature vector : ; This will be used as the input for the next BEL module, allowing for seamless integration with subsequent BEL modules implemented using RNN, thereby improving model performance.

[0031] Step 4: The BEL module simulates the emotional interaction mechanism between the amygdala and orbitofrontal cortex of the brain, and completes bionic emotion learning and generation based on the comprehensive feature vector: Step 4.1, as Figure 2 As shown in Figure 1, the BEL module is designed based on the human brain emotion learning model, imitating the emotional communication process between the amygdala and the frontal cortex. By constructing a recurrent neural network (RNN) based on the human brain emotion learning model, this module is used to process the comprehensive feature vector generated by the above fusion module. .

[0032] In step 4.2, a sliding window time series construction method is used to capture dynamic emotion changes to ensure that the BEL module can adapt to the time-varying characteristics of emotions.

[0033] In step 4.3, the activation layer uses reinforcement learning rules to learn the connection weights V of the perception nodes, ensuring that the output is changed according to the reward output by the model, so that it outputs towards the desired reinforcement level; the metric of the perception node A in the activation layer is: , is the adjustable connection weight value; the reinforcement learning rules of the activation layer are as follows: ; Represents the connection weight The adjustment amount, is the learning rate. Represents the comprehensive feature vector from the fusion module. Rew is the reinforcement signal (reward). To sum the upper limit; Represents the summed output of all sensor nodes A. This formula shows that for each sensor node, its connection weight Adjustment amount Based on current sensory input , the difference between the expected reinforcement signal Rew and the current model output.

[0034] In step 4.4, the inhibition layer receives input from the fusion module and the actual and expected reinforcement information from the activation layer. It compares the expected emotional learning reward and, if there is a mismatch, inhibits the learning function of the frontal cortex. The inhibition node O of the inhibition layer is calculated as follows: , where U is an adjustable connection weight. The metric for suppressing node O is as follows: ; Represents the adjustment amount of the connection weight U, is the learning rate, Represents the comprehensive feature vector from the fusion module. Rew is the reinforcement signal (reward). is the current output The total difference between the expected reinforcement signal Rew. This formula shows that for each output node, the adjustment of its connection weight U depends on the current sensory input and the total difference between all output nodes and the expected reinforcement signal Rew.

[0035] ; Where A is the output of the sensory node and O is the output of the inhibitory node. This formula represents the interaction mechanism between the amygdala and orbitofrontal cortex in the BEL model.

[0036] Step 5: Design the emotional positivity parameter EPP and verify the interpretability of emotion generation by comparing the EPP values ​​of generated data with the original data: Step 5.1 Design the sentiment evaluation indicator, the EPP parameter. The EPP is a standardized indicator for measuring the level of sentiment positivity. It is sampled from the sentiment analysis software SnowNLP. It calculates the sentiment probability of a text by analyzing the sentiment orientation of each word in the text. After performing a weighted mean calculation on the sample values, the values ​​are normalized. This normalization process converts the sentiment index value into a numerical representation between 0 and 1. A value of 0 corresponds to a negative sentiment, while a value of 1 corresponds to a positive sentiment.

[0037] ; variable 、 、 、 and Representing the weights of fear, sadness, anger, calmness and happiness respectively. 、 、 、 and is the corresponding sentiment score. By applying the above formula to calculate the sentiment positivity parameter EPP of each sample, the use of EPP enables the effective measurement and comparison of sentiment tendencies under different stimuli, thereby providing an objective and intuitive standard for sentiment evaluation.

[0038] Step 5.2, calculate the emotional positivity parameter EPP of the true emotion of the sample data; Step 5.3, generate the EPP corresponding to the sample data through the trained emotion generation model; Step 5.4, as Figure 3 As shown in Figure 3, the performance and interpretability of the emotion generation model are verified by comparing the average Euclidean distance between the real EPP and the generated EPP.

[0039] This invention aims to address a major issue with current deep neural network emotion generation processes: the difficulty in interpreting the reliability of deep neural network models. Therefore, this invention proposes a neuroanatomically aligned audiovisual emotion generation method that improves upon the traditional brain emotion learning (BEL) model, thereby enabling a more interpretable emotion learning and generative biomimetic simulation process. The visual module simulates the visual cortex of the brain to extract visual emotion features; the auditory module simulates the auditory cortex of the brain to extract auditory emotion features; the fusion module simulates the audiovisual fusion mechanism to optimize the fusion of visual and auditory modal emotion features; and the BEL module simulates the emotion mechanism in the brain's short emotion pathway. This approach improves the brain's emotion learning and generative biomimetic mechanisms through neuroanatomically aligned integrated modular components, enhancing the interpretability of the audiovisual emotion generation model.

[0040] Each module of the present invention corresponds to the real neural structure of the brain. By simulating the function of neural pathways, the model operation mechanism is consistent with the brain's emotion processing logic, providing a scientific basis for tracing emotion generation; the visual module uses an improved CORnet-Z to extract hierarchical features, and the auditory module simulates neuronal discharges through a pulse neural network. Combined with the fusion strategy of MLP and attention mechanism, it realizes dynamic weighted integration of audio-visual features and improves the integrity and accuracy of emotional features; the BEL module uses recursive neural networks and reinforcement learning rules, combined with sliding windows to capture time-varying features, so that the model can adaptively learn dynamic changes in emotions; the present invention also designs an emotion positivity parameter EPP, which compares the generated data with the original data by weighted mean and Euclidean distance, providing a quantitative verification standard for model performance and ensuring the interpretability and reliability of emotion generation.

[0041] The embodiments described above are merely descriptions of preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Without departing from the spirit of the present invention, various deformations, modifications, and substitutions made to the technical solutions of the present invention by ordinary technicians in this field should fall within the scope of protection determined by the claims of the present invention.

Claims

1. A neuroanatomically aligned audiovisual emotion generation method, characterized in that The following steps are involved: Step 1: extract visual emotion features from the original emotional video. The visual module simulates the function of the brain's visual cortex to process the visual emotion data. Step 2: Extract acoustic emotion features from the original emotional audio. The auditory module simulates the function of the brain's auditory cortex to process the audio emotion data. Step 3: The fusion module simulates the audiovisual fusion function of the bilateral anterior superior temporal gyrus of the brain, integrates the output signals of the visual module and the auditory module, and generates a comprehensive feature vector; Step 4: The BEL module simulates the emotional interaction mechanism between the amygdala and orbitofrontal cortex of the brain, and completes bionic emotion learning and generation based on the comprehensive feature vector; Step 5: Design the emotional positivity parameter EPP and verify the interpretability of emotion generation by comparing the EPP values ​​of generated data with the original data.

2. The method for generating audiovisual emotions based on neuroanatomical alignment according to claim 1, characterized in that: In step 1, processing the visual emotion data includes: The visual module is implemented using the improved CORnet-Z model. The improved CORnet-Z model uses convolutional blocks to build the main structure. Each convolutional block includes convolution, activation, and pooling for feature extraction and downsampling. Global average pooling is used to pool the entire feature map into one value to output the visual emotion feature as a sequence of feature vectors, which is used as the input signal of the fusion module in step 3.

3. The method for generating audiovisual emotions based on neuroanatomical alignment according to claim 1, wherein: In step 2, processing the audio emotion data includes: The auditory module is implemented through a spiking neural network, which includes an excitatory neural population and two inhibitory neural subpopulations. Different types of neuron models are defined, namely excitatory neurons, inhibitory neurons, and SOM inhibitory neurons. Provide different input currents to each neuron cluster to simulate the activity of neurons in the primary auditory cortex; construct a feature vector sequence based on the number of pulses of the three neurons, and output the acoustic emotion features as a feature vector sequence, which is used as the input signal of the fusion module in step 3.

4. The method for generating audiovisual emotions based on neuroanatomical alignment according to claim 1, wherein: Step 3 includes the following steps: In step 3.1, the fusion module is implemented using a multimodal multilayer perceptron. The visual and audio feature vector sequences obtained by the visual module and the auditory module are combined and processed through a fully connected layer and a ReLU activation function to obtain the hidden layer feature vector. In step 3.2, the two features are fused and connected separately according to the attention weight after the attention mechanism is processed, and then the feature vectors of vision and audio are spliced ​​together to form a comprehensive feature vector.

5. The method for generating audiovisual emotions based on neuroanatomical alignment according to claim 1, wherein: Step 4 includes the following steps: Step 4.1: The BEL module is based on the human brain emotion learning model and imitates the emotional communication process between the amygdala and the frontal cortex. By constructing a recursive neural network based on the human brain emotion learning model, the BEL module is used to process the comprehensive feature vector generated by the fusion module. ; Step 4.2: A sliding window time series construction method is used to capture dynamic emotion changes to ensure that the BEL module can adapt to the time-varying characteristics of emotions; In step 4.3, the activation layer uses reinforcement learning rules to learn the connection weight V of the perception node. The metric of the perception node A in the activation layer is: , is the adjustable connection weight value; the reinforcement learning rules of the activation layer are as follows: ； in, Represents the connection weight The adjustment amount, is the learning rate, Represents the comprehensive feature vector from the fusion module, Rew is the reinforcement signal, Represents the total output of all sensor nodes; To find the upper limit of the sum, Represents the summed output of all sensor nodes A; In step 4.4, the suppression layer receives the input from the fusion module and the actual and expected reinforcement information from the activation layer. The suppression node O of the suppression layer is calculated as follows: , where U is an adjustable connection weight, and the measure of suppressing node O is as follows: ； in, Represents the adjustment amount of the connection weight U, is the learning rate, Represents the comprehensive feature vector from the fusion module, Rew is the reinforcement signal, is the current output The total difference between the expected reinforcement signal Rew; For each output node, the connection weight U is adjusted based on the current sensor input. and the total difference between all output nodes and the expected reinforcement signal Rew; ； Where A is the output of the perception node and O is the output of the inhibition node.

6. The method for generating audiovisual emotions based on neuroanatomical alignment according to claim 1, wherein: Step 5 includes the following steps: Step 5.1: Design the emotional positivity parameter EPP of the emotional evaluation index, which is expressed by the following formula: ； Among them, the variable 、 、 、 and Representing the weights of fear, sadness, anger, calmness and happiness respectively. 、 、 、 and is the corresponding emotion score; when the value of the emotional positivity parameter EPP is 0, it corresponds to the negative side of emotion, and when the value of the emotional positivity parameter EPP is 1, it corresponds to the positive side of emotion; Step 5.2, calculate the emotional positivity parameter EPP of the true emotion of the sample data; Step 5.3, generate the EPP corresponding to the sample data through the trained emotion generation model; In step 5.4, the performance and interpretability of the emotion generation model are verified by comparing the average Euclidean distance between the real emotion EPP and the generated EPP.

Images