A cross-modal collaborative perception and double-graph spectrum diffusion emotion recognition method

Abstract

The application discloses a kind of cross-modal collaborative perception and double graph spectrum diffusion emotion recognition method, it is related to multi-modal information processing, graph neural network and affective computing technical field.For the problem that existing interpolation method is easy to destroy dialogue graph topological structure, produce artifact and reduce reasoning continuity when mode is missing, the application first constructs cross-modal collaborative anchor point using un-missing mode, provides prior initialization for missing mode;Again, based on speaker identity dependence and time sequence context dependence, construct double graph spectrum, carry out spectral decomposition to adjacency matrix, constrain the topological structure represented by orthogonal eigenvector unchanged in spectral eigenvalue space during the process of adding noise and denoising;Then, according to the information entropy of reconstructed node feature, generate dynamic gating weight, adaptively fuse double graph features, and obtain emotion classification results through spectral graph convolution.The application can maintain the robustness and accuracy of multi-modal emotion recognition under the condition of high missing rate and noise interference.

Description

Technical Field

[0001] This invention relates to the fields of multimodal information processing, graph neural networks and emotion computing, specifically to an emotion recognition method based on cross-modal collaborative perception and dual-graph spectrum diffusion. Background Technology

[0002] Multimodal emotion recognition aims to fuse heterogeneous signals such as text, acoustics, and vision to identify the emotional state of each utterance in a dialogue. Existing multimodal emotion recognition methods typically assume that the input modalities are relatively complete, or perform feature fusion only under slightly noisy conditions. However, in real-world applications, due to sensor failures, network packet loss, privacy occlusion, changes in lighting, and noise interference with speech, one or more modalities are often missing at certain utterance locations.

[0003] To address the problem of missing modalities, existing methods typically employ zero-padding, mean-padding, discarding missing moments, or feature imputation based on generative models. Zero-padding and discarding directly cause semantic breaks in the dialogue context; while ordinary generative imputation can generate missing features, it usually adds noise indiscriminately to the original feature space or adjacency matrix space, which can easily disrupt speaker identity dependencies, contextual temporal dependencies, and local topological structures in the dialogue graph.

[0004] Furthermore, existing fusion networks often employ fixed-weight concatenation, attention-weighted methods, or static graph convolutions, lacking explicit evaluation of the quality of reconstructed features. When the reconstruction result of a certain modality or roadmap structure is unreliable, static fusion mechanisms may still propagate high-uncertainty features to higher-order graph inference processes, leading to the amplification of erroneous sentiment evidence. Simultaneously, different modalities possess different frequency domain features and signal-to-noise ratio characteristics. Applying the same diffuse noise scheduling to textual, acoustic, and visual modalities can easily result in low-frequency semantic divergence or insufficient reconstruction of high-frequency speech details.

[0005] Therefore, it is necessary to propose a multimodal emotion recognition method that can utilize the survival modality to provide prior guidance, perform spectral space denoising and reconstruction without destroying the graph topology, and dynamically adjust the dual-graph fusion weights according to the reconstruction uncertainty. Summary of the Invention

[0006] The purpose of this invention is to provide a cross-modal collaborative perception and dual-graph spectrum diffusion emotion recognition method to solve the problems of topological structure destruction, interpolation artifact interference, lack of modal adaptability of noise scheduling, and static rigidity of fusion weights in existing multimodal emotion recognition methods in modality-deficient scenarios.

[0007] To achieve the above objectives, the present invention adopts the following technical solution: a cross-modal collaborative perception and dual-graph spectrum diffusion emotion recognition method, including a training process and a prediction process; The training process includes the following steps: A1. Acquire sample data and perform modality-aware preprocessing, extract text modality, acoustic modality and visual modality features in the dialogue, generate a modality missing mask matrix based on the missing state of each modality at each utterance position, and construct cross-modality collaborative anchor points using the surviving modalities that are not missing. A2. By using a pre-trained encoder and a linear mapping layer, the features of each modality are mapped to a latent vector space of a unified dimension. Speaker graphs and temporal graphs are constructed based on speaker identity dependency and temporal context dependency, respectively. Spectral space decomposition is then performed on the adjacency matrix of the two graphs. A3. Construct a modality-aware dual-graph spectrum diffusion reconstruction model, constrain the diffusion process to the diagonal eigenvalues ​​and node spectral features after spectral decomposition, and combine modality adaptive noise adjustment coefficients to generate reconstructed spectral features of missing modes; A4. Perform gated spectral classification based on uncertainty information entropy, calculate the information entropy of reconstructed node features to generate dynamic gating weights, perform adaptive arbitration fusion of speaker graph features and time sequence graph features, and obtain high-order node representations through spectral graph convolution; A5. Construct a joint loss function based on sentiment classification loss, spectral diffusion reconstruction loss and gating regularization loss, and update the parameters of the pre-trained encoder, dual-graph spectral diffusion reconstruction model, gating spectral classifier and classifier through backpropagation; The prediction process includes the following steps: B1. Obtain the dialogue sample to be identified and preprocess it to extract existing heterogeneous modal features and corresponding missing masks; B2. The existing heterogeneous modal features are mapped to a unified latent vector space through the encoder, and the cross-modal collaborative perception module is used to generate initial prior anchor points for the missing modalities. B3. The embedded vector with initial prior anchor points is input into the trained dual-graph spectral diffusion reconstruction model. After spectral space inverse denoising, dynamic gating fusion and spectral graph convolutional inference, the sentiment category label is output by the classifier.

[0008] Specifically, step A1 includes the following sub-steps: A11. Segment the multimodal dialogue sample into N utterances arranged in chronological order, and obtain the original text, acoustic and visual signals and their sentiment category labels for each utterance; A12. Perform time-scale alignment and cleaning on the original text, acoustic and visual signals, set a random missing rate or specify a modal masking strategy, and generate a binary modal missing mask matrix. A13. Extract the original feature sequences for each mode; utilize the feature matrix of the observed modes that are not missing. Initial feature anchors for missing modalities are generated using a multi-head cross-attention mechanism. The formula is:

[0009] in This is the zero-initialization matrix due to the lack of modes. The projected weight matrix is ​​a learnable matrix. For feature latent dimensions; The collaborative anchor point This will serve as a high-quality physical prior starting point for subsequent diffusion networks.

[0010] Specifically, step A2 includes the following sub-steps: A21. Extract the original features of each modality using a text encoder, acoustic encoder, and visual encoder respectively, and map the heterogeneous features to a unified dimension using modality-specific 1D convolution. Feature vector set ; A22. Establish a dual-graph structure: Construct dual-path graphs using node sets with unified feature representations; among them, construct a speaker graph based on the identity dependency relationship between the same speaker. Construct a sequence graph based on the past, present, and future temporal context dependencies of dialogue discourse. ; A23. Graph spatial eigenvalue decomposition: This involves analyzing the adjacency matrices of the two graphs respectively. Perform eigenvalue decomposition:

[0011] Where the orthogonal matrix Invariant feature vectors representing the topological structure of the dialogue graph. This is the diffusible diagonal eigenvalue matrix representing the energy distribution.

[0012] Specifically, step A3 includes: A31. Define a continuous-time two-path spectral diffusion fractional network, considering the node feature matrix. and eigenvalue matrix Construct a positive Gaussian-noisy stochastic differential equation; A32. Introduce a modal adaptive noise adjustment coefficient in the reverse denoising stage. Discretization reverse reconstruction is performed using the following formula:

[0013] Among the variable union , For the set of missing features, Standard Gaussian white noise, Determine the deterministic drift trajectory, This is a conditional score estimator based on a neural network; A33, after After one inverse time iteration, the network recovers the repaired high-fidelity node features from the Gaussian distribution. and the reconstructed eigenvalue matrix .

[0014] Specifically, in step A4, the execution of gated spectrum classification based on uncertainty information entropy includes the following sub-steps: A41. Reconstruction Uncertainty Measurement: For each feature node vector reconstructed from each graph under dual-path conditions. And calculate the information entropy of its internal probability distribution. The formula is:

[0015] A42. Dynamic Gated Arbitration Vector Generation: The calculated information entropy scalar is concatenated with the node features themselves, and then input into a sigmoid-activated network to learn node-level adaptive gating vectors. :

[0016] The gating vector is used to perform element-wise Hadamard products on the features of the two images. polymerization:

[0017] A43. Higher-order propagation in spectral graph neural networks: based on weight balancing parameters. The two adjacency matrices after dynamic merging and reconstruction And perform multi-layer spectral graph convolution on the fused hypergraph topology to update higher-order features:

[0018] Specifically, step A5 includes: A51. Input the updated output node representation matrix into the multilayer perceptron classifier and calculate the cross-entropy bias. ; A52. Using the original, unmissing data as the gold standard monitoring signal, calculate the spectral diffusion dual reconstruction loss. ; A53. Construct a dynamically weighted global total loss function and introduce norm pairs for the gated weight matrix. Apply regularization constraints:

[0019] in All of these are hyperparameters.

[0020] Specifically, in the prediction process, for missing modes, the cross-modal collaborative anchors first provide non-zero prior initialization, then the dual-graph spectrum diffusion reconstruction model performs unidirectional inverse denoising in the spectral space, and finally the gated spectrum classifier reduces the fusion weight of unreliable graph paths based on the information entropy of the reconstructed features.

[0021] The beneficial effects of this invention are as follows: 1. Dual Fidelity Preservation of Structure and Distribution – Dual-Graph Spectral Space Constrained Diffusion. For the first time, the noise addition and denoising processes of the diffusion model are strictly constrained within the diagonal eigenvalue space. Reconstruction is performed only on the diagonal eigenvalues ​​representing energy distribution, while the orthogonal eigenvectors representing speaker dependence and temporal dependence remain completely unchanged. This fundamentally avoids the problem of traditional generative interpolation destroying graph topology and dialogue semantic continuity due to blindly perturbing the original features, while simultaneously restoring modal energy.

[0022] 2. Pre-diffusing Collaboration and Noise Robustness – Cross-modal Collaborative Anchors and Non-zero Prior Initialization. Abandoning traditional zero-filling or pure white noise starting points, cross-modal collaborative anchors are generated using multi-head cross-attention from the non-missing modalities before diffusion, providing high-quality physical priors carrying cross-modal semantic associations for the missing modalities. Simultaneously, modality-aware adaptive noise scheduling coefficients are introduced during the reverse diffusion denoising process, accurately matching the signal-to-noise ratio characteristics and frequency domain structure of the text, acoustic, and visual modalities. This significantly suppresses artifacts caused by unreasonable divergence during diffusion, improving the adaptability and stability of multimodal reconstruction.

[0023] 3. Dynamic Adaptive Structure – Automatic Adjustment of Gated Spectrum Classification and Fusion Weights Based on Information Entropy. The fusion network is not statically fixed but incorporates uncertainty quality discrimination capabilities. The reconstruction uncertainty of each graph path is quantified by calculating the information entropy of the reconstructed node features, and this information is input into the gated spectrum classifier to dynamically generate node-level gate weights. When a graph path suffers from extremely poor repair quality under extreme modal absence, the system automatically compresses its fusion weights, minimizing the impact of low-quality repair paths on higher-order graph reasoning and final sentiment decision-making, thereby maintaining high robustness and consistency of reasoning logic in complex missing scenarios.

[0024] 4. System Collaboration and End-to-End Joint Optimization. The three mechanisms described above work synergistically under a unified modeling framework, employing a joint loss function for end-to-end training: first, cross-modal collaborative anchors provide prior guidance; second, dual-graph spectral diffusion faithfully repairs missing modal features within the spectral space; and finally, an entropy-weighted gating mechanism dynamically fuses dual-graph information based on reconstruction credibility, and sentiment classification inference is completed via spectral graph convolution. This comprehensively ensures the accuracy and stability of multimodal sentiment recognition under conditions of high missing rates and strong noise interference. Attached Figure Description

[0025] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the structures shown in these drawings without creative effort.

[0026] Figure 1 This is a flowchart of the model training process in an embodiment of the present invention; Figure 2 This is a schematic diagram illustrating the principle of dual-spectrum diffusion and gated spectrum classification in an embodiment of the present invention. Figure 3 This is a flowchart of the model prediction process in an embodiment of the present invention; Figure 4 This is a flowchart illustrating the overall process of emotion recognition based on cross-modal collaborative perception and dual-graph spectrum diffusion in an embodiment of the present invention. Detailed Implementation

[0027] It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the invention.

[0028] The following detailed description of the implementation method of the present invention is in conjunction with the accompanying drawings. The description is only a partial embodiment and not all embodiments. For clarity, representations and descriptions unrelated to the present invention are omitted in the drawings and description.

[0029] To provide a clearer understanding of the technical features, objectives, and beneficial effects of this invention, the following detailed description of the technical solution is provided. Obviously, the described embodiments are only a portion of the embodiments of this invention, not all of them, and should not be construed as limiting the scope of implementation of this invention. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without inventive effort are within the protection scope of this invention.

[0030] The present invention will be further described below with reference to the accompanying drawings and embodiments. It should be understood that the described embodiments are only for explaining the present invention and are not intended to limit the scope of protection of the present invention. Where there is no conflict, the technical features in the following embodiments can be combined with each other.

[0031] Example 1: The multimodal emotion recognition method based on cross-modal collaborative perception and dual-graph spectral diffusion in this embodiment includes two parts: a training process and a prediction process. These are described in detail below with reference to the accompanying drawings: See Figure 1 The training process includes: S11. Obtain the dialogue video dataset and segment the dialogue into chronologically ordered speech sequences. For the text, acoustic, and visual modalities, perform random masking operations according to a set missing rate to generate a modality missing mask matrix; S12. Resize the input image, tokenize the text, extract Mel frequency cepstral coefficients from the audio and perform frame segmentation to complete the standardized feature extraction of the data. S13. Cross-modal collaborative anchor initialization: Based on the binary mask matrix generated in S11, for the positions marked as missing modalities, the traditional zero-vector imputation scheme is abandoned. Survival modalities are used to construct feature priors, and the cross-attention mechanism between the survival and missing modalities is calculated to generate anchor features with semantic alignment constraints. These features serve as a high-quality physical starting point for the subsequent spectral diffusion network. As a further optimization, in step A2, mapping the features of each modality into a latent vector space of a unified dimension through a pre-trained encoder and linear mapping layer specifically includes: S21. Build a multimodal feature extraction architecture and load pre-trained models respectively: DeBERTa is used to extract contextual semantics for text, Wav2Vec 2.0 is used to extract speech features for acoustics, and MANet is used to extract facial expressions for vision. S22. The preprocessed segments are input into the encoder network. Modality-specific 1D convolutions are used to eliminate sequence length differences, mapping features to a unified latent dimension and injecting speaker position embeddings. Subsequently, a speaker graph is built based on the self-reference relationships of the same speaker. Establish a sequence diagram based on the logic of past and future time windows. To maintain local topological stability, Laplace spectral decomposition is performed on the two adjacency matrices. .

[0032] As a further optimization, step A3, which involves constructing a modality-aware dual-spectrum diffusion reconstruction model, specifically includes: S31. Regarding the eigenvectors of the two graphs and the diagonal eigenvalues ​​derived from their decomposition. The data is then fed into a continuous-time fractional network solver. Gaussian white noise is progressively added using a forward differential equation until the data is purified to the prior distribution. S32. Perform modality-aware inverse denoising: Start the inverse SDE solver. Assign large adaptive coefficients to the acoustic signal. To accelerate high-frequency fitting, a smaller allocation is made for the text signal. Prevent low-frequency semantic divergence.

[0033] S33, after Through iterative inference, the network outputs an unbiased reconstructed set of repaired features. And the characteristic values ​​of the energy distribution were restored. .

[0034] As a further optimization, step A4, which involves performing gated spectrum classification based on uncertainty information entropy, specifically includes: S41. Uncertainty Assessment: The generated node features are treated as probability distribution states, and the distribution entropy of the reconstructed features is calculated node by node using Shannon's formula. ; S42, Entropy-Gated Aggregation: The scalar entropy and node features are concatenated using Concat and then input into a single-layer linear network activated by Sigmoid, mapping to... The gated damping weights between Subsequently, using this gating weight, element-wise weighted combination operations are performed on the features of the speaker graph and sequence graph at the node granularity. Physically isolated high-entropy noise nodes; S43. Reload the fused node feature matrix into the merged spectral space topology, so that... Perform Chebyshev polynomial convolution on the graph transition operator to achieve global semantic broadcasting; S44, High-order feature output: After multiple passes, the output is a robust feature representation that combines global structure and local identity information, which is then fed into the subsequent fully connected classifier network.

[0035] As a further optimization, step A5, the method for calculating the joint loss of multiple tasks and fine-tuning the network parameters according to the uncertainty, specifically includes: S51. Calculate the cross-entropy loss for the sentiment classification task. The probability predictions are aligned with the actual sentiment labels using Softmax. S52. Extract the mean square error of the diffusion network at each time step and calculate the diffusion reconstruction loss for the score matching dimension. ; S53. Construct the total loss function with smoothness constraints: In the training batches, the Adam optimizer with momentum decay is used to iteratively calculate gradients and update model weights in reverse, completing the closed-loop training of the entire architecture.

[0036] The algorithm principle for model training in steps S1-S5 above is as follows: Figure 2 As shown.

[0037] The prediction process flow in this embodiment is shown below. Figure 3 It includes: S1. Obtain the actual dialogue data to be predicted and preprocess it, extract the existing text, acoustic and visual raw signals, and accurately locate the modal timestamp coordinates that are missing due to sensor failure. S2. The existing modalities are mapped to embedding vectors through a pre-trained encoder and linear projection layer. The collaborative perception module is invoked to extract semantics from the live sequences, and prior anchor features are forcibly injected into the missing parts. S3. The embedding matrix carrying high-quality anchors is fed into the converged dual-spectrum diffusion model for unidirectional inverse denoising. The reconstructed data flows into the dynamic gated spectrogram network. The model automatically decays the weights of bad paths by relying on the built-in entropy weight introspection mechanism, performs highly robust logical reasoning instantly, and finally outputs the sentiment category in the dictionary that is closest to the distribution of the dialogue frame through the Softmax layer.

[0038] In this step, the entire computational graph completely avoids the cascading error amplification effect caused by static fusion, maximizing the utilization of real data and the reliability of interpolated data.

[0039] See Figure 4 The multimodal emotion recognition method in this embodiment of the invention generally includes: first, preprocessing multimodal data such as text, acoustic and visual data, and generating cross-modal collaborative anchors based on the missing modalities; then constructing speaker graphs and time series graphs, and performing spectral space decomposition on the dual graph spectra; then reconstructing the missing modal features and spectral features through a dual graph spectral diffusion model; then dynamically fusing the dual graph features using an entropy weight gating mechanism, and combining spectral graph convolution to complete high-order feature propagation; finally, outputting the emotion classification result. Figure 4 This is used to summarize and demonstrate the overall processing chain of the present invention, from multimodal input to emotion classification output.

[0040] The above description is merely a preferred embodiment of the present invention. It should be understood that the present invention is not limited to the forms disclosed herein and should not be construed as excluding other embodiments. It can be used in various other combinations, modifications, and environments, and can be altered within the scope of the concept described herein through the above teachings or related technologies or knowledge. Modifications and variations made by those skilled in the art that do not depart from the spirit and scope of the present invention should be within the protection scope of the appended claims.

Claims

1. A cross-modal collaborative perception and dual-graph spectral diffusion emotion recognition method, characterized in that, This includes the training process and the prediction process; The training process includes the following steps: A1. Acquire sample data and perform modality-aware preprocessing, extract text modality, acoustic modality and visual modality features in the dialogue, generate a modality missing mask matrix based on the missing state of each modality at each utterance position, and construct cross-modality collaborative anchor points using the surviving modalities that are not missing. A2. By using a pre-trained encoder and a linear mapping layer, the features of each modality are mapped to a latent vector space of a unified dimension. Speaker graphs and temporal graphs are constructed based on speaker identity dependency and temporal context dependency, respectively. Spectral space decomposition is then performed on the adjacency matrix of the two graphs. A3. Construct a modality-aware dual-graph spectrum diffusion reconstruction model, constrain the diffusion process to the diagonal eigenvalues ​​and node spectral features after spectral decomposition, and combine modality adaptive noise adjustment coefficients to generate reconstructed spectral features of missing modes; A4. Perform gated spectral classification based on uncertainty information entropy, calculate the information entropy of reconstructed node features to generate dynamic gating weights, perform adaptive arbitration fusion of speaker graph features and time sequence graph features, and obtain high-order node representations through spectral graph convolution; A5. Construct a joint loss function based on sentiment classification loss, spectral diffusion reconstruction loss and gating regularization loss, and update the parameters of the pre-trained encoder, dual-graph spectral diffusion reconstruction model, gating spectral classifier and classifier through backpropagation; The prediction process includes the following steps: B1. Obtain the dialogue sample to be identified and preprocess it to extract existing heterogeneous modal features and corresponding missing masks; B2. The existing heterogeneous modal features are mapped to a unified latent vector space through the encoder, and the cross-modal collaborative perception module is used to generate initial prior anchor points for the missing modalities. B3. The embedded vector with initial prior anchor points is input into the trained dual-graph spectral diffusion reconstruction model. After spectral space inverse denoising, dynamic gating fusion and spectral graph convolutional inference, the sentiment category label is output by the classifier.

2. The emotion recognition method based on cross-modal collaborative perception and dual-graph spectral diffusion as described in claim 1, characterized in that, Step A1 specifically includes the following sub-steps: A11. Segment the multimodal dialogue sample into N utterances arranged in chronological order, and obtain the original text, acoustic and visual signals and their sentiment category labels for each utterance; A12. Perform time-scale alignment and cleaning on the original text, acoustic and visual signals, set a random missing rate or specify a modal masking strategy, and generate a binary modal missing mask matrix. A13. Extract the original feature sequences for each mode; utilize the feature matrix of the observed modes that are not missing. Initial feature anchors for missing modalities are generated using a multi-head cross-attention mechanism. The formula is: ;in This is the zero-initialization matrix due to the lack of modes. The projected weight matrix is ​​a learnable matrix. For feature latent dimensions; The collaborative anchor point This will serve as a high-quality physical prior starting point for subsequent diffusion networks.

3. The emotion recognition method based on cross-modal collaborative perception and dual-graph spectral diffusion as described in claim 1, characterized in that, Step A2 specifically includes the following sub-steps: A21. Extract the original features of each modality using a text encoder, acoustic encoder, and visual encoder respectively, and map the heterogeneous features to a unified dimension using modality-specific 1D convolution. Feature vector set ; A22. Establish a dual-graph structure: Construct dual-path graphs using node sets with unified feature representations; among them, construct a speaker graph based on the identity dependency relationship between the same speaker. Construct a sequence graph based on the past, present, and future temporal context dependencies of dialogue discourse. ; A23. Graph spatial eigenvalue decomposition: This involves analyzing the adjacency matrices of the two graphs respectively. Perform eigenvalue decomposition: ; where orthogonal matrix Invariant feature vectors representing the topological structure of the dialogue graph. This is the diffusible diagonal eigenvalue matrix representing the energy distribution.

4. The cross-modal collaborative perception and dual-graph spectrum diffusion emotion recognition method as described in claim 1, characterized in that, Step A3 specifically includes: A31. Define a continuous-time two-path spectral diffusion fractional network, considering the node feature matrix. and eigenvalue matrix Construct a positive Gaussian-noisy stochastic differential equation; A32. Introduce a modal adaptive noise adjustment coefficient in the reverse denoising stage. Discretization reverse reconstruction is performed using the following formula: ； Among the variable union , For the set of missing features, Standard Gaussian white noise, Determine the deterministic drift trajectory, This is a conditional score estimator based on a neural network; A33, after After one inverse time iteration, the network recovers the repaired high-fidelity node features from the Gaussian distribution. and the reconstructed eigenvalue matrix .

5. The emotion recognition method based on cross-modal collaborative perception and dual-graph spectral diffusion as described in claim 1, characterized in that, Step A4, which involves performing gated spectrum classification based on uncertainty information entropy, specifically includes the following sub-steps: A41. Reconstruction Uncertainty Measurement: For each feature node vector reconstructed from each graph under dual-path conditions. And calculate the information entropy of its internal probability distribution. The formula is: ； A42. Dynamic Gated Arbitration Vector Generation: The calculated information entropy scalar is concatenated with the node features themselves, and then input into a sigmoid-activated network to learn node-level adaptive gating vectors. : ； The gating vector is used to perform element-wise Hadamard products on the features of the two images. polymerization: ； A43. Higher-order propagation in spectral graph neural networks: based on weight balancing parameters. The two adjacency matrices after dynamic merging and reconstruction And perform multi-layer spectral graph convolution on the fused hypergraph topology to update higher-order features: 。 6. The emotion recognition method based on cross-modal collaborative perception and dual-graph spectral diffusion as described in claim 1, characterized in that, Step A5 specifically includes: A51. Input the updated output node representation matrix into the multilayer perceptron classifier and calculate the cross-entropy bias. ; A52. Using the original, unmissing data as the gold standard monitoring signal, calculate the spectral diffusion dual reconstruction loss. ; A53. Construct a dynamically weighted global total loss function and introduce norm pairs for the gated weight matrix. Apply regularization constraints: ;in All of these are hyperparameters.

7. The cross-modal collaborative perception and dual-graph spectrum diffusion emotion recognition method as described in claim 1, characterized in that, During the prediction process, for missing modes, the cross-modal collaborative anchors first provide non-zero prior initialization, then the dual-graph spectral diffusion reconstruction model performs unidirectional inverse denoising in the spectral space, and finally the gated spectral classifier reduces the fusion weight of unreliable graph paths based on the information entropy of the reconstructed features.

Images