Method and model training method for multi-modal sentiment recognition under missing modal based on hypergraph neural network

Abstract

The embodiment of the application relates to the field of artificial intelligence, and provides a multi-modal sentiment recognition method and model training method in a missing mode based on a hypergraph neural network. The multi-modal sentiment recognition method in the missing mode based on the hypergraph neural network comprises the following steps: obtaining multi-modal target data to be subjected to sentiment recognition, and encoding the target data to obtain encoded data; performing missing mode processing on the encoded data to obtain processed data in the missing mode; constructing a hypergraph neural network according to the processed data in the missing mode, performing bidirectional updating based on the hypergraph neural network, and reasoning to obtain multi-modal joint feature representation data; and performing sentiment recognition according to the multi-modal joint feature representation data to obtain sentiment judgment result data associated with the target data. The implementation of the method improves the stability and robustness of multi-modal sentiment recognition in the application scenario of the missing mode.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a multimodal emotion recognition method and model training method based on a hypergraph neural network for missing modalities. Background Technology

[0002] In the field of artificial intelligence technology, multimodal emotion recognition analyzes users' emotional states by integrating various information such as text, voice, and video. It has been widely applied in scenarios such as online education to perceive students' participation and learning status.

[0003] However, in practical applications, multimodal data is often missing due to factors such as privacy restrictions, environmental noise, or network instability, resulting in incomplete and dynamically changing modality availability. Most traditional multimodal emotion recognition methods assume that each modality is always available, and their recognition performance drops significantly when modalities are missing, making it difficult to meet the requirements of system stability and robustness in real-world application scenarios. Summary of the Invention

[0004] This application provides a method and model training method for multimodal emotion recognition under missing modalities based on hypergraph neural networks. More specifically, this application provides a method, model training method, device, and computer equipment for multimodal emotion recognition under missing modalities based on hypergraph neural networks, improving the stability and robustness of multimodal emotion recognition in application scenarios with missing modalities.

[0005] Firstly, from the perspective of model inference, embodiments of this application provide a multimodal emotion recognition method based on a hypergraph neural network for missing modalities, including:

[0006] Acquire multimodal target data to be used for emotion recognition, and encode the target data to obtain encoded data;

[0007] The encoded data is subjected to missing mode processing to obtain missing mode processed data;

[0008] A hypergraph neural network is constructed based on the data after processing the missing modalities. The hypergraph neural network is then updated bidirectionally to infer multimodal joint feature representation data.

[0009] Emotion recognition is performed based on the multimodal joint feature representation data to obtain the emotion judgment result data associated with the target data.

[0010] Optionally, in some embodiments of this application, acquiring the multimodal target data to be used for emotion recognition includes:

[0011] Based on the pre-configured preset terminal devices or preset platform interfaces in the emotion recognition scenario, acquire raw multimodal data.

[0012] The original acquired data is processed to obtain modal state labeling data;

[0013] The target data is determined based on the modal state labeling data and the original acquired data.

[0014] Optionally, in some embodiments of this application, encoding the target data to obtain encoded data includes:

[0015] Each modality in the target data is feature-encoded to generate a single-modal representation data with a preset unified dimension;

[0016] The unimodal representation data of each modality are summarized to obtain the encoded data.

[0017] Optionally, in some embodiments of this application, the target data includes text data, and the step of performing feature encoding on the data of each modality in the target data to generate single-modal representation data of a preset uniform dimension includes:

[0018] Word embedding is used to convert each unit word in the text data into a high-dimensional text vector;

[0019] The high-dimensional text vector is input into a preset sequence modeling network to extract contextual features;

[0020] The pooling layer of the neural network associated with the sequence modeling network compresses the context features into a text feature representation of the preset uniform dimension, so as to determine the single-modal representation data corresponding to the text data.

[0021] Optionally, in some embodiments of this application, the target data includes speech data, and the step of performing feature encoding on the data of each modality in the target data to generate single-modal representation data of a preset uniform dimension includes:

[0022] Acoustic features were extracted from the speech data based on audio feature extraction technology;

[0023] The acoustic features are input into a multilayer perceptron and the speech feature representation with a preset uniform dimension is output to determine the single-modal representation data corresponding to the speech data.

[0024] Optionally, in some embodiments of this application, the target data includes video data, and the step of performing feature encoding on the data of each modality in the target data to generate single-modal representation data of a preset uniform dimension includes:

[0025] Feature extraction is performed on the facial video frames in the video data to obtain facial key point data;

[0026] High-dimensional visual features are obtained by stitching together the coordinates of the facial key point data;

[0027] The high-dimensional visual features are input into a multilayer perceptron and the video feature representation with the preset uniform dimension is output to determine the single-modal representation data corresponding to the video data.

[0028] Optionally, in some embodiments of this application, the step of performing missing modality processing on the encoded data to obtain missing modality-processed data includes:

[0029] Modal missing detection is performed on the encoded data to obtain the modal missing detection results;

[0030] When the modality missing detection result is a preset modality missing, a vector with a preset feature vector shape and filled with iterable parameters is constructed based on the encoded data, which serves as a learnable missing modality token; the preset feature vector shape is the same as the feature vector shape of the encoded data of the missing modality.

[0031] The learnable missing modality tokens corresponding to each modality are used as alternative input data for the encoded data to determine the missing modality processed data.

[0032] Optionally, in some embodiments of this application, the step of constructing a hypergraph neural network based on the data after processing the missing modalities, performing bidirectional updates based on the hypergraph neural network, and inferring multimodal joint feature representation data includes:

[0033] Based on the data after processing the missing modalities, a hypergraph node and a hyperedge are constructed, and the hypergraph neural network is determined based on the hypergraph node and the hyperedge;

[0034] The hypergraph nodes and hyperedges are updated bidirectionally based on the hypergraph neural network, and inference is performed based on the bidirectionally updated hypergraph neural network to obtain the multimodal joint feature representation data.

[0035] Optionally, in some embodiments of this application, the step of constructing hypergraph nodes and hyperedges based on the data after processing the missing modalities, and determining the hypergraph neural network based on the hypergraph nodes and hyperedges, includes:

[0036] The features of different modes and different time steps in the data after processing the missing modes are represented as the hypergraph nodes;

[0037] Based on the hypergraph nodes, cross-modal hyperedges and temporal hyperedges are constructed to characterize the higher-order relationships between multiple modalities, thus obtaining the initial hypergraph structure;

[0038] Based on the variational hypergraph autoencoder, the initial hypergraph structure is adaptively optimized to obtain the hypergraph neural network.

[0039] Optionally, in some embodiments of this application, the step of adaptively optimizing the initial hypergraph structure based on the variational hypergraph autoencoder to obtain the hypergraph neural network includes:

[0040] The initial hypergraph structure is convolved to generate the embedding representations of the hypergraph nodes and the hyperedges, and the mean and variance of the latent distribution are encoded to obtain the hypergraph encoding result;

[0041] Based on the hypergraph encoding result, sampling is performed using a reparameterization technique to obtain the potential embeddings of the new hypergraph nodes and hyperedges, thus obtaining the hypergraph sampling result.

[0042] Based on the hypergraph sampling results, the dot product is calculated and the hypergraph correlation matrix is ​​reconstructed. The hypergraph is then updated according to the hypergraph correlation matrix to obtain the hypergraph neural network.

[0043] Optionally, in some embodiments of this application, the step of bidirectionally updating the hypergraph nodes and hyperedges based on the hypergraph neural network, and performing inference based on the bidirectionally updated hypergraph neural network to obtain the multimodal joint feature representation data includes:

[0044] Information propagation and aggregation are performed based on the hypergraph neural network, and the hypergraph nodes and hyperedges are updated bidirectionally.

[0045] Using available modal information, high-order inference is performed in the bidirectionally updated hypergraph neural network to implicitly complete the missing modal features, thereby obtaining the implicitly completed missing modal features.

[0046] Based on the missing modal features after implicit completion, the multimodal joint feature representation data is obtained.

[0047] Optionally, in some embodiments of this application, the step of performing information propagation and aggregation based on the hypergraph neural network, and bidirectionally updating the hypergraph nodes and the hyperedges, includes:

[0048] The features of the hypergraph nodes, the features of the hyperedges, and the weights of the hyperedges in the hypergraph neural network are used as the input to the hypergraph neural network.

[0049] For the features of all the hypergraph nodes connected by any of the hyperedges, the features of the hyperedges are updated by aggregation and mapping of the multilayer perceptron.

[0050] For the features of all the hyperedges associated with any given hypergraph node, the features of the hypergraph node are updated by aggregation and mapping of the multilayer perceptron.

[0051] Optionally, in some embodiments of this application, the step of performing emotion recognition based on the multimodal joint feature representation data to obtain emotion judgment result data associated with the target data includes:

[0052] The multimodal joint feature representation data is input into a classification network with a preset network complexity, and the original sentiment scores of various preset sentiment categories are output based on the activation function of the classification network.

[0053] The original sentiment scores are normalized to obtain probability distribution data for various preset sentiment categories;

[0054] Based on the probability distribution data, the sentiment judgment result data associated with the target data is determined.

[0055] Optionally, in some embodiments of this application, before the step of encoding the target data to obtain encoded data, the method further includes:

[0056] The modal data in the target data are subjected to uniform time alignment processing to obtain time-aligned data;

[0057] The time-aligned data is then subjected to length normalization to obtain the target data after length normalization.

[0058] Optionally, in some embodiments of this application, after the step of acquiring the multimodal target data to be used for emotion recognition, the method further includes:

[0059] Modal missing detection is performed on the target data to obtain modal missing detection results;

[0060] If the modality missing detection result is that the preset modality is completely missing, a learnable missing modality token is directly output to determine the data after the missing modality is processed.

[0061] Optionally, in some embodiments of this application, before the step of encoding the target data to obtain encoded data, the method further includes:

[0062] Modal missing detection is performed on the target data to obtain modal missing detection results;

[0063] If the modal missing detection result is that the preset modality is completely missing, the feature extraction function of the encoder corresponding to the missing modality is paused, and the placeholder representation data corresponding to the missing modality is output as the encoded data.

[0064] Optionally, in some embodiments of this application, the step of constructing a hypergraph neural network based on the data after processing the missing modalities, performing bidirectional updates based on the hypergraph neural network, and inferring multimodal joint feature representation data includes:

[0065] Hypergraph nodes and hyperedges are constructed based on the currently available modal data in the data after processing the missing modalities, and the hypergraph neural network is determined based on the hypergraph nodes and hyperedges;

[0066] A high-order association is established between the hypergraph nodes and hyperedges corresponding to the currently available modal data to perform bidirectional updates to the hypergraph neural network. Inference is then performed based on the bidirectionally updated hypergraph neural network to obtain the multimodal joint feature representation data.

[0067] Optionally, in some embodiments of this application, the step of constructing a hypergraph neural network based on the data after processing the missing modalities includes:

[0068] The changes in the modal combination state of the target data are monitored to obtain the monitoring results of the modal combination state changes;

[0069] When the monitoring result of the modal combination state change indicates that the modal combination state has changed, the hypergraph neural network is constructed by adaptive adjustment based on the preset optimization hypergraph structure strategy.

[0070] Optionally, in some embodiments of this application, the step of constructing a hypergraph neural network based on the data after processing the missing modalities, performing bidirectional updates based on the hypergraph neural network, and inferring multimodal joint feature representation data includes:

[0071] Based on the data after processing the missing modalities, a hypergraph node and a hyperedge are constructed, and the hypergraph neural network is determined based on the hypergraph node and the hyperedge;

[0072] The hypergraph nodes and hyperedges are updated bidirectionally based on the hypergraph neural network.

[0073] The feature data of each modality in the bidirectional updated hypergraph neural network are subjected to dimension alignment and scale normalization to obtain dimension-aligned and scale-normalized feature data.

[0074] The multimodal joint feature representation data is obtained by inference based on the hypergraph neural network constructed from the dimension-aligned and scale-normalized feature data.

[0075] Secondly, from the perspective of model training, embodiments of this application provide a model training method for training an artificial intelligence model involved in the multimodal emotion recognition method based on hypergraph neural networks under missing modalities, as described in the first aspect. The model training method includes:

[0076] Construct a teacher model under modal completeness conditions of the target data, and construct a student model under modal missing conditions of the target data;

[0077] Based on the feature-level knowledge transfer mechanism, the student model is trained according to the emotional representation distribution of the teacher model.

[0078] Optionally, in some embodiments of this application, the step of training the student model based on the feature-level knowledge transfer mechanism according to the sentiment representation distribution of the teacher model includes:

[0079] The sentiment representation features of the same batch of samples were extracted from both the teacher model and the student model.

[0080] Based on the sentiment representation features of the same batch of samples, calculate the distribution difference data between the feature distribution of the student model and the feature distribution of the teacher model;

[0081] Determine the knowledge transfer loss function based on the distribution difference data;

[0082] Based on the knowledge transfer loss function, the overall loss function of the student model is constructed.

[0083] Based on the overall loss function, the various model parameters of the student model are updated through backpropagation.

[0084] Optionally, in some embodiments of this application, constructing the overall loss function of the student model based on the knowledge transfer loss function includes:

[0085] Construct the classification task loss function corresponding to the multimodal sentiment classification process;

[0086] The variational hypergraph autoencoder loss function and cross-view contrast loss function corresponding to the construction and update process of the hypergraph neural network are obtained.

[0087] The overall loss function of the student model is determined by weighted summation of the classification task loss function, the variational hypergraph autoencoder loss function, the cross-view comparison loss function, and the knowledge transfer loss function.

[0088] Thirdly, embodiments of this application provide a multimodal emotion recognition device based on a hypergraph neural network for missing modalities, which has the function of implementing the multimodal emotion recognition method based on a hypergraph neural network for missing modalities provided in the first aspect above. The function can be implemented in hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions, and the modules can be software and / or hardware.

[0089] In one possible design, the multimodal emotion recognition device based on a hypergraph neural network for missing modalities includes:

[0090] The data acquisition and encoding module is used to acquire multimodal target data to be used for emotion recognition, and to encode the target data to obtain encoded data.

[0091] The missing modality processing module is used to perform missing modality processing on the encoded data to obtain missing modality processed data;

[0092] The hypergraph neural network construction and update module is used to construct a hypergraph neural network based on the data after processing the missing modalities, perform bidirectional updates based on the hypergraph neural network, and infer multimodal joint feature representation data.

[0093] The emotion recognition module is used to perform emotion recognition based on the multimodal joint feature representation data to obtain the emotion judgment result data associated with the target data.

[0094] Fourthly, embodiments of this application provide a model training apparatus having the function of implementing the model training method corresponding to the second aspect described above. The function can be implemented in hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions, and the modules can be software and / or hardware.

[0095] In one possible design, the model training device includes:

[0096] The model building module is used to build a teacher model under the condition of modal completeness of the target data, and to build a student model under the condition of modal missingness of the target data;

[0097] The knowledge transfer module is used to train the student model based on the emotion representation distribution of the teacher model, using a feature-level knowledge transfer mechanism.

[0098] In another aspect, this application provides a computer device including at least one connected processor and a memory, wherein the memory is used to store program code, and the processor is used to call the program code in the memory to execute the methods described in the above aspects.

[0099] In another aspect, embodiments of this application provide a computer storage medium including instructions that, when executed on a computer, cause the computer to perform the methods described in the above aspects.

[0100] In another aspect, this application provides a computer program product containing instructions that, when run on a computer, cause the computer to perform the methods described in the above aspects.

[0101] Compared to traditional technologies, the technical solution of this application's embodiments improves the cross-modal information utilization capability by modeling multimodal high-order association relationships based on hypergraph structures. In the case of modality loss, the missing modality features are inferred through the hypergraph information propagation mechanism, thereby improving recognition performance. A unified model structure is adopted to adapt to any combination of missing modalities, eliminating the need to train models separately for different missing modes. It can still operate stably under privacy-restricted or network-unstable conditions, thus comprehensively improving the stability and robustness of multimodal emotion recognition in application scenarios with missing modalities. Attached Figure Description

[0102] Figure 1 This is an application environment diagram from one embodiment;

[0103] Figure 2 This is a flowchart of one embodiment;

[0104] Figure 3 This is a flowchart illustrating another embodiment;

[0105] Figure 4 This is an overall system structure diagram in one embodiment;

[0106] Figure 5 This is a schematic diagram illustrating the knowledge distillation principle in one embodiment;

[0107] Figure 6 This is a structural block diagram of a multimodal emotion recognition device based on a hypergraph neural network under the missing modality in one embodiment;

[0108] Figure 7 This is a structural block diagram of a model training device in one embodiment;

[0109] Figure 8 This is an internal structural diagram of a computer device in one embodiment;

[0110] Figure 9This is a diagram of the internal structure of a computer device in another embodiment. Detailed Implementation

[0111] The terms "first," "second," etc., used in the embodiments of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in a sequence other than that illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or modules is not necessarily limited to those steps or modules explicitly listed, but may include other steps or modules not explicitly listed or inherent to these processes, methods, products, or devices. The division of modules appearing in the embodiments of this application is only a logical division. In actual applications, there may be other division methods. For example, multiple modules may be combined or integrated into another system, or some features may be ignored or not performed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be through some interface, and the indirect coupling or communication connection between modules may be electrical or other similar forms. None of these are limited in the embodiments of this application. Furthermore, the modules or sub-modules described as separate components may or may not be physically separate, may or may not be physical modules, or may be distributed among multiple circuit modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the embodiments of this application.

[0112] Figure 1 As shown in the application environment diagram of one embodiment, this application provides a multimodal emotion recognition method and model training method based on hypergraph neural networks under missing modalities, which can be applied to, for example... Figure 1 In the application scenario shown, terminal 102 communicates with server 104 via a network.

[0113] The terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, smart in-vehicle devices, etc. The server 104 can be implemented using a standalone server or a server cluster consisting of multiple servers.

[0114] It should be noted that the terminal 102 involved in the embodiments of this application can be a wired terminal or a wireless terminal, and can be a device that provides voice and / or data connectivity to a user, a handheld device with wireless connectivity, or other processing devices connected to a wireless modem. The wireless terminal can communicate with one or more core networks via a wireless access network, and the wireless terminal can be a mobile terminal, such as a mobile phone or a computer with a mobile terminal.

[0115] Figure 2 This is a flowchart illustrating one embodiment, such as... Figure 2 As shown in the embodiments of this application, the multimodal emotion recognition method based on hypergraph neural networks under missing modalities includes:

[0116] S2100: Acquire multimodal target data to be used for emotion recognition, and encode the target data to obtain encoded data.

[0117] The target data refers to multimodal data collected from scenarios such as online education or human-computer interaction that are to be used for emotion recognition, such as text data, voice data, and video data.

[0118] Encoding refers to the process of generating single-modal features with a unified dimension by processing multimodal data separately with dedicated encoders; correspondingly, encoded data refers to the data obtained after encoding the target data.

[0119] S2200 performs missing mode processing on the encoded data to obtain the missing mode processed data.

[0120] Missing modality processing refers to the operation of replacing missing modality features in encoded data with learnable missing modality tokens.

[0121] Among them, the data after missing modality processing refers to the data obtained by processing the encoded data with missing modality, that is, the data obtained by replacing it with learnable missing modality tokens.

[0122] S2300: A hypergraph neural network is constructed based on the data after processing the missing modalities. The hypergraph neural network is then used for bidirectional updates to infer multimodal joint feature representation data.

[0123] Among them, hypergraph neural network refers to a neural network model of multimodal high-order associations obtained by modeling hypergraph structure.

[0124] Among them, bidirectional update refers to the process of updating the hyperedge features based on the node features and then updating the node features based on the hyperedge features in the hypergraph.

[0125] In this context, inference refers to the computational process by which a hypergraph neural network aggregates and implicitly completes missing features through information propagation.

[0126] Among them, multimodal joint feature representation data refers to feature data obtained after hypergraph inference that incorporates higher-order associations of multiple modalities.

[0127] S2400 performs emotion recognition based on multimodal joint feature representation data to obtain emotion judgment result data associated with the target data.

[0128] Emotion recognition refers to the process of inputting multimodal joint feature representation data into a classifier to determine the emotion category.

[0129] Among them, the emotion determination result data refers to the data containing the emotion determination result output by emotion recognition.

[0130] Compared to traditional technologies, the embodiments of this application first acquire multimodal target data to be used for emotion recognition, encode the target data to obtain encoded data, then process the encoded data for missing modalities to obtain processed data for missing modalities, and then construct a hypergraph neural network based on the processed data for missing modalities. Based on the hypergraph neural network, bidirectional updates are performed to infer multimodal joint feature representation data, and finally, emotion recognition is performed based on the multimodal joint feature representation data to obtain the emotion judgment result data associated with the target data. The technical solution of the embodiments of this application models high-order multimodal associations based on a hypergraph structure, improving the ability to utilize cross-modal information. In the case of missing modalities, the missing modal features are inferred through the hypergraph information propagation mechanism, improving recognition performance. A unified model structure is adopted to adapt to any combination of missing modalities, eliminating the need to train models separately for different missing modes. It can still operate stably under privacy-restricted or network-unstable conditions, thereby comprehensively improving the stability and robustness of multimodal emotion recognition in application scenarios with missing modalities.

[0131] Optionally, in some embodiments of this application, obtaining target data of the multimodal emotion recognition to be performed includes: obtaining raw multimodal data based on a preset terminal device or preset platform interface pre-configured in the emotion recognition scenario; performing modal state marking processing on the raw data to obtain modal state marking data; and determining target data based on the modal state marking data and the raw data.

[0132] Among them, the emotion recognition scenario refers to the specific application scenario in which emotion recognition tasks are carried out, such as online education or human-computer interaction scenarios; the preset terminal device refers to the terminal device pre-configured for data collection in the emotion recognition scenario; naturally, the preset platform interface is the platform interface pre-configured for data acquisition in the emotion recognition scenario.

[0133] Raw data refers to unprocessed text, voice, video, and other modal data obtained from the aforementioned terminals or interfaces.

[0134] Modal state labeling processing refers to the operation of labeling the complete or missing states of each modality in the original acquired data; modal state labeling data refers to the labeled data obtained after completing the modal state labeling of the original acquired data.

[0135] The target data refers to the data to be identified obtained by fusing modal state label data and original acquired data.

[0136] For example, this application supports the collection of multimodal emotion data, including modal information such as text, voice and video; the collection source is dialogue content, voice signals and video clips in online education or human-computer interaction scenarios; the collection method is to obtain existing multimodal feature data through terminal devices or platform interfaces.

[0137] In this embodiment, it is necessary to perform modal state labeling on the data of missing modalities to provide a basis for subsequent processing. Specifically, the modal state labeling process can assign a state label bit to each modality (text, speech, video), for example, binary bits (0 / 1), where 0 indicates that the data of that modality is missing, and 1 indicates that the data of that modality is complete. Then, based on the values ​​of the three state label bits of the three modalities, a three-dimensional state label vector (a, b, c) is constructed as modal state label data, which is input into the model along with the original multimodal acquisition data. That is, the modal state label data and the original acquisition data constitute the target data and are then input into the subsequent model.

[0138] For example, if text, voice, and video are all present, the state label vector is (1, 1, 1); if video is missing, but only text and voice are present, the state label vector is (1, 1, 0); if voice and video are missing, but only text is present, the state label vector is (1, 0, 0).

[0139] In this embodiment, multimodal data is acquired and modal states are labeled through terminal devices or platform interfaces to clarify the integrity and missing status of modalities, providing accurate basis for subsequent processing and enhancing the model's adaptability to multimodal data.

[0140] Optionally, in some embodiments of this application, encoding the target data to obtain encoded data includes: performing feature encoding on the data of each modality in the target data to generate single-modal representation data of a preset uniform dimension; and summarizing the single-modal representation data of each modality to obtain encoded data.

[0141] Feature encoding refers to the process of extracting features from each modality of data and mapping them to a specified format.

[0142] Here, the preset unified dimension refers to the dimension to which all modal features must be uniformly transformed; for example, the preset unified dimension is a 512-dimensional single-modal representation. Single-modal representation data refers to the unified-dimensional feature data obtained after feature encoding of a single modality.

[0143] In this context, "aggregation" refers to the operation of integrating the single-modal representation data of each modality into a whole; "encoded data" refers to the feature data obtained after aggregating the single-modal representation data of all modalities.

[0144] For example, feature encoding is performed on each modality of data to generate a single modality representation with a unified dimension. The text modality uses a sequence modeling network to extract context-related features, while the speech and video modalities are feature-mapped through a multilayer perceptual network.

[0145] In this embodiment, after features are extracted from each modality's data by its respective encoder, they are uniformly mapped to a 512-dimensional single-modality representation. This unified dimensional setting can effectively eliminate the differences in dimensionality and distribution of the original features of different modalities, which facilitates the subsequent consistent modeling of multimodal nodes by the hypergraph neural network. At the same time, in the case of missing modalities, the missing modalities can directly participate in inference with placeholder representations of the same dimension, thereby maintaining the stability of the model structure and inference process. Compared with low-dimensional representations, 512-dimensional vectors have stronger expressive power and are more conducive to characterizing complex emotional semantics and cross-modal high-order relationships, balancing performance and engineering feasibility.

[0146] In this embodiment, by encoding each modal data separately and generating a unified dimensional representation, the encoded data is obtained by summarizing the data, which eliminates the differences in features between different modalities, improves data compatibility, and enhances the stability and efficiency of subsequent processing.

[0147] Optionally, in some embodiments of this application, the target data includes text data. Feature encoding is performed on the data of each modality in the target data to generate single-modal representation data of a preset uniform dimension. This includes: converting each unit word in the text data into a high-dimensional text vector through word embedding; inputting the high-dimensional text vector into a preset sequence modeling network to extract context features; and compressing the context features into a text feature representation of a preset uniform dimension based on the pooling layer of the neural network associated with the sequence modeling network to determine the single-modal representation data corresponding to the text data.

[0148] Word embedding refers to the technique of converting text terms into high-dimensional vectors to represent semantic information; unit terms refer to the smallest linguistic unit in text data that can be independently encoded, such as characters or words; high-dimensional text vectors refer to vectors containing semantic information obtained after word embedding transformation.

[0149] The preset sequence modeling network refers to a pre-configured network used to extract sequence context features. The preset sequence modeling network can be a gated recurrent unit (GRU) network, or a long short-term memory (LSTM) network, a transformer, or other neural networks.

[0150] Among them, contextual features refer to the features extracted from the text sequence that reflect the semantic relationship of the context.

[0151] In this context, the pooling layer of a neural network refers to the network layer used to compress the feature dimension in a sequence modeling network; compression refers to the process of converting high-dimensional context features into low-dimensional features through pooling operations; and the single-modal representation data corresponding to text data refers to the unified-dimensional feature data ultimately generated from the text modality.

[0152] For example, the text modality uses a sequence modeling network to extract context-related features. The specific steps are as follows: (1) Collect the text sentences of students asking questions online in the online education scenario, such as the question "I don't understand the derivation process of this formula"; (2) Convert each character or word in the text sentence into a high-dimensional text vector (e.g., a 512-dimensional vector) through word embedding; (3) Input the high-dimensional text vector into the sequence modeling network to extract context features; (4) Compress the extracted context features into a text feature representation of a uniform dimension (e.g., 512 dimensions) through a pooling layer (e.g., mean pooling).

[0153] In this embodiment, text data is processed through word embedding, sequence modeling network and pooling layer to accurately extract context features and compress them to a unified dimension, thereby improving the accuracy of text feature representation, enhancing the robustness of text modality coding, and ensuring the effect of subsequent multimodal fusion.

[0154] Optionally, in some embodiments of this application, the target data includes speech data. Feature encoding is performed on the data of each modality in the target data to generate single-modal representation data with a preset unified dimension. This includes: extracting acoustic features from the speech data based on audio feature extraction technology; inputting the acoustic features into a multilayer perceptron and outputting a speech feature representation with a preset unified dimension to determine the single-modal representation data corresponding to the speech data.

[0155] Audio feature extraction technology refers to the technical means of extracting features that can characterize acoustic properties from speech data; acoustic features refer to the extracted feature data that reflects the acoustic properties of speech.

[0156] Among them, Multi-Layer Perceptron (MLP) refers to a neural network for feature mapping composed of stacked fully connected layers; the single-modal representation data corresponding to speech data refers to the unified-dimensional feature data finally generated from the speech modality.

[0157] For example, the speech modality is feature-mapped through a multilayer perceptron. The specific steps include: (1) collecting the speech signal of a student asking a question in an online education scenario (duration 3 seconds, sampling rate 16kHz); (2) extracting acoustic features of a certain dimension (e.g., 20 dimensions) through audio feature extraction technology (e.g., Mel-frequency cepstral coefficients MFCC); (3) inputting the acoustic features into an MLP (e.g., its hidden layer has three layers, with dimensions of 256, 192, and 128 respectively), and finally outputting speech features of a unified dimension (e.g., 128 dimensions).

[0158] In this embodiment, acoustic features are extracted using audio feature extraction technology, mapped to a unified dimension via a multi-layer perceptual network, accurately representing speech features, improving the accuracy of speech coding, enhancing the adaptability of speech modal features, and ensuring the effect of unified multimodal modeling.

[0159] Optionally, in some embodiments of this application, the target data includes video data. Feature encoding is performed on the data of each modality in the target data to generate single-modal representation data of a preset unified dimension. This includes: extracting features from facial video frames in the video data to obtain facial key point data; stitching together the coordinates of the facial key point data to obtain high-dimensional visual features; inputting the high-dimensional visual features into a multilayer perceptron and outputting a video feature representation of a preset unified dimension to determine the single-modal representation data corresponding to the video data.

[0160] Among them, facial video frames refer to single-frame image data containing the face region in video data; facial key point data refer to coordinate data representing key positions of the face.

[0161] Among them, stitching refers to the operation of integrating the coordinates of multiple facial key points into continuous features; high-dimensional visual features refer to the feature data obtained after stitching the coordinates.

[0162] Among them, the single-modal representation data corresponding to video data refers to the unified dimensional feature data ultimately generated from the video modality.

[0163] For example, the video modality is feature-mapped through a multilayer perceptron. The specific steps may include: (1) acquiring facial video frames when students ask questions in an online education scenario; (2) extracting features from the facial video frames, such as extracting key points of the face and splicing them together according to the coordinates of the key points to obtain high-dimensional visual features; (3) inputting the high-dimensional visual features into the MLP (for example, its hidden layer has two layers with dimensions of 256 and 128 respectively), and finally outputting video features of a unified dimension (such as 128 dimensions).

[0164] In this embodiment, facial key points are extracted and stitched together to form visual features, which are then mapped to a unified dimension through a multi-layer perceptual network to accurately represent the emotional features of the video. This improves the accuracy of video encoding, enhances the adaptability of visual features, and ensures the effectiveness of multimodal unified modeling.

[0165] In another embodiment, the process of encoding the target data to obtain encoded data further includes introducing speaker or context embeddings for encoding to enhance the expressive power of single-modal features. Speaker embedding refers to using a small-scale MLP network to encode the speaker's user identifier (such as ID) and fusing the encoding result into the encoded data; context embedding refers to introducing contextual embedding features into the encoded data during the text modality data processing.

[0166] Optionally, in some embodiments of this application, missing modality processing is performed on the encoded data to obtain missing modality processed data, including: performing modality missing detection on the encoded data to obtain a modality missing detection result; if the modality missing detection result is a preset modality missing, constructing a vector with a preset feature vector shape filled with iterable parameters based on the encoded data as a learnable missing modality token; and using the learnable missing modality tokens corresponding to each modality as alternative input data for the encoded data to determine the missing modality processed data.

[0167] Modal missing detection refers to the detection operation that determines whether each modality in the encoded data is missing; the modal missing detection result is the result of whether the modality is complete or missing after modal missing detection.

[0168] Among them, the preset mode missing can be the missing of any mode; the preset feature vector shape is the same as the feature vector shape of the encoded data of the missing mode; the vector filled with iterable parameters refers to the feature vector filled with optimizable parameters.

[0169] Among them, the learnable missing modality token, also known as the learnable missing modality label, refers to the missing modality replacement vector that can be optimized through training iterations; the missing modality processed data refers to the complete feature data obtained after replacing the missing modality.

[0170] For example, when a missing modality is detected, a learnable missing modality label is used to replace the original feature input; specifically, the learnable missing modality label is a vector with the same shape as the original missing modality feature vector, filled with iterable parameters, used to learn prior knowledge of a certain modality during training after replacing the original feature input.

[0171] In this embodiment, by detecting missing modalities and replacing missing modalities with learnable tokens, the input structure is kept consistent, the model's adaptability to missing modalities is improved, the stability of feature processing is enhanced, and the accuracy of subsequent inference is guaranteed.

[0172] Optionally, in some embodiments of this application, a hypergraph neural network is constructed based on the data after processing for missing modalities, and bidirectional updates are performed based on the hypergraph neural network to infer multimodal joint feature representation data. This includes: constructing hypergraph nodes and hyperedges based on the data after processing for missing modalities, determining the hypergraph neural network based on the hypergraph nodes and hyperedges; performing bidirectional updates on the hypergraph nodes and hyperedges based on the hypergraph neural network, and performing inference based on the bidirectionally updated hypergraph neural network to obtain multimodal joint feature representation data.

[0173] In this context, a hypergraph node refers to a feature carrier that maps the features of each modality to the basic units in the hypergraph structure; a hyperedge refers to a structural unit that connects multiple nodes in the hypergraph and represents higher-order associations of multiple modalities.

[0174] Among them, the bidirectional updated hypergraph neural network refers to the network model after the nodes and hyperedge features have been iteratively updated.

[0175] In this embodiment, a hypergraph neural network is constructed based on the processed data and bidirectional updates of node hyperedges are completed. This fully explores multimodal high-order associations, implicitly completes missing modal features, improves the accuracy of feature fusion, and enhances the robustness of multimodal reasoning.

[0176] Optionally, in some embodiments of this application, constructing hypergraph nodes and hyperedges based on the data after missing modal processing, and determining the hypergraph neural network based on the hypergraph nodes and hyperedges, includes: representing the features of different modalities and different time steps in the data after missing modal processing as hypergraph nodes; constructing cross-modal hyperedges and temporal hyperedges based on the hypergraph nodes to characterize the high-order correlation between multiple modalities and obtain an initial hypergraph structure; and adaptively optimizing the initial hypergraph structure based on a variational hypergraph autoencoder to obtain a hypergraph neural network.

[0177] Here, a time step refers to a discrete time unit that divides sequence data according to the time dimension.

[0178] Hyperedges include cross-modal hyperedges and temporal hyperedges. Cross-modal hyperedges refer to hyperedges that connect nodes of hypergraphs of different modalities and represent the association between modalities; temporal hyperedges refer to hyperedges that connect nodes of hypergraphs of different time steps and represent temporal association.

[0179] Among them, higher-order association refers to the complex association between multiple elements formed by connecting multiple nodes with hyperedges; the initial hypergraph structure refers to the original hypergraph structure after constructing nodes and hyperedges without optimization.

[0180] Among them, Variational Hypergraph Auto-Encoder (VHGAE) refers to a model used for autoencoding adaptive optimization of hypergraph structures; adaptive optimization refers to the optimization operation in which the model automatically adjusts the hypergraph structure according to data features.

[0181] For example, the process of constructing hypergraph nodes and hyperedges based on the data after processing for missing modalities, and determining the hypergraph neural network based on the hypergraph nodes and hyperedges, is also called the hypergraph construction and modeling process. Specifically, it includes: S4100, representing the features of different modalities and different time steps as hypergraph nodes; S4200, constructing cross-modal hyperedges and temporal hyperedges to characterize the higher-order correlation between multiple modalities; S4300, adaptively optimizing the initial hypergraph structure to remove redundant connections, thereby improving modeling efficiency.

[0182] Specifically, in step S4100, the representation of a hypergraph node is a combination of a hypergraph node identifier and a feature vector (node ​​features) of a unified dimension. The hypergraph node identifier distinguishes different nodes and consists of the modality type (text, speech, video) and a specific time step (e.g., t1, t100). Taking a unified dimension of 128 dimensions as an example, the hypergraph node for text data at time step t1 can be represented as: “text_t1, (a1, b1, c1, ...)”; where “text_t1” is the hypergraph node identifier corresponding to the t1 time step of the text modality; and “(a1, b1, c1, ...)” is the unified dimension feature vector (128 dimensions) corresponding to the text data or the “learnable missing modality label”. Similarly, the hypergraph node for speech data at time step 2 can be represented as: speech_t2, (a2, b2, c2, ...); and the hypergraph node for video data at time step 3 can be represented as: video_t3, (a3, b3, c3, ...).

[0183] Specifically, in step S4200, a hyperedge can connect multiple nodes. The construction process of cross-modal hyperedge and temporal hyperedge (also known as the graph construction process) includes: (1) connecting all nodes of all modalities at the same time; (2) connecting all nodes of all time steps of the same modality.

[0184] In this embodiment, multimodal temporal features are set as hypergraph nodes and two types of hyperedges are constructed. The hypergraph is optimized by a variational hypergraph autoencoder to accurately characterize multimodal high-order temporal associations, improve the accuracy of hypergraph modeling, enhance the network's ability to capture complex associations, and ensure the effectiveness of subsequent feature inference.

[0185] Optionally, in some embodiments of this application, an adaptive optimization of the initial hypergraph structure based on a variational hypergraph autoencoder is performed to obtain a hypergraph neural network, including: convolving the initial hypergraph structure to generate embedding representations of hypergraph nodes and hyperedges, and encoding the mean and variance of the latent distribution to obtain a hypergraph encoding result; sampling based on the hypergraph encoding result using a reparameterization technique to obtain new latent embeddings of hypergraph nodes and hyperedges to obtain a hypergraph sampling result; calculating the dot product and reconstructing the hypergraph association matrix based on the hypergraph sampling result, and updating the hypergraph according to the hypergraph association matrix to obtain a hypergraph neural network.

[0186] Among them, hypergraph convolution refers to the operation of performing convolution on the initial hypergraph structure to extract features; embedding representation refers to the low-dimensional feature vector obtained after convolution of hypergraph nodes and hyperedges; latent distribution refers to the probability distribution that the hypergraph features follow in the latent space; and hypergraph encoding result refers to the data result obtained by this encoding step.

[0187] Among them, reparameterization technique refers to the parameter transformation method to solve the problem of non-differentiability of latent variable sampling; latent embedding refers to the feature vectors of hypergraph nodes and hyperedges obtained from latent space sampling; hypergraph sampling result refers to the data result obtained by this sampling step.

[0188] Among them, dot product refers to the operation of performing vector dot product on the sampling results of the hypergraph; hypergraph incidence matrix refers to the matrix that represents the connection relationship between hypergraph nodes and hyperedges; hypergraph update refers to the operation of adjusting the hypergraph structure based on the reconstructed incidence matrix.

[0189] For example, in step S4300, the adaptive optimization process refers to dynamically adjusting the hypergraph topology based on the correlation and importance of the node features of each node connected by the hyperedge. The specific steps include: (1) First, initialize a fully connected hypergraph structure, where each modality (text, audio, visual) of each utterance is a node and connected by two types of hyperedges: intramodal hyperedges (connecting nodes of the same modality in all utterances) and intermodal hyperedges (connecting nodes of different modalities in the same utterance). (2) Next, the core adjustment process is implemented through a variational hypergraph autoencoder (VHGAE). (3) Encoding: Convolve the initial hypergraph using a hypergraph neural network to generate embedding representations of nodes and hyperedges, and encode the mean and variance of their latent distribution. (4) Sampling: Sample from the above distribution using reparameterization techniques to obtain new latent embeddings of nodes and hyperedges, introducing randomness to explore a better structure. (5) Decoding: Using the new embedding obtained from sampling, the dot product is calculated and the Gumbel-Softmax function is applied to reconstruct the hypergraph's association matrix, thereby generating a new hypergraph with more concise and efficient connections. (6) Finally, in order to stabilize the uncertainty brought about by the above stochastic process, the model adopts contrastive learning: two slightly different hypergraph views are generated through two VHGAE paths with shared parameters, and the corresponding node representations (positive sample pairs) are brought closer and the non-corresponding node representations (negative sample pairs) are pushed further away, thereby enhancing the robustness of the model and ensuring the stability of the structural adjustment.

[0190] In this embodiment, the hypergraph is optimized by hypergraph convolution, reparameterized sampling, and reconstruction of the correlation matrix, which accurately characterizes the latent space features of the hypergraph, improves the rationality of the hypergraph structure, and enhances the network's adaptive modeling capability.

[0191] Optionally, in some embodiments of this application, bidirectional updates of hypergraph nodes and hyperedges are performed based on a hypergraph neural network, and inference is performed based on the bidirectionally updated hypergraph neural network to obtain multimodal joint feature representation data. This includes: performing information propagation and aggregation based on the hypergraph neural network to bidirectionally update hypergraph nodes and hyperedges; performing high-order inference in the bidirectionally updated hypergraph neural network using available modal information to implicitly complete missing modal features to obtain implicitly completed missing modal features; and obtaining multimodal joint feature representation data based on the implicitly completed missing modal features.

[0192] Information propagation refers to the process of feature information exchange and transmission between nodes and hyperedges in a hypergraph; aggregation refers to the operation of merging and integrating multi-source information propagated in a hypergraph.

[0193] Among them, available modal information refers to the modal feature information that is completely present in the data after missing modal processing; implicit completion processing refers to the operation of indirectly restoring missing modal features based on available modal information. Naturally, the missing modal features after implicit completion refer to the feature data obtained after completion by higher-order inference.

[0194] For example, the process of bidirectionally updating hypergraph nodes and hyperedges based on a hypergraph neural network, and then performing inference based on the bidirectionally updated hypergraph neural network to obtain multimodal joint feature representation data, is also known as the hypergraph neural network inference process. Specifically, it includes: performing information propagation and aggregation based on the hypergraph neural network (i.e., the adaptively optimized hypergraph structure) to achieve bidirectional updates between nodes and hyperedges; performing high-order inference in the hypergraph using available modal information to implicitly complete missing modal features; and obtaining robust multimodal joint representations for subsequent sentiment discrimination.

[0195] In this embodiment, bidirectional updates are achieved through hypergraph information propagation and aggregation. Missing features are implicitly completed by relying on available modal high-order inference, and multimodal information is fully integrated to improve the accuracy of feature completion and enhance the robustness of the model.

[0196] Optionally, in some embodiments of this application, information propagation and aggregation are performed based on a hypergraph neural network, and bidirectional updates are performed on hypergraph nodes and hyperedges, including: using the features of hypergraph nodes, the features of hyperedges, and the weights of hyperedges as inputs to the hypergraph neural network; updating the features of hyperedges by aggregation and mapping of a multilayer perceptron for the features of all hypergraph nodes connected to any hyperedge; and updating the features of hypergraph nodes by aggregation and mapping of a multilayer perceptron for the features of all hyperedges associated with any hypergraph node.

[0197] The weight of a hyperedge refers to a value used to characterize the importance of the hyperedge and adjust the weight of information propagation.

[0198] Updating the features of a hyperedge refers to replacing the original hyperedge features with the result of the node aggregation mapping; updating the features of a hypergraph node refers to replacing the original node features with the result of the hyperedge aggregation mapping.

[0199] For example, the steps involved in the hypergraph neural network inference process may be: (1) taking the optimized node features, hyperedge features, and hyperedge weights of the hypergraph as input to the hypergraph neural network; (2) updating the hyperedge feature vector based on all node features connected by the hyperedge through aggregation (average calculation) and MLP mapping; (3) updating the node feature vector again based on all hyperedge features associated with the node through aggregation and MLP mapping; (4) taking the automatically completed modal features after the update as a multimodal joint representation for subsequent sentiment discrimination.

[0200] In this embodiment, the input node hyperedge features and weights are aggregated and mapped to complete the bidirectional update of nodes and hyperedges, which enhances the information interaction within the hypergraph and improves the accuracy of feature updates.

[0201] Optionally, in some embodiments of this application, emotion recognition is performed based on multimodal joint feature representation data to obtain emotion judgment result data associated with target data, including: inputting multimodal joint feature representation data into a classification network with a preset network complexity; outputting original emotion scores for various preset emotion categories based on the activation function of the classification network; normalizing the original emotion scores to obtain probability distribution data for various preset emotion categories; and determining the emotion judgment result data associated with target data based on the probability distribution data.

[0202] Among them, preset network complexity refers to the pre-defined structural complexity index of the classification network; classification network refers to the neural network used to determine the sentiment category of multimodal joint features; classification network with preset network complexity refers to a network with low complexity, such as MLP and Support Vector Machine (SVM) networks.

[0203] For example, in the optimization process of the classifier structure, a unified classifier structure that is independent of the number of modes is adopted, that is, a simple classifier, such as SVM, is used to reduce the risk of overfitting, speed up the convergence speed, and enable the model to directly output results under the conditions of complete modes and the absence of any whole modes; the classifier only depends on the joint representation after hypergraph inference, avoiding strong dependence on the existence of specific modes.

[0204] Here, the original sentiment score refers to the unnormalized score of each sentiment category directly output by the classification network; the probability distribution data refers to the probability value data corresponding to each preset sentiment category after normalization. The preset sentiment categories include, but are not limited to, happiness, sadness, confusion, and calmness.

[0205] For example, the node representations corresponding to each modality are fused (hypergraph aggregation), and the multimodal node features after hypergraph aggregation are directly concatenated to form a unified multimodal feature representation. The corresponding sentiment category results are output through a classification network (such as a simple, easily convergent MLP). Sentiment recognition is supported under conditions of complete modalities and arbitrary modality absence. The specific data form of the sentiment category results is a fixed-dimensional probability vector, whose dimension is consistent with the preset total number of sentiment categories. Each element in the probability vector corresponds to the predicted probability of a sentiment category, with a value between 0 and 1, and the sum of all elements is 1.

[0206] More specifically, the probability vector can be generated by combining the output layer of the classification network with an activation function. First, the raw sentiment scores for various preset sentiment categories are obtained, and then normalized to form a probability distribution. Finally, the category corresponding to the element with the highest probability is selected as the final sentiment determination result. For example, if four sentiment categories are preset—happiness, sadness, confusion, and calmness—the output vector might be (p1, p2, p3, p4). If p3 is the maximum probability among these, then the final sentiment category is determined to be confusion.

[0207] In this embodiment, the joint features are input into the classification network to output the original score. After normalization to obtain the probability distribution, the emotion result is determined, which improves the accuracy of emotion recognition, enhances the rationality of the scoring result, and ensures the reliability and stability of emotion determination.

[0208] Optionally, in some embodiments of this application, before the step of encoding the target data to obtain encoded data, the method further includes: performing unified time alignment processing on each modal data in the target data to obtain time-aligned data; and performing length normalization processing on the time-aligned data to obtain length-normalized target data.

[0209] Among them, time alignment processing refers to the preprocessing operation of matching each modal data to the same time reference according to the time dimension; length normalization processing refers to the preprocessing operation of adjusting each modal data to a preset uniform length.

[0210] For example, during the preprocessing optimization process, before generating a single-modal representation with a uniform dimension, the inputs of each modality are subjected to uniform time alignment and length normalization to reduce the impact of different modal sampling rates and temporal differences on subsequent modeling.

[0211] In this embodiment, time alignment and length normalization preprocessing are performed on each modality data in sequence to unify the data time base and length dimension, thereby enhancing the adaptability of multimodal data.

[0212] Optionally, in some embodiments of this application, after obtaining the multimodal target data to be used for emotion recognition, the method further includes: performing modality missing detection on the target data to obtain a modality missing detection result; and if the modality missing detection result indicates that the preset modality is completely missing, directly outputting a learnable missing modality token to determine the data after missing modality processing.

[0213] Among them, the modality missing detection result is the overall missing of a preset modality, which means that a certain modality is completely missing; direct output means that the corresponding data is generated directly and transmitted outward without additional feature processing.

[0214] For example, during the optimization process of preprocessing, when a certain modality is detected to be missing in its entirety, a missing identifier (a learnable missing modality token) is directly output to avoid invalid features from participating in subsequent calculations (such as the encoding process).

[0215] In this embodiment, modal missing detection is first performed on the target data. When overall modal missing is detected, a learnable missing token is directly output, which simplifies the missing data processing process, improves the efficiency of data preprocessing, enhances the model's adaptability to completely missing modalities, and ensures the continuity of subsequent processing.

[0216] Optionally, in some embodiments of this application, before the step of encoding the target data to obtain encoded data, the method further includes: performing modal missing detection on the target data to obtain modal missing detection results; if the modal missing detection results indicate that the preset modality is completely missing, performing a pause on the feature extraction function of the encoder corresponding to the missing modality, and outputting placeholder representation data corresponding to the missing modality as encoded data.

[0217] Among them, the missing modality encoder refers to the network module specifically used to encode the features of the missing modality data; the pause operation refers to the operation of temporarily stopping the encoder feature extraction function; and the placeholder representation data refers to the preset feature data used to replace the missing modality encoding features.

[0218] For example, during the encoder optimization process, independent encoder structures are set for different modalities to fully model the differences in feature distribution among the modalities; specifically, an optimal encoder is designed for the characteristics of each modality, such as an encoder adapted for semantic feature extraction for text modality, an encoder suitable for acoustic feature capture for speech modality, and an encoder for visual feature analysis for video modality.

[0219] When a certain modality is completely missing, the corresponding encoder does not participate in feature extraction, but only outputs a placeholder representation corresponding to the missing state; the placeholder representation is different for different modalities.

[0220] In this embodiment, the entire modality is detected as missing before encoding and the corresponding encoder is paused. The output placeholder is used as the encoding data, which reduces invalid calculations, improves encoding processing efficiency, and enhances the flexibility and adaptability of the model.

[0221] Optionally, in some embodiments of this application, a hypergraph neural network is constructed based on the data after missing modality processing, and bidirectional updates are performed based on the hypergraph neural network to infer multimodal joint feature representation data. This includes: constructing hypergraph nodes and hyperedges based on the currently available modality data in the data after missing modality processing; determining the hypergraph neural network based on the hypergraph nodes and hyperedges; establishing high-order associations for the hypergraph nodes and hyperedges corresponding to the currently available modality data to perform bidirectional updates on the hypergraph neural network; and performing inference based on the bidirectionally updated hypergraph neural network to obtain multimodal joint feature representation data.

[0222] Among them, currently available modal data refers to the modal feature data that actually exists and can be used in the data after missing modal processing.

[0223] For example, in the adaptive optimization process of the hypergraph structure, the hypergraph node and hyperedge structure are dynamically constructed according to the currently available modes, and high-order associations are established only for existing modes to reduce the computational cost of bidirectional updates.

[0224] In this embodiment, a hypergraph is constructed based on the currently available modal data and a high-order association is established to complete bidirectional updates. This focuses on modeling effective modal information, improving the accuracy of feature association mining and increasing the efficiency of data computation.

[0225] Optionally, in some embodiments of this application, constructing a hypergraph neural network based on the data after missing modal processing includes: monitoring the changes in the modal combination state of the target data to obtain a monitoring result of the changes in the modal combination state; and when the monitoring result of the changes in the modal combination state indicates that the modal combination state has changed, adaptively adjusting based on a preset optimization hypergraph structure strategy to construct a hypergraph neural network.

[0226] Among them, the monitoring result of modal combination state change refers to the result of whether the modal combination has changed after monitoring; the optimization strategy of hypergraph structure refers to the pre-set strategy for adjusting the hypergraph structure to adapt to data changes; and the adaptive adjustment refers to the dynamic optimization operation of automatically adjusting the hypergraph structure according to the changes in modal combination.

[0227] For example, in the adaptive optimization process of a hypergraph structure, when modality combinations change, the hypergraph structure can adaptively adjust based on its inherent functions, such as hypergraph structure optimization strategies, without requiring manual redefinition of connection rules. Here, "changes in modality combinations" refers to the possibility that the data acquisition method or quality may change over time. For instance, an interruption in speech acquisition might result in complete speech acquisition at earlier time steps, while data at later time steps is missing.

[0228] In this embodiment, the hypergraph structure is adaptively adjusted by monitoring changes in modality combination states, enabling the hypergraph neural network to adapt to changes in modality combination and improving the model's dynamic adaptation capability.

[0229] Optionally, in some embodiments of this application, a hypergraph neural network is constructed based on the data after processing the missing modalities, and bidirectional updates are performed based on the hypergraph neural network to infer multimodal joint feature representation data. This includes: constructing hypergraph nodes and hyperedges based on the data after processing the missing modalities; determining the hypergraph neural network based on the hypergraph nodes and hyperedges; performing bidirectional updates on the hypergraph nodes and hyperedges based on the hypergraph neural network; performing dimension alignment and scale normalization processing on the feature data of each modality in the bidirectionally updated hypergraph neural network to obtain dimension-aligned and scale-normalized feature data; and performing inference based on the hypergraph neural network constructed from the dimension-aligned and scale-normalized feature data to obtain multimodal joint feature representation data.

[0230] Among them, dimension alignment refers to the operation of adjusting the feature data of each modality to the same dimension for feature unification; scale normalization refers to the operation of calibrating the numerical scale of each modality feature to the same range.

[0231] For example, in the optimization process of multimodal feature fusion, it is preferable to perform dimension alignment and scale normalization on each modality feature before hypergraph inference. Considering that there may be time delay between modalities, the focus is on realigning in time to ensure the stability of cross-modal information transmission.

[0232] In addition, in this embodiment, the complementary fusion of multimodal features is achieved by modeling high-order relationships in a hypergraph, rather than by a simple splicing method that relies on fixed weights.

[0233] In this embodiment, a hypergraph is constructed and bidirectional updates of node hyperedges are completed. Inference is performed after dimension alignment and scale normalization of features, unifying the dimensions and scale of multimodal features, thus ensuring the stability and effectiveness of multimodal inference.

[0234] Figure 3 This is a flowchart illustrating one embodiment, such as... Figure 3 As shown, the model training method provided in this application embodiment is used to train the artificial intelligence model involved in the multimodal emotion recognition method based on hypergraph neural networks under missing modalities, specifically including:

[0235] S3100, construct a teacher model under the condition of modal completeness of the target data, and construct a student model under the condition of modal missingness of the target data.

[0236] Among them, modal integrity condition refers to the task condition in which all modalities in the target data are complete and without missing parts; teacher model refers to the benchmark model trained under modal integrity condition and possessing high-quality feature representation ability.

[0237] Among them, the modality missing condition refers to the task condition in which some modalities are missing in the target data; the student model refers to the model to be optimized that is trained under the modality missing condition and needs to learn the ability of the teacher model.

[0238] S3200, based on a feature-level knowledge transfer mechanism, trains the student model according to the emotional representation distribution of the teacher model.

[0239] Among them, the feature-level knowledge transfer mechanism refers to the mechanism of transferring the knowledge of the teacher model to the student model at the feature level.

[0240] Compared to traditional technologies, in this embodiment, a teacher model under modal integrity conditions of the target data and a student model under modal missing conditions of the target data are first constructed. Then, based on a feature-level knowledge transfer mechanism, the student model is trained according to the emotion representation distribution of the teacher model. The technical solution of this embodiment introduces missing modality labeling and a feature-level knowledge transfer mechanism to improve the stability and consistency of emotion recognition under missing modality conditions, maintain a certain recognition performance under modal integrity conditions, and improve the training effect of the model without sacrificing the effect of complete modality in exchange for robustness.

[0241] Optionally, in some embodiments of this application, based on a feature-level knowledge transfer mechanism, the student model is trained according to the sentiment representation distribution of the teacher model, including: extracting sentiment representation features of the same batch of samples from both the teacher model and the student model; calculating the distribution difference data between the feature distribution of the student model and the feature distribution of the teacher model based on the sentiment representation features of the same batch of samples; determining a knowledge transfer loss function based on the distribution difference data; constructing an overall loss function for the student model based on the knowledge transfer loss function; and updating various model parameters of the student model through backpropagation based on the overall loss function.

[0242] The same batch of samples refers to the same set of data samples that are simultaneously input into the teacher model and the student model for feature extraction.

[0243] Among them, the distribution difference data refers to the numerical data that quantitatively represents the degree of difference in the distribution of student and teacher model characteristics.

[0244] Among them, the knowledge transfer loss function refers to the loss calculation function that measures the difference in feature distribution during the knowledge transfer process; the overall loss function refers to the total loss function of the student model that integrates the knowledge transfer loss and the task loss.

[0245] Here, model parameters refer to the various weights and biases in the student model that can be adjusted through training iterations. Specifically, the various model parameters of the student model include the model parameters of the artificial intelligence model involved in the multimodal emotion recognition method based on hypergraph neural networks under missing modalities. This artificial intelligence model can refer to the overall model built or integrated from multiple sub-models, or it can refer to all the sub-models involved. More specifically, the various model parameters of the student model include the encoding model used in the encoding process, the models involved in the construction, updating, and inference processes of the hypergraph neural network, and the model parameters of the classification model in the emotion recognition process.

[0246] For example, the process of training a student model based on the sentiment representation distribution of the teacher model using a feature-level knowledge transfer mechanism is also known as the knowledge distillation process or the knowledge transfer and alignment process. Specifically, this includes: constructing a teacher model under modality-complete conditions and a student model under modality-missing conditions; enabling the student model to learn the sentiment representation distribution of the teacher model through a feature-level knowledge transfer mechanism; and improving the consistency and stability of sentiment representation under modality-missing conditions.

[0247] More specifically, the transfer and alignment process may include: (1) extracting the sentiment representation features of the same batch of samples from the teacher model and the student model respectively; (2) calculating the difference between the feature distribution of the student model and the feature distribution of the teacher model (e.g., mean squared error); (3) using the difference in distribution as the knowledge transfer loss, combining it with the task loss of the student model (e.g., sentiment classification loss) to construct a total loss function, which together guides the training of the student model; (4) updating the parameters of the student model through backpropagation based on the total loss function.

[0248] In this embodiment, features of samples from the same batch are extracted and distribution differences are calculated. After constructing a loss function, parameters are updated via backpropagation, achieving accurate feature-level knowledge transfer, improving the training efficiency of the student model, and ensuring recognition accuracy under modality missing conditions.

[0249] Optionally, in some embodiments of this application, the overall loss function of the student model is constructed based on the knowledge transfer loss function, including: constructing the classification task loss function corresponding to the multimodal emotion classification process; constructing the variational hypergraph autoencoder loss function and the cross-view comparison loss function corresponding to the construction and update process of the hypergraph neural network; and determining the overall loss function of the student model by performing a weighted summation based on the classification task loss function, the variational hypergraph autoencoder loss function, the cross-view comparison loss function, and the knowledge transfer loss function.

[0250] Among them, the classification task loss function refers to the function that measures the deviation between the sentiment classification prediction result of the classification model and the actual result; the variational hypergraph autoencoder loss function refers to the function that measures the structural reconstruction deviation during the construction and updating of the hypergraph; and the cross-view contrast loss function refers to the contrast loss calculation function that measures the feature consistency between different views in a multimodal model.

[0251] For example, in the optimization process of the loss function design, in addition to the sentiment classification loss, a feature alignment constraint based on the complete modality model is introduced to improve representation consistency under missing modality conditions; different training objectives are weighted and combined to balance classification performance and representation stability. Here, the feature alignment constraint of the complete modality model refers to minimizing the difference between the sentiment features extracted by the complete modality model and the missing modality model using kl loss.

[0252] The specific formula for calculating the training objective (overall loss function) is as follows:

[0253] ;

[0254] in, , , These are the weighting coefficients for the corresponding losses. Specifically, the overall loss function includes:

[0255] (1) The loss function for the classification task, denoted as It represents the average loss of multiple missing cases and is used to handle multimodal emotion classification tasks. It gives higher weight to difficult samples and improves classification accuracy.

[0256] (2) The loss function of the variational hypergraph autoencoder is denoted as... Also known as VHGAE loss, it is the loss generated by the VHGAE module; it consists of a KL divergence term and a reconstruction term; it is used to constrain the generation and feature reconstruction of the hypergraph structure, ensuring that the hypergraph can effectively model multimodal high-order associations.

[0257] (3) Contrastive Loss, denoted as This is used to promote consistency in feature representation across different views and enhance the model's generalization ability.

[0258] (4) Knowledge transfer loss function, denoted as Also known as Knowledge Distillation Loss, it refers to the KL divergence of the feature vectors extracted from the complete modality model and the missing modality model. By measuring the difference in feature distribution between the complete modality model and the missing modality model through KL divergence, knowledge transfer is achieved and the performance in missing modality scenarios is improved.

[0259] In this embodiment, an overall loss function is constructed to train the multi-dimensional constrained model, which improves the training accuracy of the student model, enhances the model's feature representation and classification capabilities, and ensures the effect of emotion recognition under modality loss.

[0260] The technical research process and other technical details of this application are described below with reference to a specific embodiment.

[0261] In traditional techniques, the following shortcomings exist in addressing the modality loss problem in multimodal emotion recognition: (1) Simulating modality loss through data augmentation makes it difficult to effectively compensate for missing modal information during the inference stage; (2) Using generation or reconstruction mechanisms to infer missing modalities results in complex model structures, unstable training, and high computational costs; (3) Methods based on joint representation or modality transformation rely heavily on pairwise modal relationships, making it difficult to model high-order associations between multimodalities and insufficiently adaptable to complex missing patterns.

[0262] Based on this, this application provides a multimodal emotion recognition method for missing modalities based on hypergraph neural networks, which can also be called a multimodal emotion recognition method for missing modalities robust based on hypergraph neural networks. For ease of description in the embodiments, it can also be simply referred to as the emotion recognition method of this application. The details are as follows.

[0263] The emotion recognition method presented in this application is specifically applied to the field of multimodal emotion computing technology. Addressing the problem of missing multimodal data in scenarios such as online education due to privacy restrictions, environmental noise, and network instability, it extracts features from available text, speech, and video modal information and constructs a hypergraph structure to represent multimodal relationships. By establishing high-order cross-modal relationships within the hypergraph, and utilizing the information propagation mechanism of the hypergraph neural network, it infers and completes the feature representations of missing modalities from existing modalities, thereby obtaining a unified multimodal emotion representation.

[0264] Furthermore, the emotion recognition method of this application introduces a knowledge transfer mechanism, enabling the model under missing modality conditions to learn the discriminative ability of complete modality emotion features. Finally, it outputs the corresponding emotion recognition result.

[0265] The emotion recognition method proposed in this application does not require separate training of models for different modal missing modes. It can maintain stable recognition performance under both complete and incomplete modal conditions, and is suitable for application scenarios such as learning status and emotion perception in online education.

[0266] The emotion recognition method of this application has the following technical advantages: (1) It models multimodal high-order associations based on hypergraph structure, which improves the cross-modal information utilization capability compared with pairwise modality modeling methods. (2) In the case of modality missing, the missing modality features are inferred through the hypergraph information propagation mechanism to avoid a significant decrease in recognition performance. (3) It adopts a unified model structure to adapt to any combination of missing modalities, without the need to train models separately for different missing modes. (4) It introduces missing modality labeling and feature-level knowledge transfer mechanism to improve the stability and consistency of emotion recognition under missing modality conditions. (5) It maintains recognition performance comparable to existing methods under complete modality conditions, without sacrificing the effect of complete modality for robustness. (6) It is applicable to real-world scenarios such as online education and can still operate stably under conditions of restricted privacy or unstable network.

[0267] The experimental results of the emotion recognition method of this application are described in detail below with reference to Tables 1 to 3.

[0268] Table 1:

[0269]

[0270] Table 1 is used to show the accuracy comparison of the two datasets under different modal missing conditions. The bold text indicates the best performance. In Table 1, Methods 1 to 8 are the experimental methods in the traditional technology involved in this embodiment. The first dataset is the "MELD" dataset and the second dataset is the "IEMOCAP" dataset. "Text" can be denoted as "(t)", "text and video" can be denoted as "(t,v)", "text and audio" can be denoted as "(t,a)", and "text, video and audio" can be denoted as "(t,a,v)", which represents the combination state of different modal data. For example, "(t,a,v)" is a combination in which text, audio and video modal data are all complete.

[0271] Table 2:

[0272]

[0273] Table 2 shows the ablation experiments of the proposed framework components; bold text indicates the best performance for each dataset.

[0274] Table 3:

[0275]

[0276] Table 3 shows the comparison under full modal conditions; Methods A to E are the experimental methods in the conventional technology involved in this embodiment; the first dataset is the “MELD” dataset and the second dataset is the “IEMOCAP” dataset.

[0277] Referring to Tables 1 to 3, in terms of experiments, the emotion recognition method proposed in this application maintains stable performance in multimodal emotion recognition tasks under various modality missing conditions, with an average recognition accuracy of approximately 0.66-0.69. It can still effectively complete emotion discrimination even when only one or some modalities are available. Under modality complete conditions, the system's recognition accuracy and F1 score are at the same or better level than existing methods. Experimental results show that the modality completion mechanism based on hypergraph modeling can effectively alleviate the performance degradation caused by modality missing data and is suitable for emotion analysis tasks with incomplete multimodal data in practical scenarios such as online education.

[0278] In summary, the emotion recognition method of this application has the following advantages: it avoids training multiple models for different modality loss scenarios, using a single model to solve multiple modality loss scenarios, resulting in low training costs; it is suitable for large-scale application scenarios such as online education and corporate training, with controllable promotion costs; the system is implemented in software, facilitating rapid integration into existing teaching or management platforms. It helps achieve continuous perception and early intervention of emotional states in scenarios such as online education; it reduces reliance on manual emotion assessment, alleviating the problem of insufficient professional resources; and it is non-invasive, requiring no additional intervention, meeting society's demand for privacy-friendly and sustainable intelligent perception technologies. Furthermore, it requires no additional sensors worn by the user or additional data collection permissions; the system only relies on the text, voice, and video data that the terminal device can originally obtain.

[0279] In another embodiment, the implementation of the emotion recognition method of this application also has the following alternatives.

[0280] (1) Alternative model structure: In the implementation of the hypergraph neural network, other graph learning models or their equivalent variants that can model higher-order relationships can be used to model the complex relationships between multimodal features; the specific number of layers, parameter scale and update method of the model do not constitute a limitation of this application.

[0281] (2) Modal type alternatives: The multimodal emotion recognition method described in this application is not limited to text, audio and video modalities, but can also be extended to physiological signals, behavioral trajectories or other modal inputs that can reflect emotional states, and is still applicable when some or all modalities are missing.

[0282] (3) Alternatives for missing modal representation: For modalities that are completely missing, their placeholder representation can be a fixed vector (for simplicity, computational efficiency and stable training), a learnable vector, or an equivalent representation obtained by mapping other modalities. The specific implementation does not affect the core technical effect of this application. For modalities that are completely missing, their placeholder representation can be a fixed vector, combined with "implicit completion of missing modal features". At this time, the completion can still be performed normally. Multiple completion mechanisms are coupled with each other to achieve the same effect.

[0283] (4) Alternative solutions for feature fusion: In addition to hypergraph reasoning, the fusion process of multimodal features can also be achieved through equivalent high-order relation modeling or unified representation learning. As long as cross-modal information compensation can be completed when the overall modality is missing, it should be regarded as an equivalent alternative solution of this application.

[0284] (5) Alternative application scenarios: In addition to online education scenarios, this application can also be deployed in application environments where there is a risk of overall data loss in multimodal applications such as remote conferencing, human-computer interaction, and intelligent customer service. Its specific application areas do not constitute a limitation on this application.

[0285] In another embodiment, the emotion recognition method of this application can be applied to the following scenarios.

[0286] (1) In online education and distance learning scenarios, monitor students' emotional state in the classroom (such as focus, confusion, fatigue). Assist teachers in learning intervention and personalized teaching (if sadness is detected, the teacher assumes the student is encountering difficulties). Adapt to the loss of video or audio due to privacy or network issues.

[0287] (2) In the context of remote conferencing and collaboration platforms, analyze the emotions of participants in real time (such as interest, satisfaction, and anxiety). Provide emotional feedback to assist in meeting management or discussion optimization (if most people have negative emotions, it can be assumed that most people do not agree with the current content). It can handle situations where some microphones or cameras fail or the network is interrupted.

[0288] (3) In intelligent customer service and call center scenarios, identify the emotional state (such as anger, confusion, satisfaction) in the customer's voice and text. Dynamically adjust customer service strategies or automatically push response content (if anger is detected, use more friendly language). It can still complete emotion judgment even in the absence of voice or video input.

[0289] (4) In human-computer interaction and intelligent assistant scenarios, smart home assistants, robots, or virtual humans perceive user emotions. They adjust interaction strategies or recommended content based on the user's current mood (e.g., providing comfort if low mood is detected). They adapt to situations where some perceptual modalities are missing, such as camera obstruction or voice recognition failure.

[0290] (5) In the context of mental health and emotion monitoring, conduct emotion assessments based on multimodal data (voice, facial expressions, text) of patients or users. Provide real-time mental health tips or emotion regulation suggestions (if anxiety is detected, guide the user to relax). The monitoring capability can be maintained even when some physiological signals are missing.

[0291] (6) In the context of corporate training and employee status monitoring, assess trainee participation and emotional fluctuations during remote training courses. Optimize training content and interaction methods (provide reminders when negative employee behavior is detected). Support emotion recognition under different device conditions, such as text or audio input only.

[0292] (7) In the context of intelligent driving and in-vehicle environment monitoring, monitor the driver's emotional state (such as fatigue, anxiety, anger) to improve driving safety. Even when the vehicle camera or microphone fails, the driver's state can still be determined through the available modalities.

[0293] (8) Analyze user emotions and engagement in virtual reality / augmented reality scenarios, VR / AR games, or education. Adjust the difficulty of the virtual environment or tasks based on user emotions (reduce the game difficulty if user impatience is detected). Can handle situations where some sensors are lost or the signal is unstable.

[0294] (9) In social media and online interaction analysis scenarios, analyze the sentiment of text, voice, or video content uploaded by users. Provide content recommendations or public opinion analysis (if a user is detected to be angry about a certain type of content, reduce recommendations and extract the user's tag to label that type of content, thus reducing the correlation). Sentiment recognition can be completed when unimodal data is available.

[0295] (10) In the context of health rehabilitation and sports training, assess the psychological state of the patient through multimodal perception (facial expression, voice, movement trajectory, physiological signals). Adjust training plans and rehabilitation strategies (if the user (athlete) is found to be emotionally agitated during training, adjust the training content and reduce risky strength training). Support scenarios where some perception modalities are missing, such as when video or voice is unavailable.

[0296] In another embodiment, the advantage of the emotion recognition method of this application is that:

[0297] (1) Innovation in hypergraph structure modeling: Hypergraph nodes are used to represent multimodal features at different time steps to achieve high-order cross-modal correlation modeling. Cross-modal hyperedges and temporal hyperedges are constructed to enable the inference of missing modal features from single modal or partial modal information. Adaptive optimization of the hypergraph structure is supported, dynamically adjusting nodes and hyperedges according to the actual available modalities without the need for manual design.

[0298] (2) Introduction of missing modality robustness mechanism: Learnable missing modality label vectors are introduced to replace missing features, enabling a unified model to handle arbitrary combinations of missing modalities. A modality masking strategy is adopted during the training phase to enable the model to adapt to different missing modes. This avoids the performance degradation problem of traditional zero-filling or feature discarding methods.

[0299] (3) Knowledge Transfer and Feature Alignment Innovation: Within the teacher-student model framework, feature-level knowledge transfer is performed on the student model under the missing modality condition using the teacher model under the complete modality condition. This ensures the consistency between the sentiment representation under the missing modality condition and the representation under the complete modality condition in the feature space, improving recognition stability. It can be combined with a multi-task loss function to achieve a balance between classification performance and representation consistency.

[0300] (4) Unified multimodal emotion recognition model setup: The same model structure can handle both complete and arbitrary missing modalities simultaneously, eliminating the need to train multiple models for different missing modalities. The classifier is independent of the number of modalities, relying solely on the joint representation after hypergraph inference to output the emotion category. It supports the expansion of multimodal input types, including text, speech, video, physiological signals, and motion trajectories.

[0301] (5) Multimodal information complementarity and inference innovation: Through hypergraph information propagation, inference and completion between higher-order modalities are achieved, rather than relying solely on simple splicing or weighted fusion. Cross-modal information utilization capabilities can be retained even when a single modality or some modalities exist. This improves the generalization and robustness of the model under complex missing modes.

[0302] (6) System Deployment and Application Flexibility: The modular system design allows for deployment on the server or local end, adapting to different privacy and network conditions. It is applicable to a wide range of scenarios, including online education, remote conferencing, intelligent customer service, human-computer interaction, autonomous driving, VR / AR, and mental health monitoring. It supports continuous training and incremental updates, enhancing the system's adaptability in dynamic environments.

[0303] In addition, in some embodiments, corresponding to the multimodal emotion recognition method based on hypergraph neural networks under missing modalities in the above embodiments, this application also provides an emotion recognition system for implementing the emotion recognition method of this application, specifically including:

[0304] (1) Data acquisition module, used to acquire existing multimodal feature data through terminal equipment or platform interface; to mark the status of missing modes, providing a basis for subsequent processing.

[0305] (2) Single-modal feature encoding module, which is used to encode the features of each modality data separately and generate a single-modal representation with a unified dimension; speaker or context embedding is introduced to enhance the expressive power of single-modal features.

[0306] (3) Missing modality processing module, which is used to replace the original feature input with learnable missing modality labels when a certain modality is detected; through the modality masking mechanism in the training phase, the model can adapt to multiple combinations of missing modalities;

[0307] The "modal masking mechanism" refers to randomly masking one or more modalities from the complete multimodal data during training, replacing the training sample data with "learnable missing modality labels." Essentially, this means that the training dataset is a pre-defined mixture of complete and various missing sample data. This ensures network structure consistency under both complete and incomplete modal conditions, avoiding zero-padding or feature loss.

[0308] (4) Hypergraph construction and modeling module, used to represent the features of different modalities and different time steps as hypergraph nodes; construct cross-modal hyperedges and temporal hyperedges to characterize the high-order association between multiple modalities; and adaptively optimize the initial hypergraph structure to remove redundant connections.

[0309] (5) Hypergraph Neural Network Inference Module: This module is used to propagate and aggregate information based on the hypergraph neural network, and realize bidirectional updates between nodes and hyperedges; it uses available modal information to perform high-order inference in the hypergraph, implicitly completes missing modal features; and obtains robust multimodal joint representations for subsequent sentiment discrimination.

[0310] (6) Knowledge transfer and alignment module, used to construct teacher model under modality complete condition and student model under modality missing condition; through feature-level knowledge transfer mechanism, the student model learns the emotional representation distribution of teacher model.

[0311] (7) Emotion recognition module, which is used to fuse the node representations corresponding to each modality to form a unified multimodal feature representation; output the corresponding emotion category results through the classification network; and support emotion recognition under conditions of complete modality and absence of any modality.

[0312] In addition, the system deployment method of the emotion recognition system is as follows: the system is implemented in a modular way and can be integrated into existing online education or interactive platforms as software components; it supports centralized inference on the server side or local deployment to meet the needs of privacy protection and large-scale application; the model supports continuous training and updating to adapt to changes in data distribution in different application scenarios.

[0313] Figure 4 This is an overall system architecture diagram in one embodiment. In one embodiment, the overall architecture of the emotion recognition system is as follows: Figure 4 As shown.

[0314] Figure 5 This is a schematic diagram illustrating the principle of knowledge distillation in one embodiment. In one embodiment, the principle of the knowledge distillation process of this application is as follows: Figure 5 As shown.

[0315] It should be noted that any technical feature in any of the above embodiments provided in this application is also applicable to any of the following embodiments provided in this application, and similar details will not be repeated hereafter.

[0316] Figure 6 This is a structural block diagram of a multimodal emotion recognition device based on a hypergraph neural network under missing modalities in one embodiment, with reference to... Figure 6 The multimodal emotion recognition device based on hypergraph neural networks for missing modalities includes:

[0317] The data acquisition and encoding module 601 is used to acquire multimodal target data to be used for emotion recognition and to encode the target data to obtain encoded data.

[0318] The missing modality processing module 602 is used to perform missing modality processing on the encoded data to obtain the missing modality processed data;

[0319] The hypergraph neural network construction and update module 603 is used to construct a hypergraph neural network based on the data after processing the missing modalities, perform bidirectional updates based on the hypergraph neural network, and infer multimodal joint feature representation data.

[0320] The emotion recognition module 604 is used to perform emotion recognition based on multimodal joint feature representation data to obtain emotion judgment result data associated with the target data.

[0321] In this embodiment of the application, based on, as follows Figure 6 The connections between the modules or units shown in the diagram enhance the stability and robustness of multimodal emotion recognition in application scenarios where a modality is missing.

[0322] Figure 7 Here is a structural block diagram of a model training device in one embodiment, with reference to Figure 7 The model training device includes:

[0323] The model building module 701 is used to build a teacher model under the condition of modal completeness of the target data, and to build a student model under the condition of modal missingness of the target data;

[0324] The knowledge transfer module 702 is used to train the student model based on the emotion representation distribution of the teacher model, using a feature-level knowledge transfer mechanism.

[0325] In this embodiment of the application, based on, as follows Figure 7 The connections between the modules or units shown in the diagram improve the model training effect through the cooperation between these modules or units.

[0326] In another embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 8As shown, it includes a processor, memory, input / output interfaces, and a communication interface. The processor, memory, and input / output interfaces are connected via a system bus, and the communication interface is connected to the system bus via the input / output interfaces. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The database stores relevant data. The input / output interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communication with external terminals via a network connection. The computer program can be executed by the processor to implement the various methods described in the above embodiments.

[0327] In yet another embodiment, a computer device is provided, such as a terminal, whose internal structure diagram may be as follows: Figure 9 As shown, it includes a processor, memory, input / output interface, communication interface, display unit, and input device. The processor, memory, and input / output interface are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interface. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The input / output interface is used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. The computer program can be executed by the processor to implement the various methods described in the above embodiments.

[0328] Those skilled in the art will understand that Figure 8 and Figure 9 The structure shown is only a block diagram of a part of the structure related to the solution of this application, and does not constitute a limitation on the computer device on which the solution of this application is applied. It may also include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements, in order to realize the function of the terminal or server.

[0329] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0330] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the systems, devices, equipment, modules or units described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0331] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, devices, or methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or modules may be electrical, mechanical, or other forms.

[0332] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0333] Furthermore, the functional modules in the various embodiments of this application can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can be stored in a computer-readable storage medium.

[0334] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0335] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, optical fiber) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a server or data center that integrates one or more available media. The available medium may be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium, or a semiconductor medium (e.g., a solid-state drive), etc.

[0336] The technical solutions provided by the embodiments of this application have been described in detail above. Specific examples have been used in the embodiments of this application to illustrate the principles and implementation methods of the embodiments of this application. The description of the above embodiments is only for the purpose of helping to understand the methods and core ideas of the embodiments of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the embodiments of this application. Therefore, the content of this specification should not be construed as a limitation on the embodiments of this application.

Claims

1. A multimodal emotion recognition method based on a hypergraph neural network under missing modalities, characterized in that, The method includes: Acquiring multimodal target data for emotion recognition specifically includes: acquiring raw multimodal data based on a pre-configured terminal device or platform interface in the emotion recognition scenario; performing modal state labeling processing on the raw data to obtain modal state labeling data; and determining the target data based on the modal state labeling data and the raw data. Modal missing detection is performed on the target data to obtain modal missing detection results; if the modal missing detection results indicate that the preset modal is completely missing, a learnable missing modal token is directly output to determine the data after missing modal processing. If the modality missing detection result is not a complete missing preset modality, the target data is encoded to obtain encoded data; The encoded data is subjected to missing modality processing to obtain missing modality processed data; specifically, this includes: performing modality missing detection on the encoded data to obtain a modality missing detection result; if the modality missing detection result indicates a preset modality missing, based on the encoded data, constructing a vector with a preset feature vector shape filled with iterable parameters, as a learnable missing modality token; the preset feature vector shape is the same as the feature vector shape of the encoded data of the missing modality; using the learnable missing modality tokens corresponding to each modality as substitute input data for the encoded data to determine the missing modality processed data; A hypergraph neural network is constructed based on the data after processing the missing modalities. The hypergraph neural network is then updated bidirectionally to infer multimodal joint feature representation data. Specifically, this includes: constructing hypergraph nodes and hyperedges based on the data after processing the missing modalities; determining the hypergraph neural network based on the hypergraph nodes and hyperedges; performing information propagation and aggregation based on the hypergraph neural network, and updating the hypergraph nodes and hyperedges bidirectionally; performing high-order inference in the bidirectionally updated hypergraph neural network using available modal information to implicitly complete the missing modal features, obtaining implicitly completed missing modal features; and obtaining the multimodal joint feature representation data based on the implicitly completed missing modal features. Emotion recognition is performed based on the multimodal joint feature representation data to obtain the emotion judgment result data associated with the target data.

2. The method according to claim 1, characterized in that, The process of encoding the target data to obtain encoded data includes: Each modality in the target data is feature-encoded to generate a single-modal representation data with a preset unified dimension; The unimodal representation data of each modality are summarized to obtain the encoded data.

3. The method according to claim 2, characterized in that, The target data includes text data. The step of performing feature encoding on the data of each modality in the target data to generate single-modal representation data with a preset unified dimension includes: Word embedding is used to convert each unit word in the text data into a high-dimensional text vector; The high-dimensional text vector is input into a preset sequence modeling network to extract contextual features; The pooling layer of the neural network associated with the sequence modeling network compresses the context features into a text feature representation of the preset uniform dimension, so as to determine the single-modal representation data corresponding to the text data.

4. The method according to claim 2, characterized in that, The target data includes speech data. The step of performing feature encoding on the data of each modality in the target data to generate single-modal representation data with a preset unified dimension includes: Acoustic features were extracted from the speech data based on audio feature extraction technology; The acoustic features are input into a multilayer perceptron and the speech feature representation with a preset uniform dimension is output to determine the single-modal representation data corresponding to the speech data.

5. The method according to claim 2, characterized in that, The target data includes video data. The step of performing feature encoding on the data of each modality in the target data to generate single-modal representation data with a preset unified dimension includes: Feature extraction is performed on the facial video frames in the video data to obtain facial key point data; High-dimensional visual features are obtained by stitching together the coordinates of the facial key point data; The high-dimensional visual features are input into a multilayer perceptron and the video feature representation with the preset uniform dimension is output to determine the single-modal representation data corresponding to the video data.

6. The method according to claim 1, characterized in that, The step of constructing hypergraph nodes and hyperedges based on the data after processing the missing modalities, and determining the hypergraph neural network based on the hypergraph nodes and hyperedges, includes: The features of different modes and different time steps in the data after processing the missing modes are represented as the hypergraph nodes; Based on the hypergraph nodes, cross-modal hyperedges and temporal hyperedges are constructed to characterize the higher-order associations between multiple modalities, thus obtaining the initial hypergraph structure; Based on the variational hypergraph autoencoder, the initial hypergraph structure is adaptively optimized to obtain the hypergraph neural network.

7. The method according to claim 6, characterized in that, The variational hypergraph autoencoder adaptively optimizes the initial hypergraph structure to obtain the hypergraph neural network, including: The initial hypergraph structure is convolved to generate the embedding representations of the hypergraph nodes and the hyperedges, and the mean and variance of the latent distribution are encoded to obtain the hypergraph encoding result; Based on the hypergraph encoding result, sampling is performed using a reparameterization technique to obtain the potential embeddings of the new hypergraph nodes and hyperedges, thus obtaining the hypergraph sampling result. Based on the hypergraph sampling results, the dot product is calculated and the hypergraph correlation matrix is ​​reconstructed. The hypergraph is then updated according to the hypergraph correlation matrix to obtain the hypergraph neural network.

8. The method according to claim 1, characterized in that, The information propagation and aggregation based on the hypergraph neural network, and the bidirectional updating of the hypergraph nodes and the hyperedges, include: The features of the hypergraph nodes, the features of the hyperedges, and the weights of the hyperedges in the hypergraph neural network are used as the input to the hypergraph neural network. For the features of all the hypergraph nodes connected by any of the hyperedges, the features of the hyperedges are updated by aggregation and mapping of the multilayer perceptron. For the features of all the hyperedges associated with any given hypergraph node, the features of the hypergraph node are updated by aggregation and mapping of the multilayer perceptron.

9. The method according to claim 1, characterized in that, The step of performing emotion recognition based on the multimodal joint feature representation data to obtain emotion judgment result data associated with the target data includes: The multimodal joint feature representation data is input into a classification network with a preset network complexity, and the original sentiment scores of various preset sentiment categories are output based on the activation function of the classification network. The original sentiment scores are normalized to obtain probability distribution data for various preset sentiment categories; Based on the probability distribution data, the sentiment judgment result data associated with the target data is determined.

10. The method according to claim 1, characterized in that, Before the step of encoding the target data to obtain encoded data, the method further includes: The modal data in the target data are subjected to uniform time alignment processing to obtain time-aligned data; The time-aligned data is then subjected to length normalization to obtain the target data after length normalization.

11. The method according to claim 1, characterized in that, The process of constructing a hypergraph neural network based on the data after processing the missing modalities includes: The changes in the modal combination state of the target data are monitored to obtain the monitoring results of the modal combination state changes; When the monitoring result of the modal combination state change indicates that the modal combination state has changed, the hypergraph neural network is constructed by adaptive adjustment based on the preset optimization hypergraph structure strategy.

12. The method according to claim 1, characterized in that, The process involves constructing a hypergraph neural network based on the data after processing the missing modalities, performing bidirectional updates based on the hypergraph neural network, and inferring multimodal joint feature representation data, including: Based on the data after processing the missing modalities, a hypergraph node and a hyperedge are constructed, and the hypergraph neural network is determined based on the hypergraph node and the hyperedge; The hypergraph nodes and hyperedges are updated bidirectionally based on the hypergraph neural network. The feature data of each modality in the bidirectional updated hypergraph neural network are subjected to dimension alignment and scale normalization to obtain dimension-aligned and scale-normalized feature data. The multimodal joint feature representation data is obtained by inference based on the hypergraph neural network constructed from the dimension-aligned and scale-normalized feature data.

13. A model training method for training the artificial intelligence model involved in the multimodal emotion recognition method based on hypergraph neural networks under missing modalities as described in any one of claims 1 to 12, characterized in that, The model training method includes: Construct a teacher model under modal completeness conditions of the target data, and construct a student model under modal missing conditions of the target data; Based on the feature-level knowledge transfer mechanism, the student model is trained according to the emotional representation distribution of the teacher model.

14. The method according to claim 13, characterized in that, The feature-level knowledge transfer mechanism trains the student model based on the sentiment representation distribution of the teacher model, including: The sentiment representation features of the same batch of samples were extracted from both the teacher model and the student model. Based on the sentiment representation features of the same batch of samples, calculate the distribution difference data between the feature distribution of the student model and the feature distribution of the teacher model; Determine the knowledge transfer loss function based on the distribution difference data; Based on the knowledge transfer loss function, the overall loss function of the student model is constructed. Based on the overall loss function, the various model parameters of the student model are updated through backpropagation.

15. The method according to claim 14, characterized in that, The step of constructing the overall loss function of the student model based on the knowledge transfer loss function includes: Construct the classification task loss function corresponding to the multimodal sentiment classification process; The variational hypergraph autoencoder loss function and cross-view contrast loss function corresponding to the construction and update process of the hypergraph neural network are obtained. The overall loss function of the student model is determined by weighted summation of the classification task loss function, the variational hypergraph autoencoder loss function, the cross-view comparison loss function, and the knowledge transfer loss function.

16. A multimodal emotion recognition device based on a hypergraph neural network under missing modalities, characterized in that, The multimodal emotion recognition device based on hypergraph neural networks for missing modalities includes: The data acquisition and encoding module is used to acquire multimodal target data to be used for emotion recognition, and to encode the target data to obtain encoded data; specifically, it is used to: acquire raw multimodal data based on a pre-configured preset terminal device or preset platform interface in the emotion recognition scenario; perform modal state marking processing on the raw data to obtain modal state marking data; and determine the target data based on the modal state marking data and the raw data. The device is also used to: perform modal missing detection on the target data to obtain a modal missing detection result; and, if the modal missing detection result is that a preset modality is completely missing, directly output a learnable missing modality token to determine the data after missing modality processing. A missing modality processing module is used to process the encoded data for missing modality when the missing modality detection result is not a complete missing preset modality, to obtain missing modality processed data. Specifically, it is used to: perform missing modality detection on the encoded data to obtain a missing modality detection result; when the missing modality detection result is a missing preset modality, construct a vector filled with iterable parameters with a preset feature vector shape based on the encoded data, as a learnable missing modality token; the preset feature vector shape is the same as the feature vector shape of the encoded data of the missing modality; and use the learnable missing modality tokens corresponding to each modality as substitute input data for the encoded data to determine the missing modality processed data. The hypergraph neural network construction and update module is used to construct a hypergraph neural network based on the data after processing the missing modalities, perform bidirectional updates based on the hypergraph neural network, and infer multimodal joint feature representation data. Specifically, it is used to: construct hypergraph nodes and hyperedges based on the data after processing the missing modalities; determine the hypergraph neural network based on the hypergraph nodes and hyperedges; perform information propagation and aggregation based on the hypergraph neural network, and perform bidirectional updates on the hypergraph nodes and hyperedges; perform high-order inference in the bidirectionally updated hypergraph neural network using available modal information, perform implicit completion processing on the missing modal features, and obtain implicitly completed missing modal features; and obtain the multimodal joint feature representation data based on the implicitly completed missing modal features. The emotion recognition module is used to perform emotion recognition based on the multimodal joint feature representation data to obtain the emotion judgment result data associated with the target data.

17. A model training apparatus for training the artificial intelligence model involved in the multimodal emotion recognition device based on a hypergraph neural network under missing modalities as described in claim 16, characterized in that, The model training device includes: The model building module is used to build a teacher model under the condition of modal completeness of the target data, and to build a student model under the condition of modal missingness of the target data; The knowledge transfer module is used to train the student model based on the emotion representation distribution of the teacher model, using a feature-level knowledge transfer mechanism.

18. A computer device, characterized in that, The computer device includes: At least one processor and memory; The memory is used to store program code, and the processor is used to call the program code stored in the memory to execute the method as described in any one of claims 1 to 15.

19. A computer storage medium, characterized in that, It includes instructions that, when executed on a computer, cause the computer to perform the method as described in any one of claims 1 to 15.

20. A computer program product, characterized in that, It includes instructions that, when executed on a computer, cause the computer to perform the method as described in any one of claims 1 to 15.

Images