Method and system for sentiment recognition of multi-modal missing data based on dynamic completion

Abstract

The application provides a multi-modal missing data emotion recognition method and system based on dynamic completion, relates to the field of emotion recognition, and comprises the following steps: calculating a modality reliability weight vector, determining a modality combination and performing dynamic alignment, constructing an evolution trajectory graph to predict missing node features, dividing a time sequence completion interval based on information entropy distribution to perform adaptive feature completion, and finally realizing emotion recognition. The application can effectively handle the multi-modal data missing problem, improve the cross-modal feature capturing capability and emotion recognition accuracy, and enhance the system robustness.

Description

TECHNICAL FIELD

[0001] The present application relates to emotion recognition technology, and in particular to a multi-modal missing data emotion recognition method and system based on dynamic completion. BACKGROUND

[0002] Emotion recognition is an important research direction in the field of artificial intelligence, and has wide application value in human-computer interaction, intelligent service, mental health monitoring and other fields. Multi-modal emotion recognition can more comprehensively understand and recognize human emotional state by fusing voice, text, visual and other multi-modal information. With the popularization of intelligent devices and the development of multi-modal data acquisition technology, the problems of multi-modal emotion recognition in practical application scenarios are increasingly prominent, especially the problem of missing modalities often occurring in real environments.

[0003] Existing multi-modal emotion recognition methods mainly adopt fixed modal combination or simple completion strategies, which are difficult to effectively handle the problem of missing modalities in complex scenarios. These methods usually assume that all modal data is complete and available, or adopt a fixed modal combination strategy, lacking dynamic detection and adaptive processing capabilities for missing modalities. In addition, existing technologies often ignore the temporal dependency between modalities, and cannot accurately capture the dynamic evolution process of emotional expression, resulting in a significant decrease in recognition accuracy under incomplete modalities. Lack of evaluation mechanism for modal quality and reliability, unable to dynamically adjust the contribution weight of each modality in the fusion process according to the completeness and quality of the modalities, leading to a sharp decline in model performance when the modalities are missing or of poor quality. Ignoring the temporal correlation and evolution law between multi-modal data, unable to effectively utilize the complementary information between modalities for missing data completion, difficult to capture the continuous change characteristics of emotional expression. Lack of adaptive completion strategy for different time segments, using a unified completion method to process all missing data, failing to consider the information entropy difference of emotional expression in different time segments, resulting in poor completion effect and affecting the final emotion recognition accuracy. SUMMARY

[0004] The embodiments of the present application provide a multi-modal missing data emotion recognition method and system based on dynamic completion, which can solve the problems in the prior art.

[0005] In a first aspect, the embodiments of the present application provide a multi-modal missing data emotion recognition method based on dynamic completion, comprising:

[0006] Obtaining multi-modal input data, detecting the data missing position of each modality, calculating the modality information completeness value and quality score, and generating a modality reliability weight vector;

[0007] correlation coefficients between modalities in multi-modal input data, determining modal combinations based on the correlation coefficients and a reliability weight vector, detecting a time sequence offset of the modal combinations, performing dynamic alignment on the multi-modal input data according to the time sequence offset, and extracting time sequence features from the data after the dynamic alignment;

[0008] constructing the time sequence features as nodes of an evolution trajectory graph, calculating time sequence dependency strengths between the nodes, dividing the nodes into feature groups according to the time sequence dependency strengths, and generating predicted features of missing nodes in combination with time sequence evolution rules of the modal combinations;

[0009] calculating information entropy distributions of the time sequence features, dividing time sequence completion intervals according to the information entropy distributions, adaptively completing the predicted features in the time sequence completion intervals to obtain multi-modal fusion features, and identifying a sample sentiment type based on the multi-modal fusion features.

[0010] obtaining multi-modal input data, detecting data missing positions of each modality, calculating modality information integrity values and quality scores, and generating a modality reliability weight vector, including:

[0011] obtaining multi-modal input data including video, audio and text, calculating continuity difference values of adjacent data segments in the multi-modal input data, marking difference positions higher than a difference threshold as data missing positions, and generating a data missing mark sequence;

[0012] calculating time sequence change features of non-missing data indicated by the data missing mark sequence, performing feature fusion on the time sequence change features to obtain time sequence correlation degrees, and calculating information integrity values of each modality based on the time sequence correlation degrees;

[0013] extracting content features and structure features of the non-missing data and combining the features according to a preset weight to obtain quality features, and calculating quality scores of each modality according to the quality features;

[0014] calculating feature importance coefficients from the information integrity values and the quality scores, performing normalization processing on the feature importance coefficients to obtain a modality reliability weight vector, and using the modality reliability weight vector to represent the importance of each modality feature in feature fusion.

[0015] calculating correlation coefficients between modalities in multi-modal input data, determining modal combinations based on the correlation coefficients and a reliability weight vector, detecting a time sequence offset of the modal combinations, performing dynamic alignment on the multi-modal input data according to the time sequence offset, and extracting time sequence features from the data after the dynamic alignment, including:

[0016] The time sequence entropy values of each modality feature in the multi-modal input data are calculated, an adaptive sampling probability distribution is constructed based on the time sequence entropy values, the multi-modal input data is resampled according to the adaptive sampling probability distribution to obtain a sampling sequence, and a cross-correlation coefficient between the modalities in the sampling sequence is calculated;

[0017] The cross-correlation coefficient and the time sequence entropy value are combined to construct a dynamic constraint matrix, the dynamic constraint matrix is multiplied by a reliability weight vector to obtain a modality confidence, a combination threshold is set according to the modality confidence, and a modality combination is constructed for a modality pair higher than the combination threshold.

[0018] The time sequence offset feature is obtained based on eigenvalue decomposition of the dynamic constraint matrix, the time sequence offset feature is reconstructed to obtain a time sequence offset, and the sampling sequence is dynamically aligned according to the time sequence offset to obtain an aligned sequence.

[0019] The change feature of the aligned sequence is extracted, and is weighted and fused with a suppression vector to obtain a time sequence feature.

[0020] The time sequence offset feature is obtained based on eigenvalue decomposition of the dynamic constraint matrix, the time sequence offset feature is reconstructed to obtain a time sequence offset, and the sampling sequence is dynamically aligned according to the time sequence offset to obtain an aligned sequence, including:

[0021] Eigenvalue decomposition is performed on the dynamic constraint matrix, the decay curvature of the eigenvalue is calculated to determine the number of principal eigenvalues, the eigenvectors corresponding to the principal eigenvalues are extracted, and are mapped to a nonlinear time sequence space to obtain a time sequence offset feature;

[0022] A multi-scale time sequence sliding window is constructed based on the time sequence offset feature, a local time sequence change value is extracted in each sliding window, a global time sequence trend value is obtained by performing probability density estimation on the local time sequence change value, and the local time sequence change value and the global time sequence trend value are combined and reconstructed to obtain a time sequence offset.

[0023] The relative displacement relationship between the modalities is calculated according to the time sequence offset, the sampling sequence is time sequence corrected, and an initial aligned sequence is generated;

[0024] The time sequence consistency value and the feature correlation value of the initial aligned sequence are calculated, the time sequence offset is adjusted according to the consistency value and the correlation value, and the initial aligned sequence is optimized to obtain an aligned sequence.

[0025] The time sequence feature is constructed as a node of an evolution trajectory graph, the time sequence dependence strength between the nodes is calculated, the nodes are divided into feature groups according to the time sequence dependence strength, and the prediction feature of a missing node is generated in combination with the time sequence evolution rule of the modality combination, including:

[0026] The time sequence feature is constructed as a node of an evolution trajectory graph, the length of a time window is dynamically adjusted according to the change trend of the time sequence feature, historical information is extracted in the time window to construct a node feature.

[0027] The time sequence dependence strength between nodes is calculated by using multi-dimensional information entropy and state transition probability, the time sequence dependence strength is corrected in combination with the time sequence consistency of node features, and finally the time sequence dependence strength is obtained;

[0028] The nodes are clustered according to the time sequence dependence strength, the clustering boundary is determined based on the similarity of node features, the nodes are divided into feature groups, and the time sequence evolution law of the nodes in the feature groups is obtained;

[0029] The time sequence evolution law of the mode combination is obtained by analyzing the node features of different feature groups, the cooperative evolution matrix of the feature groups is constructed, the predicted features of the missing nodes are generated by combining the cooperative evolution matrix and the time sequence evolution law;

[0030] The predicted features are compared with the actual features, the prediction deviation is calculated, the cooperative evolution matrix is optimized according to the prediction deviation, and the predicted features are updated.

[0031] The information entropy distribution of the time sequence features is calculated, the time sequence completion interval is divided according to the information entropy distribution, the multi-modal fusion features are obtained by adaptively completing the predicted features in the time sequence completion interval, and the sample emotion type is identified based on the multi-modal fusion features, including:

[0032] The conditional information entropy and the joint information entropy of the time sequence features in the time dimension are calculated, the information entropy distribution is generated by combination, the feature mutation points are identified according to the gradient change of the information entropy distribution, and the region between adjacent feature mutation points is determined as the time sequence completion interval;

[0033] The frequency distribution and the cumulative distribution of the time sequence features in the time sequence completion interval are extracted, the probability density function is constructed, and the time sequence completion constraint condition is generated according to the change law of the probability density function and the historical data;

[0034] The feature importance is calculated based on the time sequence completion constraint condition, the predicted features are progressively weighted according to the feature importance, and the multi-modal fusion features are generated by adaptively completing the predicted features using the progressive weighting results;

[0035] The correlation coefficient and the mutual information value of the multi-modal fusion features in the time dimension and the feature dimension are calculated, the time sequence correlation degree is obtained by combination, the feature weighting is iteratively optimized according to the time sequence correlation degree, and the optimized multi-modal fusion features are generated, and the optimized multi-modal fusion features are mapped to the emotion feature space for sample emotion recognition.

[0036] In a second aspect of the embodiment of the application, a multi-modal missing data emotion recognition system based on dynamic completion is provided, including:

[0037] The first unit is configured to acquire multi-modal input data, detect data missing positions of each mode, calculate a mode information integrity value and a quality score, and generate a mode reliability weight vector;

[0038] The second unit is configured to calculate a cross-correlation coefficient between modes in the multi-modal input data, determine a mode combination based on the cross-correlation coefficient and the reliability weight vector, detect a time sequence offset of the mode combination, dynamically align the multi-modal input data according to the time sequence offset, and extract a time sequence feature from the dynamically aligned data;

[0039] The third unit is configured to construct the time sequence feature as a node of an evolution trajectory graph, calculate a time sequence dependence strength between nodes, divide the nodes into feature groups according to the time sequence dependence strength, and generate a predicted feature of a missing node in combination with a time sequence evolution rule of the mode combination.

[0040] The fourth unit is configured to calculate an information entropy distribution of the time sequence feature, divide a time sequence completion interval according to the information entropy distribution, adaptively complete the predicted feature in the time sequence completion interval to obtain a multi-modal fusion feature, and identify a sample emotion type based on the multi-modal fusion feature.

[0041] In a third aspect, an electronic device is provided, including:

[0042] a processor;

[0043] a memory for storing processor-executable instructions;

[0044] The processor is configured to invoke the instructions stored in the memory to execute the method described above.

[0045] In a fourth aspect, a computer-readable storage medium is provided, which stores computer program instructions, and the computer program instructions are executed by a processor to implement the method described above.

[0046] In the embodiment, the reliability weight vector is generated by calculating the mode information integrity value and the quality score, the optimal mode combination is determined in combination with the cross-correlation coefficient between modes, efficient screening of multi-modal data is achieved, and the accuracy of emotion recognition in the missing data scenario is effectively improved. The dynamic alignment is performed by using the time sequence offset, the time sequence feature is constructed as an evolution trajectory graph, the feature groups are divided according to the time sequence dependence strength, the time sequence evolution rule between different modes can be fully captured, and the mismatching problem of multi-modal data in the time sequence dimension is solved. The time sequence completion interval is divided based on the information entropy distribution, the predicted feature is adaptively completed, the missing mode data is accurately recovered and fused, the robustness of the model in the complex and changeable environment is improved, and the emotion recognition result is more reliable. BRIEF DESCRIPTION OF DRAWINGS

[0047] Figure 1 A flowchart of a method for multi-modal missing data sentiment recognition based on dynamic completion according to an embodiment of the present application is shown in FIG. 1.

[0048] Figure 2 A flowchart of a multi-modal fusion and sentiment recognition technology based on time sequence feature analysis according to an embodiment of the present application is shown in FIG. 2. DETAILED DESCRIPTION

[0049] To make the objectives, technical solutions, and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described below in connection with the drawings of the embodiments of the present application. Obviously, the described embodiments are only a part but not all of the embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0050] The technical solutions of the present application will be described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes can not be described in some embodiments.

[0051] Figure 1 A flowchart of a method for multi-modal missing data sentiment recognition based on dynamic completion according to an embodiment of the present application is shown in FIG. 1. Figure 1 The method includes the following steps.

[0052] Obtaining multi-modal input data, detecting data missing positions of each modality, calculating modality information integrity values and quality scores, and generating a modality reliability weight vector.

[0053] Calculating inter-correlation coefficients between modalities in the multi-modal input data, determining a modality combination based on the inter-correlation coefficients and the reliability weight vector, detecting a time sequence offset of the modality combination, dynamically aligning the multi-modal input data according to the time sequence offset, and extracting time sequence features from the dynamically aligned data.

[0054] Constructing the time sequence features as nodes of an evolution trajectory graph, calculating time sequence dependency strengths between the nodes, dividing the nodes into feature groups according to the time sequence dependency strengths, generating predicted features of missing nodes in combination with a time sequence evolution law of the modality combination, and obtaining multi-modal fusion features by adaptively completing the predicted features in a time sequence completion interval.

[0055] Calculating information entropy distributions of the time sequence features, dividing the time sequence completion interval according to the information entropy distributions, obtaining multi-modal fusion features by adaptively completing the predicted features in the time sequence completion interval, and recognizing a sentiment type of a sample based on the multi-modal fusion features.

[0056] In an optional embodiment, multi-modal input data is acquired, data missing positions in each modality are detected, a modality information integrity value and a quality score are calculated, and a modality reliability weight vector is generated, which includes:

[0057] Multi-modal input data including video, audio and text is acquired, a continuity difference value between adjacent data segments in the multi-modal input data is calculated, a difference position higher than a difference threshold is marked as a data missing position, and a data missing marking sequence is generated;

[0058] A timing change feature is calculated for non-missing data indicated by the data missing marking sequence, a timing correlation degree is obtained by feature fusion of the timing change feature, and an information integrity value of each modality is calculated based on the timing correlation degree;

[0059] Content features and structure features of the non-missing data are extracted, and a quality feature is obtained by combining according to a preset weight, and a quality score of each modality is calculated according to the quality feature;

[0060] A feature importance coefficient is calculated based on the information integrity value and the quality score, a modality reliability weight vector is obtained by normalizing the feature importance coefficient, and the modality reliability weight vector is used to represent the importance of each modality feature in feature fusion.

[0061] In this embodiment, multi-modal input data including video, audio and text is acquired. Taking a sentiment interaction scene as an example, video data contains visual information such as user facial expressions and body movements, audio data contains acoustic features such as speech content and tone changes, and text data is speech transcription or user input text content. For the collected data, it is necessary to first detect the possible data missing positions in each modality.

[0062] For multi-modal input data, a continuity difference value between adjacent data segments is calculated. For video data, the continuity can be measured by calculating the pixel difference or the Euclidean distance of feature vectors between adjacent frames. For example, for facial expression video, the facial key point coordinates of each frame are extracted, and the key point position change amount between adjacent frames is calculated. When the change amount between two frames exceeds a preset threshold (such as the average displacement of key points being greater than 10 pixels), it can be determined that the continuity is abnormal. For audio data, the acoustic feature difference (such as the Euclidean distance of mel frequency cepstral coefficients) between adjacent audio segments can be used to detect abnormalities. For text data, the semantic similarity of adjacent text segments can be calculated, and when the semantic coherence suddenly decreases, it may indicate that there is data missing.

[0063] The difference position higher than the difference threshold is marked as a data missing position. For example, for video data, if the feature difference value between two frames is 0.85 and the preset threshold is 0.7, the position is marked as a missing position. Similar processing is performed on audio and text data, and finally a data missing marking sequence is generated. The sequence can be represented as a binary vector, where "1" indicates that the corresponding position data is normal, and "0" indicates that the corresponding position data is missing.

[0064] For the non-missing data indicated by the data missing marking sequence, the timing change feature is calculated. For video data, the change rate, change amplitude and change direction of facial expression and other timing features can be extracted; for audio data, the pitch change trend, speech speed change and volume change and other timing features can be extracted; for text data, the change of emotional word distribution, sentence structure and other timing features can be extracted.

[0065] The above timing change features are fused to obtain the timing correlation degree. Feature fusion can use a weighted average method to combine each timing feature according to a preset weight. For example, for video data, the weights of facial expression change rate, change amplitude and change direction can be set to 0.4, 0.3 and 0.3 respectively for weighted fusion.

[0066] Based on the timing correlation degree, the information integrity value of each modality is calculated. The information integrity value reflects the continuity and integrity degree of the modality data. The calculation method can be: multiplying the timing correlation degree by the proportion of non-missing data, and then comparing it with a reference integrity value (such as 0.8) to obtain a relative integrity score. For example, if the timing correlation degree of video data is 0.75 and the proportion of non-missing data is 0.9, the information integrity value can be calculated as 0.75x0.9÷0.8=0.84. Next, the content features and structure features of the non-missing data are extracted. For video data, the content features include facial expression types, intensity, etc., and the structure features include the geometric configuration relationship of facial key points; for audio data, the content features include the emotional tendency of speech content, and the structure features include the prosody pattern of speech; for text data, the content features include the frequency of emotional words, and the structure features include the syntactic structure complexity.

[0067] The content features and structure features are combined according to a preset weight to obtain quality features. For example, for video data, the weights of content features and structure features can be set to 0.6 and 0.4; for audio data, they can be set to 0.5 and 0.5; for text data, they can be set to 0.7 and 0.3. According to the quality features, the quality score of each modality is calculated, and a mapping function can be used to map the feature value to the score interval of 0-1.

[0068] The information integrity value and the quality score are used to calculate a feature importance coefficient. The calculation method can be to weight and average the information integrity value and the quality score, for example, the weights of 0.4 and 0.6 can be assigned respectively. The feature importance coefficient is normalized to obtain a modal reliability weight vector. The normalization processing ensures that the sum of the weights of all modalities is 1, for example, the weights of video, audio and text are 0.4, 0.35 and 0.25 respectively. The weight vector is used to represent the importance of each modality feature in subsequent feature fusion. Assuming that in the multi-modal data collected in a certain emotion recognition scene, the video data has a 2-second freeze frame at 30 seconds, the audio data has a 1-second silence at 45 seconds, and the text data is basically complete. The information integrity values of video, audio and text calculated by the above method are 0.92, 0.95 and 0.98 respectively; the quality scores are 0.85, 0.80 and 0.90 respectively. The feature importance coefficients are calculated by combining these values, and the modal reliability weight vector obtained after normalization is [0.32, 0.30, 0.38], which indicates that in this emotion recognition task, the reliability of the text modality is the highest and should be given the largest weight, followed by the video modality, and finally the audio modality.

[0069] By dynamically detecting the missing position of multi-modal data and evaluating the reliability of each modality, the processing capability of the emotion recognition system for incomplete data is improved; with the help of time sequence correlation calculation and quality scoring mechanism, the effectiveness of each modality data is accurately evaluated; by using the adaptive weight fusion strategy, the system can adjust the importance of each modality according to the real-time data quality, and the robustness of the system in complex environment is enhanced; the performance decline problem of traditional fixed weight method in data missing scene is effectively solved, and the accuracy and stability of multi-modal emotion recognition are improved.

[0070] In an optional implementation, a cross-correlation coefficient between modalities in multi-modal input data is calculated, a modality combination is determined based on the cross-correlation coefficient and the reliability weight vector, a time sequence offset of the modality combination is detected, and a time sequence feature is extracted from the dynamically aligned data.

[0071] A time sequence entropy value of each modality feature in the multi-modal input data is calculated, an adaptive sampling probability distribution is constructed based on the time sequence entropy value, the multi-modal input data is resampled to obtain a sampling sequence according to the adaptive sampling probability distribution, and a cross-correlation coefficient between modalities in the sampling sequence is calculated.

[0072] A dynamic constraint matrix is constructed by combining the cross-correlation coefficient and the time sequence entropy value, a modality confidence is obtained by multiplying the dynamic constraint matrix and the reliability weight vector, a combination threshold is set according to the modality confidence, and a modality combination is constructed by pairing modalities higher than the combination threshold.

[0073] The time sequence offset feature is obtained by eigenvalue decomposition based on a dynamic constraint matrix, the time sequence offset is reconstructed by reconstructing the time sequence offset feature, and the alignment sequence is obtained by dynamically aligning the sampling sequence according to the time sequence offset.

[0074] The change feature of the alignment sequence is extracted, and the time sequence feature is obtained by weighted fusion with the suppression vector.

[0075] In a specific implementation, the multi-modal input data includes video data, audio data and text data. First, each modal data is preprocessed, the video data is obtained by frame extraction technology to obtain a key frame sequence, the audio data is converted into a spectrum feature sequence through spectrum analysis, and the text data is converted into a text feature sequence through word segmentation and vectorization processing. The preprocessed modal data is represented as a feature vector sequence and stored in the system memory for subsequent processing.

[0076] When calculating the time sequence entropy value of each modal feature in the multi-modal input data, a sliding window technique is used to segment the feature sequence. For the feature sequence of each modality, the window size is set to 128 time points, and the window overlap rate is 50%. In each window, the probability density function of the feature distribution is calculated, which is estimated by counting the frequency of the feature value falling into the predefined interval. Taking the video feature as an example, the feature value range is divided into 32 equal-width intervals, the number of feature values in each interval is calculated and normalized to obtain the probability estimate. The time sequence entropy value is calculated by the Shannon entropy formula, and a high entropy value indicates that the modality has a larger change in time sequence and has a higher information amount.

[0077] Based on the calculated time sequence entropy value, an adaptive sampling probability distribution is constructed. Specifically, the time sequence entropy values of each modality are normalized to obtain weight coefficients. For example, if the time sequence entropy values of the video, audio and text modalities are 0.85, 0.65 and 0.45, the normalized weight coefficients are 0.44, 0.33 and 0.23. These weight coefficients are used to construct a mixed Gaussian distribution as a probability distribution function for time sequence sampling. According to this probability distribution function, the original sequence is resampled, and more sampling points are obtained in the high-entropy value region. In practical applications, a multi-modal data with a length of 10 minutes and a sampling rate of 30 Hz may be resampled into a sampling sequence of 2000 key time sequence points.

[0078] For the resampled sampling sequence, the cross-correlation coefficient between modalities is calculated. The calculation process uses a sliding window technique with a window size of 64 time points. For the video and audio modality pair, the Pearson correlation coefficient is calculated in each window. For example, the cross-correlation coefficient between the video motion feature and the audio volume feature in a window is 0.78, indicating that the two modalities are highly correlated in that window period. The cross-correlation coefficients of all modality pairs in all time windows are calculated to form a cross-correlation coefficient matrix.

[0079] The cross-correlation coefficient is combined with the time series entropy value to construct a dynamic constraint matrix. The construction method of this matrix is to weight the average of each element of the cross-correlation coefficient matrix and the time series entropy value of the corresponding modal pair. The weight parameter is determined as 0.7 and 0.3 through validation set optimization. For example, the cross-correlation coefficient of the video-audio modal pair is 0.78, and the weighted average of the corresponding time series entropy value is 0.76, so the element value in the dynamic constraint matrix is 0.774.

[0080] The dynamic constraint matrix is multiplied by the reliability weight vector to obtain the modal confidence. The reliability weight vector is pre-calculated based on the signal-to-noise ratio and integrity index of each modal. For example, in a specific application scenario, the reliability weights of video, audio, and text modal are 0.8, 0.75, and 0.6, respectively. The confidence of each modal pair is calculated by matrix multiplication, such as the confidence of the video-audio modal pair is 0.619.

[0081] According to the modal confidence, set the combination threshold, the threshold is set as the mean of all modal confidences plus 0.5 times the standard deviation, which is 0.58 in this example. The modal pairs higher than the threshold are constructed into modal combinations. In the example, the video-audio modal combination is retained because the confidence 0.619 is higher than the threshold 0.58, while the text-audio modal combination is filtered out because the confidence 0.51 is lower than the threshold.

[0082] Based on the eigenvalue decomposition of the dynamic constraint matrix, the time series offset feature is obtained. The eigenvalue decomposition is performed on the sub-matrix of the dynamic constraint matrix of the retained modal combination, and the first three feature vectors are extracted as the time series offset feature. These feature vectors are converted into actual time offsets through nonlinear mapping, with the unit of milliseconds. For example, the time series offset of the video-audio modal combination is determined as 120 milliseconds, indicating that the video data leads the audio data by 120 milliseconds.

[0083] According to the calculated time series offset, the sampling sequence is dynamically aligned. For the video-audio modal combination, the audio data time axis is shifted by 120 milliseconds to align with the video data time point. The alignment operation is realized through interpolation calculation to maintain the continuity of the data. The aligned data is stored as an aligned sequence, which retains the index information of the original sampling points, facilitating the backtracking to the original data.

[0084] The change features of the aligned sequences are extracted, mainly focusing on the gradient information and local change patterns of each modality data. The inter-frame difference is calculated for video features, the spectral change rate is calculated for audio features, and the semantic conversion strength is calculated for text features. These change features are weighted and fused with the pre-trained suppression vectors, which are used to reduce the impact of noise and redundant information. For example, the weight corresponding to the video background noise in the suppression vector is 0.1, and the weight corresponding to the main body motion is 0.9. The weighted and fused time sequence features have higher discriminability and stability, and can effectively represent the time sequence dynamic characteristics of multi-modal data.

[0085] Through the above detailed steps, dynamic alignment of multi-modal data and effective time sequence feature extraction are realized, so that the system can better understand and analyze complex data containing multiple information modalities.

[0086] In an optional implementation, the time sequence offset features are obtained based on eigenvalue decomposition of the dynamic constraint matrix, the time sequence offset is reconstructed from the time sequence offset features, and the aligned sequences are obtained by dynamically aligning the sampling sequences according to the time sequence offset.

[0087] Eigenvalue decomposition is performed on the dynamic constraint matrix, the decay curvature of the eigenvalue is calculated to determine the number of principal eigenvalues, the eigenvectors corresponding to the principal eigenvalues are extracted, and the time sequence offset features are obtained by mapping to the nonlinear time sequence space;

[0088] Based on the time sequence offset features, a multi-scale time sequence sliding window is constructed, local time sequence change values are extracted in each sliding window, probability density estimation is performed on the local time sequence change values to obtain global time sequence trend values, and the local time sequence change values and the global time sequence trend values are combined to reconstruct the time sequence offset;

[0089] The relative displacement relationship between the modalities is calculated according to the time sequence offset, the time sequence of the sampling sequences is corrected, and the initial aligned sequences are generated;

[0090] The time sequence consistency value and the feature correlation value of the initial aligned sequences are calculated, the time sequence offset is adjusted according to the consistency value and the correlation value, the initial aligned sequences are optimized, and the aligned sequences are obtained.

[0091] In the embodiment, first, eigenvalue decomposition is performed on the dynamic constraint matrix, the decay curvature of the eigenvalues is calculated to determine the number of principal eigenvalues, the eigenvectors corresponding to the principal eigenvalues are extracted and mapped to a nonlinear time series space to obtain time series offset features; a multi-scale time series sliding window is constructed based on the time series offset features, local time series change values are extracted in each sliding window, and global time series trend values are obtained by performing probability density estimation on the local time series change values; the local time series change values and the global time series trend values are combined to reconstruct time series offset amounts; the relative displacement relationship between modes is calculated according to the time series offset amounts, the sampling sequence is time series corrected, and an initial aligned sequence is generated; the time series consistency value and the feature correlation value of the initial aligned sequence are calculated, the time series offset amounts are adjusted according to the consistency value and the correlation value, and the initial aligned sequence is optimized to obtain an aligned sequence.

[0092] The process of eigenvalue decomposition on the dynamic constraint matrix first obtains a sampling sequence dataset to be processed, which contains time series data of different modalities, such as video data and audio data. The sampling sequence has different degrees of offset on the time axis. When constructing the dynamic constraint matrix, the time series distance between different sampling points is calculated to form an n×n square matrix, where n is the length of the sampling sequence. Specifically, for any two points i and j in the sampling sequence, the time series distance between them is calculated as the element value of the constraint matrix, which reflects the correlation of the two sampling points in the time dimension. For example, for a certain video-audio dataset, 600 frames of video data and corresponding audio data can be selected to construct a 600×600 dynamic constraint matrix.

[0093] Eigenvalue decomposition is performed on the constructed dynamic constraint matrix to obtain a set of eigenvalues and corresponding eigenvectors. These eigenvalues are arranged in descending order, and the ratio between adjacent eigenvalues is calculated to obtain the decay curvature of the eigenvalues. By setting a threshold for the decay curvature, the number of principal eigenvalues is determined. In practical applications, the threshold is usually set to between 0.1 and 0.2, i.e., when the ratio between adjacent eigenvalues is less than the threshold, it is considered that the subsequent eigenvalues have little contribution to the system and can be ignored. For example, for the above-mentioned 600×600 constraint matrix, the first 12 eigenvalues are determined as principal eigenvalues after eigenvalue decomposition.

[0094] The eigenvectors corresponding to these principal eigenvalues are extracted, and each eigenvector represents the projection of time series data in the feature space. These eigenvectors are mapped to a nonlinear time series space to obtain time series offset features. Nonlinear mapping is realized by kernel functions, common kernel functions include Gaussian kernel function, polynomial kernel function, etc. For actual cases, the Gaussian kernel function is selected, the kernel parameter is set to 0.5, and the 12 principal eigenvectors are mapped to the nonlinear space to obtain a 600×12-dimensional time series offset feature matrix.

[0095] When constructing the multi-scale time sliding window based on the time offset feature, windows of different sizes are set, such as 10 frames, 20 frames, and 30 frames in three scales. For each scale of the window, the local time change value in each window is extracted by sliding on the time offset feature. The local time change value is obtained by calculating the difference between adjacent frames in the window. For example, for a window size of 10 frames, the difference between adjacent frames in the 10 frames is calculated to obtain 9 local time change values. By sliding on the entire time offset feature, a plurality of groups of local time change values are obtained.

[0096] The probability density estimation is performed on these local time change values to obtain the global time trend value. The kernel density estimation method is used for probability density estimation, and a suitable bandwidth parameter such as 0.8 is selected to estimate the distribution of the local time change value. According to the estimated probability density function, the global time trend value at each time point is calculated. The local time change value and the global time trend value are combined by weighting, and the weight ratio is 6:4. The time offset is reconstructed. For 600 frames of data, 600 time offsets are finally obtained, representing the time adjustment amount required for each frame.

[0097] The relative displacement relationship between the modes is calculated according to the time offset. For video and audio two modes, the relative displacement on the time axis is calculated. For example, if the 100th frame of the video corresponds to the 105th frame of the audio, the relative displacement between the two frames is 5 frames. According to the calculated relative displacement relationship, the time sequence is corrected to generate an initial alignment sequence. The time sequence correction is realized by time interpolation or deletion of sampling points, such as linear interpolation, cubic spline interpolation, etc. For actual cases, the cubic spline interpolation method is used to adjust the video sequence to the state of alignment with the audio sequence.

[0098] The time consistency value and the feature correlation value of the initial alignment sequence are calculated. The time consistency value represents the smoothness of the sequence on the time axis after alignment, and the standard deviation of the time change between adjacent frames is calculated. The smaller the standard deviation, the higher the consistency. The feature correlation value represents the correlation degree between different modal data, and the cosine similarity between the feature vectors of the two modes is calculated. The higher the similarity, the stronger the correlation. According to the calculated consistency value and correlation value, an optimization objective function is constructed according to a weight ratio of 7:3, and the time offset is adjusted by the gradient descent method.

[0099] In the optimization process, the learning rate is set to 0.01, the iteration number is 100, and the iteration is terminated in advance when the change rate of the objective function is less than 0.001. The adjusted time offset is applied to the initial alignment sequence for further correction to obtain the final alignment sequence.

[0100] The time sequence offset characteristics are extracted by eigenvalue decomposition of a dynamic constraint matrix, and the time sequence asynchronization problem of multi-modal data is solved; the local and global time sequence changes are analyzed by using a multi-scale time sequence sliding window, and the time sequence offset characteristics are accurately captured; the relative displacement correction between modes is realized based on the reconstructed time sequence offset, and the time sequence alignment accuracy of data is improved; the alignment sequence is optimized by a dual evaluation mechanism of time sequence consistency and feature correlation, and the time sequence matching degree between modes is enhanced; the time sequence deviation in the multi-modal data acquisition process is effectively eliminated, a high-quality synchronous data basis is provided for subsequent feature fusion and emotion recognition, and the overall system performance is improved.

[0101] In an optional implementation, the time sequence features are constructed as nodes of an evolution trajectory graph, the time sequence dependency strength between nodes is calculated, the nodes are divided into feature groups according to the time sequence dependency strength, and the prediction features of missing nodes are generated in combination with the time sequence evolution law of modal combinations, including:

[0102] The time sequence features are constructed as nodes of an evolution trajectory graph, the length of a time window is dynamically adjusted according to the change trend of the time sequence features, and the historical information is extracted in the time window to construct node features;

[0103] The time sequence dependency strength between nodes is calculated by using multi-dimensional information entropy and state transition probability, the time sequence dependency strength is corrected in combination with the time sequence consistency of node features, and the final time sequence dependency strength is obtained;

[0104] The nodes are clustered according to the time sequence dependency strength, the clustering boundary is determined based on the similarity of node features, the nodes are divided into feature groups, and the time sequence evolution law of the nodes in the feature groups is obtained;

[0105] The time sequence evolution law of modal combinations is obtained by analyzing the node features of different feature groups, a collaborative evolution matrix of the feature groups is constructed, the prediction features of missing nodes are generated in combination of the collaborative evolution matrix and the time sequence evolution law;

[0106] The prediction features are compared with actual features, a prediction deviation is calculated, the collaborative evolution matrix is optimized according to the prediction deviation, and the prediction features are updated.

[0107] In one embodiment, the time series features are first constructed as nodes of the evolutionary trajectory graph. For a given time series dataset, containing multiple feature dimensions, each feature dimension corresponds to a time series. The length of the time window is dynamically adjusted based on the trend of the feature. Specifically, the volatility of the time series data at different time scales is calculated. When it is detected that the volatility of a certain feature in recent data exceeds 150% of the historical average, the time window of the feature is shortened to 75% of the original; when the volatility is less than 50% of the historical average, the time window is extended to 125% of the original. For example, for a certain temperature sensor data, the initial window length is set to 60 minutes, and when it is detected that the temperature volatility in the last 10 minutes is 3.2°C (exceeding 150% of the historical average volatility of 2.0°C), the window is adjusted to 45 minutes. Within the determined time window, the statistical features of the historical data are extracted as node features, including mean, standard deviation, kurtosis, skewness, maximum value, minimum value, and main frequency components after Fourier transform.

[0108] Next, the time series dependence strength between nodes is calculated, using a method combining multi-dimensional information entropy and state transition probability to quantify the dependence between nodes. For any two nodes A and B, the feature values are first discretized into a finite number of states, such as dividing the temperature value into several state intervals according to an interval of 5°C. The joint information entropy of nodes A and B, as well as their marginal information entropy, is calculated to obtain the preliminary time series dependence strength. At the same time, a state transition matrix is constructed to record the probability distribution of node B appearing in each state at the next time step when node A is in different states. For example, when the temperature sensor is in the state of 25-30°C, the probability of the humidity sensor appearing in the 60%-70% interval at the next time step is 0.72. The information entropy calculation result and the state transition probability are combined to obtain the initial time series dependence strength. Subsequently, the dependence strength is modified in combination with the time series consistency of the node features. The time series consistency is obtained by calculating the correlation of the change trends of the two node features. When the consistency of the change trends of the two nodes is higher than 75%, the dependence strength is enhanced; when the consistency is lower than 30%, the dependence strength is weakened. Specifically, for a certain pressure sensor and a flow sensor, the initial dependence strength is 0.68, and due to the consistency of the change trends reaching 82%, the modified dependence strength is increased to 0.79.

[0109] Based on the calculated time-dependent strength, the nodes are clustered and divided. The improved spectral clustering algorithm is used, and the time-dependent strength matrix is used as the input of the similarity matrix. To determine the optimal number of clusters, the quality of different clustering results is evaluated using the silhouette coefficient, and the clustering scheme with the largest silhouette coefficient is selected. When determining the clustering boundary, the similarity of the node features is introduced as an auxiliary criterion, and the cosine similarity between the node feature vectors is calculated. When the dependence strength between two nodes is close to the clustering boundary (the difference is less than 0.1), if the feature similarity is higher than 0.85, they are divided into the same feature group. For example, 10 sensor nodes in an industrial environment are divided into 3 feature groups after clustering analysis: environmental parameter group (temperature, humidity, pressure), equipment state group (vibration, noise, current) and product quality group (size, weight, density, strength). For each feature group, the time series pattern of the nodes in the group is extracted to identify periodic, trend and abnormal patterns, and the time series evolution law of the nodes in the group is obtained.

[0110] Further analysis of the association between different feature groups can obtain the time series evolution law of the modal combination. By calculating the lag correlation between different feature groups, the time delay and strength of the inter-group influence are determined. For example, the change in the environmental parameter group will affect the equipment state group after 30 minutes, with an influence strength of 0.64; the change in the equipment state group will affect the product quality group after 45 minutes, with an influence strength of 0.78. Based on this, a collaborative evolution matrix of the feature groups is constructed, and the matrix elements represent the influence strength and time delay between different feature groups. Combining the collaborative evolution matrix with the internal time series evolution law of each group, the predicted features of the missing nodes can be generated.

[0111] For the prediction of missing nodes, first determine the feature group to which the node belongs, then combine the real-time data of other nodes in the same group and the inter-group relationship recorded in the collaborative evolution matrix to generate the predicted feature value. Taking the missing density sensor data in the product quality group as an example, using the size and weight data in the same group, combined with the historical data of the equipment state group 45 minutes ago, the missing density value is predicted to be 1.42 g / cm 3 .

[0112] To verify the prediction accuracy, compare the predicted features with the actual features, and calculate the root mean square error, mean absolute percentage error and other indicators. When the predicted density value is 1.42 g / cm 3 , the actual value is 1.45 g / cm 3 , the relative error is 2.07%. According to the prediction bias, the parameters in the collaborative evolution matrix are dynamically adjusted. When the cumulative prediction error exceeds the preset threshold (such as 5%), the matrix optimization process is triggered, and the matrix elements are updated by gradient descent method, so that the prediction result gradually approaches the actual value. The updated collaborative evolution matrix will be used for subsequent prediction tasks, forming a closed-loop optimization mechanism.

[0113] The time sequence characteristics are visualized as nodes by constructing an evolution trajectory diagram, which facilitates capturing complex time sequence patterns; the length of the time window is dynamically adjusted to adapt to the change frequency of different characteristics, thereby improving the accuracy of feature extraction; the time sequence dependence strength is calculated using multi-dimensional information entropy and state transition probability, which accurately quantifies the correlation degree between nodes; feature groups are formed based on node clustering to explore the internal time sequence evolution law and enhance the collaborative analysis capability between modalities; a collaborative evolution matrix is constructed to guide the prediction of missing node features, thereby realizing intelligent completion of data; the prediction model is optimized through prediction deviation feedback, thereby improving the accuracy of multi-modal missing data recovery and providing more complete and reliable feature input for sentiment recognition.

[0114] As shown in Figure 2 , a multi-modal fusion and sentiment recognition technology process based on time sequence feature analysis of the embodiment is shown.

[0115] In an optional implementation, the information entropy distribution of the time sequence characteristics is calculated, the time sequence completion interval is divided according to the information entropy distribution, the multi-modal fusion features are obtained by adaptively completing the predicted features in the time sequence completion interval, and the sample sentiment type is recognized based on the multi-modal fusion features, including:

[0116] The conditional information entropy and the joint information entropy of the time sequence characteristics in the time dimension are calculated, the information entropy distribution is generated by combination, the feature mutation points are identified according to the gradient change of the information entropy distribution, and the region between adjacent feature mutation points is determined as the time sequence completion interval;

[0117] The frequency distribution and the cumulative distribution of the time sequence characteristics in the time sequence completion interval are extracted, the probability density function is constructed, and the time sequence completion constraint condition is generated according to the change law of the probability density function and the historical data;

[0118] The feature importance is calculated based on the time sequence completion constraint condition, the predicted features are progressively weighted according to the feature importance, and the multi-modal fusion features are generated by adaptively completing the predicted features using the progressive weighting results;

[0119] The correlation coefficient and the mutual information value of the multi-modal fusion features in the time dimension and the feature dimension are calculated, the time sequence correlation degree is obtained by combination, the feature weighting is iteratively optimized according to the time sequence correlation degree, the optimized multi-modal fusion features are generated, and the optimized multi-modal fusion features are mapped to the sentiment feature space for sample sentiment recognition.

[0120] The embodiment first needs to calculate the conditional information entropy and the joint information entropy of the time sequence feature in the time dimension, and combine to generate the information entropy distribution. The conditional information entropy reflects the uncertainty of the current time feature value for the past time feature value, which can be obtained by calculating the conditional probability distribution of the current time feature value under the condition of the past time feature value. The joint information entropy represents the overall uncertainty of the feature values at multiple time points, which can be obtained by calculating the joint probability distribution of the feature values at multiple time points. Taking the audio emotion feature as an example, the emotion feature sequence of 5 seconds of continuous audio data is extracted. For each time point t, the conditional information entropy of the feature at this time point and the features at the previous 1 second to 3 seconds is calculated, and the joint information entropy of the features at this time point and the previous 1 second to 3 seconds is calculated. For video features and text features, similar methods are used to calculate their respective conditional information entropy and joint information entropy.

[0121] The conditional information entropy and the joint information entropy are combined in proportion, such as giving the conditional information entropy a weight of 0.6 and the joint information entropy a weight of 0.4, to obtain the comprehensive information entropy distribution. The feature mutation point is identified according to the gradient change of the information entropy distribution. The feature mutation point represents the position where the gradient change of the information entropy distribution is significant. Taking the audio emotion feature as an example, the first-order difference of the information entropy distribution is calculated, and when the absolute value of the difference exceeds a preset threshold (such as 0.25), the time point is marked as a feature mutation point. In actual scenarios, for example, when a user changes from a calm state to an excited state, the information entropy of the audio feature will change significantly, and a feature mutation point may occur at this time.

[0122] The region between adjacent feature mutation points is determined as the time sequence completion interval. For example, for 20 seconds of emotion interaction data, feature mutation points are detected at the 5th second and the 12th second, and the 5-12 second interval is determined as a time sequence completion interval. In multi-modal data, there may be multiple time sequence completion intervals, which need to be processed respectively.

[0123] The frequency distribution and the cumulative distribution of the time sequence feature in the time sequence completion interval are extracted. The frequency distribution describes the frequency of the feature value, and the cumulative distribution represents the probability that the feature value is less than or equal to a certain value. Taking a facial expression video as an example, the facial expression intensity feature in the time sequence completion interval is extracted, the frequency of different intensity values is counted, and the cumulative probability of each intensity value is calculated. A probability density function is constructed, which describes the probability density of the feature value at a certain point. For continuous features, the kernel density estimation method can be used to construct the probability density function; for discrete features, a histogram or a discrete probability distribution function can be used.

[0124] The time series completion constraint condition is generated according to the probability density function and the change rule of historical data. The time series completion constraint condition includes the value range, the change trend and the boundary condition of the characteristic value and the like. For example, for the voice emotion intensity feature, it can be obtained according to the historical data analysis that the voice emotion intensity usually fluctuates in the range of 0.7-0.9 in the excited emotion state, and the change trend is smooth, which is used as the constraint condition.

[0125] The feature importance is calculated based on the time series completion constraint condition. The feature importance reflects the contribution degree of each feature to the emotion recognition. The calculation method can be based on the stability, the discrimination degree and the time series correlation of the feature. The stability is calculated by the variance of the feature in the time series completion interval, and the smaller the variance is, the more stable the feature is; the discrimination degree is evaluated by the discrimination ability of the feature to different emotion categories, and the mutual information or correlation analysis can be used; the time series correlation is evaluated by the correlation degree of the feature and the historical feature. For the facial expression video, the importance of the eyebrow position feature is 0.8, the importance of the mouth corner position feature is 0.7, and the importance of the eye opening and closing feature is 0.6.

[0126] The prediction feature is progressively weighted according to the feature importance. The progressive weighting means that the weighting processing is sequentially performed from high to low according to the feature importance. The weight of the eyebrow position feature with the highest importance is adjusted first, and then the weights of the mouth corner position feature and the eye opening and closing feature are sequentially adjusted. For the missing video frame, the eyebrow position of the missing frame can be predicted according to the eyebrow position of the previous and subsequent frames, and the weight of 0.8 is given; then the mouth corner position is predicted and the weight of 0.7 is given; finally, the eye opening and closing feature is predicted and the weight of 0.6 is given.

[0127] The progressive weighting result is used to generate the multi-modal fusion feature by adaptively completing the prediction feature. The adaptive completion means that the completion strategy is dynamically adjusted according to the time series characteristics and the importance of the feature. For the slowly changing feature, linear interpolation completion can be used, and for the rapidly changing feature, nonlinear fitting completion can be used. The completed features of each modality are fused to generate the multi-modal fusion feature. For example, the video feature, the audio feature and the text feature after completion are weighted and fused according to the proportion of 0.4:0.3:0.3.

[0128] The correlation coefficient and mutual information value of the multi-modal fusion feature in the time dimension and the feature dimension are calculated, and a time sequence correlation degree is obtained by combination. The time dimension correlation coefficient reflects the change rule of the fusion feature with time, and the feature dimension correlation coefficient reflects the dependency relationship between different features. The mutual information value measures the mutual dependency between two variables. For example, the Pearson correlation coefficient of the fusion feature at adjacent time points is 0.85, the average correlation coefficient between different features is 0.72, the mutual information value at adjacent time points is 0.68, and the average mutual information value between different features is 0.64. These indicators are combined according to a preset weight (such as 0.3:0.3:0.2:0.2) to obtain the time sequence correlation degree.

[0129] The feature is iteratively optimized according to the time sequence correlation degree, and an optimized multi-modal fusion feature is generated. In the iterative optimization process, the weight of each feature is adjusted to maximize the time sequence correlation degree. For example, the time sequence correlation degree of the initial fusion feature is 0.75, and after 3 rounds of iterative optimization, the weights of the video, audio and text features are adjusted to 0.45:0.25:0.3, and at this time the time sequence correlation degree is improved to 0.82.

[0130] The optimized multi-modal fusion feature is mapped to a sentiment feature space for sample sentiment recognition. The sentiment feature space is a pre-defined multi-dimensional space, and each dimension corresponds to a sentiment attribute. The mapping method can use a nonlinear transformation or a deep neural network. In the sentiment feature space, the distance or similarity between the sample feature and each sentiment category center is calculated, and the sample is classified into the sentiment category with the closest distance or the highest similarity. For example, after mapping the fusion feature to the sentiment feature space, the similarity with the "happy", "sad", "angry", "fear", "surprise" and "disgust" six basic emotion categories is 0.82, 0.15, 0.05, 0.04, 0.08 and 0.06 respectively, so the sample is identified as the "happy" emotion type.

[0131] By accurately identifying the feature mutation point through the information entropy distribution, the intelligent division of the time sequence completion interval is realized; the rationality and continuity of feature completion are ensured; the progressive weighting strategy driven by feature importance is adopted to improve the accuracy of multi-modal data completion; the iterative optimization of feature weighting is realized through the calculation of time sequence correlation degree, which enhances the expression ability of multi-modal fusion feature; the adaptive completion mechanism can dynamically adjust the completion strategy according to the feature time sequence characteristics, effectively processing complex emotion change scenarios; the multi-dimensional correlation analysis ensures the time sequence consistency of the completed feature, and improves the accuracy and robustness of emotion recognition, especially in the case of data missing, still maintaining high recognition performance.

[0132] In a second aspect of the embodiment of the present application, a multi-modal missing data emotion recognition system based on dynamic completion is provided, and the system comprises:

[0133] The first unit is configured to acquire multi-modal input data, detect data missing positions of each mode, calculate a mode information integrity value and a quality score, and generate a mode reliability weight vector;

[0134] The second unit is configured to calculate a cross-correlation coefficient between modes in the multi-modal input data, determine a mode combination based on the cross-correlation coefficient and the reliability weight vector, detect a time sequence offset of the mode combination, dynamically align the multi-modal input data according to the time sequence offset, and extract a time sequence feature from the dynamically aligned data.

[0135] The third unit is configured to construct the time sequence feature as a node of an evolution trajectory graph, calculate a time sequence dependence strength between nodes, divide the nodes into feature groups according to the time sequence dependence strength, generate a predicted feature of a missing node based on a time sequence evolution rule of the mode combination, and combine the predicted feature with the multi-modal input data.

[0136] The fourth unit is configured to calculate an information entropy distribution of the time sequence feature, divide a time sequence completion interval according to the information entropy distribution, adaptively complete the predicted feature in the time sequence completion interval to obtain a multi-modal fusion feature, and identify a sample sentiment type based on the multi-modal fusion feature.

[0137] In a third aspect, an electronic device is provided, including:

[0138] a processor;

[0139] a memory for storing processor-executable instructions;

[0140] The processor is configured to invoke the instructions stored in the memory to execute the method described above.

[0141] In a fourth aspect, a computer-readable storage medium is provided, which stores computer program instructions. When the computer program instructions are executed by a processor, the method described above is implemented.

[0142] The present application can be a method, device, system and / or computer program product. The computer program product can include a computer readable storage medium having computer readable program instructions stored therein, which are used to perform various aspects of the present application.

[0143] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement to part or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for sentiment recognition of multi-modal missing data based on dynamic completion, characterized in that, The method comprises the following steps: acquiring multi-modal input data including video, audio and text, detecting data missing positions of each modality, calculating modality information integrity value and quality score, and generating modality reliability weight vector; calculating the cross-correlation coefficient between the modalities in the multi-modal input data, determining the modality combination based on the cross-correlation coefficient and the reliability weight vector, detecting the time sequence offset of the modality combination, and dynamically aligning the multi-modal input data according to the time sequence offset; and extracting the time sequence feature from the dynamically aligned data; constructing the time sequence feature into a node of an evolution trajectory graph, calculating the time sequence dependence strength between the nodes, dividing the nodes into feature groups according to the time sequence dependence strength, and generating predicted features of missing nodes in combination with the time sequence evolution law of the modality combination; calculating the conditional information entropy and the joint information entropy of the time sequence feature in the time dimension, combining to generate information entropy distribution, identifying feature mutation points according to the gradient change of the information entropy distribution, and determining the time sequence completion interval as the region between adjacent feature mutation points; extracting the frequency distribution and cumulative distribution of the time sequence feature in the time sequence completion interval, constructing a probability density function, and generating time sequence completion constraint conditions according to the change law of the probability density function and the historical data; calculating the feature importance in the time sequence completion constraint condition, progressively weighting the predicted features according to the feature importance, and adaptively completing the predicted features to generate multi-modal fusion features using the progressive weighting results; calculating the correlation coefficient and mutual information value of the multi-modal fusion features in the time dimension and the feature dimension, combining to obtain the time sequence correlation degree, iteratively optimizing the feature weighting according to the time sequence correlation degree, and generating optimized multi-modal fusion features; and mapping the optimized multi-modal fusion features to a sentiment feature space to perform sample sentiment recognition.

2. The method of claim 1, wherein, The method comprises the following steps: calculating the continuity difference value of adjacent data segments in the multi-modal input data, marking the difference positions higher than the difference threshold as data missing positions, and generating a data missing mark sequence; calculating the time sequence change feature of the non-missing data indicated by the data missing mark sequence, obtaining the time sequence correlation degree by feature fusion of the time sequence change feature, and calculating the information integrity value of each modality based on the time sequence correlation degree; extracting the content feature and structure feature of the non-missing data, and obtaining the quality feature by combining the content feature and the structure feature according to a preset weight; and calculating the quality score of each modality according to the quality feature; calculating the feature importance coefficient from the information integrity value and the quality score, normalizing the feature importance coefficient to obtain a modality reliability weight vector, and using the modality reliability weight vector to represent the importance of each modality feature in feature fusion.

3. The method of claim 1, wherein, calculating the cross-correlation coefficient between the modalities in the multi-modal input data, determining the modality combination based on the cross-correlation coefficient and the reliability weight vector, detecting the time sequence offset of the modality combination, and dynamically aligning the multi-modal input data according to the time sequence offset; and extracting the time sequence feature from the dynamically aligned data, The time sequence entropy values of each modality feature in the multi-modal input data are calculated, an adaptive sampling probability distribution is constructed based on the time sequence entropy values, the multi-modal input data is resampled according to the adaptive sampling probability distribution to obtain a sampling sequence, and a cross-correlation coefficient between the modalities in the sampling sequence is calculated; The cross-correlation coefficient and the time sequence entropy value are combined to construct a dynamic constraint matrix, the dynamic constraint matrix is multiplied by a reliability weight vector to obtain a modality confidence, a combination threshold is set according to the modality confidence, and a modality combination is constructed for a modality pair higher than the combination threshold; The time sequence offset feature is obtained based on the eigenvalue decomposition of the dynamic constraint matrix, the time sequence offset feature is reconstructed to obtain a time sequence offset, and the sampling sequence is dynamically aligned according to the time sequence offset to obtain an aligned sequence; The change feature of the aligned sequence is extracted and fused with a suppression vector to obtain a time sequence feature.

4. The method of claim 3, wherein, The time sequence offset feature is obtained based on the eigenvalue decomposition of the dynamic constraint matrix, the time sequence offset feature is reconstructed to obtain a time sequence offset, and the sampling sequence is dynamically aligned according to the time sequence offset to obtain an aligned sequence, including: The eigenvalue decomposition of the dynamic constraint matrix is performed, the decay curvature of the eigenvalue is calculated to determine the number of principal eigenvalues, the eigenvectors corresponding to the principal eigenvalues are extracted, and the eigenvectors are mapped to a nonlinear time sequence space to obtain a time sequence offset feature; A multi-scale time sequence sliding window is constructed based on the time sequence offset feature, a local time sequence change value is extracted in each sliding window, a global time sequence trend value is obtained by probability density estimation of the local time sequence change value, and the local time sequence change value and the global time sequence trend value are combined and reconstructed to obtain a time sequence offset; The relative displacement relationship between the modalities is calculated according to the time sequence offset, the sampling sequence is time sequence corrected, and an initial aligned sequence is generated; The time sequence consistency value and the feature correlation value of the initial aligned sequence are calculated, the time sequence offset is adjusted according to the consistency value and the correlation value, the initial aligned sequence is optimized, and an aligned sequence is obtained.

5. The method of claim 1, wherein, The time sequence feature is constructed as a node of an evolution trajectory graph, the time sequence dependence strength between nodes is calculated, the nodes are divided into feature groups according to the time sequence dependence strength, and the prediction feature of a missing node is generated by combining the time sequence evolution law of the modality combination, including: The time sequence feature is constructed as a node of an evolution trajectory graph, the length of a time window is dynamically adjusted according to the change trend of the time sequence feature, historical information is extracted in the time window to construct a node feature; The multi-dimensional information entropy and the state transition probability are used to calculate the time sequence dependence strength between nodes, the time sequence dependence strength is corrected according to the time sequence consistency of the node feature, and the final time sequence dependence strength is obtained; The nodes are clustered according to the time sequence dependence strength, the clustering boundary is determined based on the similarity of the node features, the nodes are divided into feature groups, and the time sequence evolution law of the nodes in the feature groups is obtained; The time sequence evolution law of the modality combination is obtained by analyzing the node features of different feature groups, a cooperative evolution matrix of the feature groups is constructed, the prediction feature of a missing node is generated by combining the cooperative evolution matrix and the time sequence evolution law, and The prediction feature is compared with the actual feature, a prediction deviation is calculated, and the cooperative evolution matrix is optimized according to the prediction deviation to update the prediction feature.

6. A multi-modal missing data sentiment recognition system based on dynamic completion for implementing the method of any one of the preceding claims 1-5, characterized in that, ​ The first unit is configured to acquire multi-modal input data including video, audio and text, detect data missing positions of each modality, calculate modality information completeness values and quality scores, and generate a modality reliability weight vector; The second unit is configured to calculate inter-correlation coefficients between modalities in the multi-modal input data, determine a modality combination based on the inter-correlation coefficients and the reliability weight vector, detect a time sequence offset of the modality combination, perform dynamic alignment on the multi-modal input data according to the time sequence offset, and extract time sequence features from the dynamically aligned data; The third unit is configured to construct the time sequence features into nodes of an evolution trajectory graph, calculate time sequence dependency strengths between the nodes, divide the nodes into feature groups according to the time sequence dependency strengths, generate predicted features of missing nodes in combination with a time sequence evolution rule of the modality combination, and perform adaptive completion on the predicted features in a time sequence completion interval to obtain multi-modal fusion features. The fourth unit is configured to calculate an information entropy distribution of the time sequence features, divide the time sequence completion interval according to the information entropy distribution, perform adaptive completion on the predicted features in the time sequence completion interval to obtain multi-modal fusion features, and identify a sample sentiment type based on the multi-modal fusion features.

7. An electronic device, comprising: The computer program instructions are executed by the processor to implement the method of any one of claims 1-5. The computer program instructions are executed by the processor to implement the method of any one of claims 1-5. ​ ​ 8. A computer-readable storage medium having stored thereon computer program instructions, wherein, ​

Images