Holographic display digital human speech recognition enhancement method based on multi-modal interaction learning

Abstract

The application discloses a holographic display digital human voice recognition enhancement method based on multi-modal interaction learning, and comprises the following steps: S1, collecting and preprocessing data to generate a multi-modal feature matrix; S2, inputting the multi-modal feature matrix into an adaptive multi-modal interaction kernel to generate multi-modal interaction features; S3, inputting the multi-modal interaction features into an emotion convolution block to generate semantic-emotion mapping features; S4, inputting the semantic-emotion mapping features into a cross-attention generator to output multi-modal fusion features; S5, inputting the multi-modal fusion features into a multi-modal fusion-decomposition bidirectional network to generate multi-modal features that have been decoupled and optimized; S6, inputting the multi-modal features into a holographic-voice coexistence optimization module to generate voice recognition and holographic display data; and S7, applying a self-supervised learning and contrast learning strategy to optimize parameters and realize adaptive adjustment. The application realizes enhancement of holographic display digital human voice recognition by using a multi-modal interaction learning method.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of multi-modal interactive learning and holographic display, and particularly relates to a holographic display digital human voice recognition enhancement method based on multi-modal interactive learning. BACKGROUND

[0002] In the field of multi-modal interactive technology and holographic display systems, with the continuous development of artificial intelligence and human-computer interaction technology, the combination of voice recognition and holographic display plays a crucial role in enhancing user experience and intelligent interaction. Current technology has achieved the enhancement of holographic display systems through multi-modal data (such as voice, vision, and environmental sound), enabling holographic digital humans to provide more natural interaction in voice communication and visual presentation. However, in complex multi-modal environments, achieving efficient, accurate, and real-time multi-modal data fusion and voice recognition still faces many challenges.

[0003] Existing holographic display systems and voice recognition technologies mainly rely on single modal or simple multi-modal fusion methods, which makes the system lack sufficient flexibility and accuracy when facing complex situations and dynamic environments. Some commonly used methods such as convolutional neural networks and recurrent neural networks, although perform well in specific tasks, often perform poorly in handling the synchronization and correlation of multi-modal data. These methods cannot fully capture the complex relationships and dependencies between multi-modal information such as voice, vision, and environmental sound, especially in terms of multi-modal feature weight adjustment and dynamic updating, making it difficult to respond to changes in context during user interaction. Therefore, existing voice recognition technologies are prone to recognition errors or delayed responses in complex emotional or dynamic environmental situations, making it difficult for holographic digital humans to achieve natural and smooth interaction.

[0004] In recent years, some technologies have attempted to improve the interactivity and responsiveness of the system through multi-modal deep learning models, such as using multi-head attention mechanisms and long short-term memory networks to enhance feature extraction and time series analysis. However, these methods have certain limitations in handling the depth and feature decoupling of multi-modal fusion. Joint representation and optimization of multi-modal data is a challenge, and traditional multi-modal fusion techniques often achieve feature combination through simple concatenation or weighted averaging, which leads to information redundancy and conflict between features, reducing the robustness and generalization ability of the model. In addition, existing systems lack sufficient context awareness in feature processing after fusion and decoupling, making the holographic display and voice recognition modules unable to synchronize and coordinate, thereby affecting the overall interaction effect.

[0005] The existing multi-modal interaction system still has the problem of insufficient dynamic modeling and updating of multi-modal feature relationships when dealing with complex scenes. Graph convolution networks and graph attention networks perform well in handling static relationships between nodes, but cannot meet the needs of efficient interaction in terms of dynamic modeling and real-time weight adjustment of multi-modal features. Due to the lack of dynamic adaptive mechanism, the system cannot update the interaction graph in time when dealing with real-time changes in multi-modal data, resulting in limited responsiveness and accuracy of speech recognition. In addition, the existing emotion understanding module mostly uses a single emotion analysis method, lacking the ability to jointly model emotions and semantics, limiting the speech recognition performance of the system in complex emotional scenarios.

[0006] To solve the above problems, the prior art attempts to enhance the time series analysis of multi-modal features by combining convolutional neural networks and bidirectional long short-term memory networks, but in the process of multi-modal fusion and decomposition, there are still problems of information loss and feature conflict, which cannot realize the deep fusion and fine-grained decoupling of multi-modal features, resulting in the inability to accurately extract and maintain feature independence and relevance in complex multi-modal interaction.

[0007] Therefore, although the prior art has made some progress in multi-modal data fusion and speech recognition, there are still deficiencies in realizing the real-time, context awareness and emotional complexity processing of holographic display digital human speech recognition.

[0008] Therefore, how to provide a holographic display digital human speech recognition enhancement method based on multi-modal interaction learning is a problem that those skilled in the art need to solve. SUMMARY

[0009] One object of the present application is to provide a holographic display digital human speech recognition enhancement method based on multi-modal interaction learning. The present application integrates dynamic graph convolution networks, adaptive multi-modal interaction kernels, emotion convolution blocks and cross-attention generators through a multi-modal interaction learning method, and realizes the enhancement of holographic display digital human speech recognition. This method significantly improves the context awareness and situational understanding of the system through deep fusion and decoupling optimization of multi-modal data. Combined with the global and time series processing advantages of the transformer and bidirectional long short-term memory network, it ensures the real-time synchronization of speech recognition and holographic display. The present application has the advantages of strong real-time, high emotional adaptability and natural and smooth interaction, solving the deficiencies of the prior art in complex scenarios.

[0010] The holographic display digital human speech recognition enhancement method based on multi-modal interaction learning according to the embodiment of the present application comprises the following steps:

[0011] S1, collecting voice data, visual data and environmental sound data of a user through a multi-sensor array, performing standardization and noise reduction preprocessing on the collected data, and generating a multi-modal feature matrix, the multi-modal feature matrix including voice features, visual features and environmental features;

[0012] S2, inputting the multi-modal feature matrix into an adaptive multi-modal interaction kernel, constructing and dynamically updating an interaction graph using a dynamic graph convolution network, adjusting the weights of the interaction graph through a graph attention mechanism, and generating optimized multi-modal interaction features;

[0013] S3, inputting the optimized multi-modal interaction features into an emotion convolution block, extracting visual emotion features and combining them with voice features to generate semantic-emotion mapping features;

[0014] S4, inputting the semantic-emotion mapping features into a cross-attention generator to perform feature cross-mapping and enhancement in a shared space, and outputting multi-modal fusion features with context awareness capability;

[0015] S5, inputting the multi-modal fusion features with context awareness capability into a multi-modal fusion-decomposition bidirectional network to perform deep fusion processing and generate decomposition path output, and generating decoupled and optimized multi-modal features;

[0016] S6, inputting the decoupled and optimized multi-modal features into a holographic-speech coexistence optimization module, processing using a hybrid architecture of transformer and bidirectional long short-term memory network, and generating real-time synchronized speech recognition and holographic display data;

[0017] S7, based on real-time synchronization data feedback information, applying self-supervised learning and contrast learning strategies for parameter optimization, and realizing adaptive adjustment of parameters through federated learning mechanism.

[0018] Optionally, the S1 specifically includes: performing spectral analysis on the voice data and extracting mel frequency cepstral coefficients to generate voice features; extracting visual features from the visual data using a double-branch convolution attention network; and performing background processing on the environmental sound data and extracting environmental features.

[0019] Optionally, the S2 specifically includes:

[0020] S21, inputting the multi-modal feature matrix after standardization and noise reduction preprocessing into an adaptive multi-modal interaction kernel;

[0021] S22, constructing an initial interaction graph G=(V, E) using a dynamic graph convolution network, where V represents a node set, each node corresponds to features of different modalities, E represents an edge set, and the weights of the edges are calculated according to the feature similarity and time dependence between nodes;

[0022] S23, in the initial interaction graph G, updating the node feature by a dynamic graph convolution network:

[0023]

[0024] wherein, represents the feature vector of node i in the l+1 layer, σ represents the activation function, N(i) represents the neighborhood set of node i, and α ij represents the attention weight between node i and node j, W (l) represents the weight matrix of the l-th layer, γ represents the neighborhood feature balance factor, represents the second-order neighborhood aggregation function, represents the feature vector of node p in the l-th layer, represents the feature vector of node j in the l-th layer, β represents the global feature balance coefficient, represents the pooling operation of node j, and b(1) represents the bias vector of the l-th layer.

[0025] S24, applying a graph attention mechanism to adjust the weights of the nodes and edges of the interaction graph;

[0026] S25, re-distributing the node features by the interaction graph adjusted by the graph attention mechanism, balancing the weights of different modal features by a multi-level feature aggregation and adaptive normalization mechanism, and generating multi-modal fusion features;

[0027] S26, outputting the multi-modal interaction features optimized by the dynamic graph convolution network and the graph attention mechanism.

[0028] Optionally, the S3 specifically comprises:

[0029] S31, inputting the optimized multi-modal interaction features into an emotion convolution block, the emotion convolution block being composed of multiple convolution layers and nonlinear activation units, extracting emotion features in visual data, the convolution layers using different convolution kernel sizes to capture emotion information of different scales and generating a preliminary emotion feature map;

[0030] S32, performing multi-layer convolution and pooling operation on the generated preliminary emotion feature map, the convolution layer using a nonlinear activation function for feature mapping, and the pooling layer being used to reduce the dimension of the preliminary emotion feature map, retain significant emotion features and reduce noise;

[0031] S33, applying an emotion attention mechanism to perform weighted processing on the extracted emotion features:

[0032]

[0033] wherein, β ijattention weight of position (i, j) in the preliminary emotion feature map, a represents a learnable parameter vector, W e weight matrix of the attention mechanism, E ij feature value of position (i, j) in the preliminary emotion feature map, E pq feature value of position (p, q) in the preliminary emotion feature map, E kl feature value of position (k, l) in the preliminary emotion feature map, E st feature value of position (s, t) in the preliminary emotion feature map, ||E ij -E pq || 2 square of the Euclidean distance between position (i, j) and position (p, q) in the preliminary emotion feature map, δ represents an adjustment factor, b e bias term, exp represents an exponential function, ||E kl -E st || 2 square of the Euclidean distance between position (k, l) and position (s, t) in the step emotion feature map, M represents the height of the preliminary emotion feature map, N represents the width of the preliminary emotion feature map, and ReLU represents an activation function.

[0034] S34, applying the calculated attention weight to the emotion feature map to generate a weighted emotion feature;

[0035] S35, jointly modeling the weighted emotion feature and the input speech feature to realize the combination of the emotion feature and the speech feature through a multi-modal fusion function, and generating a semantic-emotion mapping feature;

[0036] S36, outputting the generated semantic-emotion mapping feature as an input of the cross-attention generator.

[0037] Optionally, the S4 specifically comprises:

[0038] S41, inputting the semantic-emotion mapping feature into the cross-attention generator to uniformly embed different modal features, and the modal features have an aligned representation in a shared space;

[0039] S42, performing a multi-level cross-attention operation on the input modal features in the embedded shared space, identifying and strengthening the potential association between cross-modal features through a parallel attention mechanism, and forming a synergistic relationship between multiple contexts;

[0040] S43, performing hierarchical screening on the features subjected to the cross-attention operation, hierarchically evaluating the interaction features through different feature weight distribution strategies, retaining the most relevant interaction features and suppressing noise and redundant information.

[0041] S44, bidirectional enhancement processing is performed on the screened interaction features, global context information and local context information are extracted from each modality using a context association module;

[0042] S45, a normalization and adaptive adjustment strategy is applied to the interaction features after bidirectional enhancement, the balance of the interaction feature expression is dynamically adjusted, and adaptive distribution in the shared space is realized;

[0043] S46, output the multi-modal fusion features with context awareness.

[0044] Optionally, the S5 specifically includes:

[0045] S51, input the multi-modal fusion features with context awareness into a multi-modal fusion-decomposition bidirectional network, the multi-modal fusion-decomposition bidirectional network includes a fusion path and a decomposition path, the fusion path is used for deep fusion processing between multi-modal data, and the decomposition path is used for separating and optimizing the fused features;

[0046] S52, a multi-head feature fusion mechanism is applied in the fusion path, input features of different modalities are processed in layers, each head aggregates multi-modal features through different weight matrices and activation functions, and complex correlations between modalities are captured;

[0047] S53, in the last stage of the fusion path, a hierarchical feature synthesis module is introduced, weighted summation and nonlinear transformation are performed on the features of each modality, the hierarchical feature synthesis module adaptively combines the features according to the weight adjustment of each modality and context information, and generates a deep fusion multi-modal feature representation;

[0048] S54, the deep fusion multi-modal feature representation is transmitted to the decomposition path, the decomposition path applies an adaptive decomposition module, and the deep fusion multi-modal feature representation is decomposed into a plurality of sub-feature sets with independent information through matrix decomposition and feature decomposition strategies;

[0049] S55, the context-dependent decoupling unit is used in the decomposition path to further process the features, the weights and mutual relationships of each sub-feature set are dynamically adjusted, the feature expression is optimized, and the decoupling unit maintains the balance of information flow when decomposing the features;

[0050] S56, output the multi-modal features after decoupling optimization, the multi-modal features have independence and recognition ability after decomposition and optimization.

[0051] Optionally, the S6 specifically includes:

[0052] S61, input the decoupled and optimized multi-modal features into a holographic-voice coexistence optimization module, the holographic-voice coexistence optimization module is composed of a transformer network and a bidirectional long short-term memory network, and is used for realizing combination of global and time sequence processing of the multi-modal features;

[0053] S62, in the transformer network, the decoupled and optimized multi-modal features are processed by using a multi-head self-attention mechanism, global dependency relationships in different positions in the feature sequence are identified, relationships between features are established from different angles through the multi-head attention mechanism, rich context information is captured, and global feature representation is generated;

[0054] S63, the global feature representation processed through the transformer network is transmitted to the bidirectional long short-term memory network, and time sequence features are further extracted, the bidirectional long short-term memory network simultaneously captures history and future information of the input features through forward and backward paths, and generates bidirectional time sequence features;

[0055] S64, the global feature representation output by the transformer is combined with the bidirectional time sequence features output by the bidirectional long short-term memory network, joint optimization is performed through a feature weighted fusion module, the feature weighted fusion module dynamically adjusts weights of the global feature representation and the bidirectional time sequence features according to context importance of the features;

[0056] S65, in the process of feature fusion, an adaptive feature balancing strategy is applied, so that the global feature representation and the bidirectional time sequence features remain coordinated and consistent in different scenes, the adaptive feature balancing strategy automatically adjusts weight proportions of the features by monitoring performances of the features in real time;

[0057] S66, output the fused and optimized multi-modal real-time synchronization feature data, the multi-modal real-time synchronization feature data is used for decision processes of instant rendering of a voice recognition module and a holographic display system, and finally real-time synchronization voice recognition and holographic display data are generated.

[0058] The beneficial effects of the present application are:

[0059] Firstly, the present application generates a high-quality multi-modal feature matrix by using multi-modal data collected by a multi-sensor array, after standardization and denoising preprocessing, ensuring the reliability and accuracy of data input. The introduction of the adaptive multi-modal interaction kernel and the dynamic graph convolution network enables the multi-modal features to be fully expressed in the process of constructing and dynamically updating the interaction graph, and the weight of the interaction graph is adjusted through the graph attention mechanism, realizing real-time feature optimization and enhancing the adaptability of the system in different situations.

[0060] In addition, the application of the emotion convolution block and the construction of the semantic-emotion mapping layer can effectively extract visual emotion features in a complex emotional scene and combine them with voice features, thereby generating semantic-emotion mapping features with better situational understanding, and improving the performance of voice recognition in an emotional environment.

[0061] The design of the multi-modal fusion-decomposition bidirectional network provides an efficient solution for deep fusion and fine-grained decoupling of multi-modal features. Through this network, the system avoids information redundancy and feature conflict while maintaining feature independence and information integrity, ensuring the smoothness of multi-modal data in the fusion and decomposition process. The holographic-voice coexistence optimization module combines the advantages of the transformer and the bidirectional long short-term memory network, achieving efficient combination of global feature extraction and time series analysis, thereby generating real-time synchronized voice recognition and holographic display data. This module integrates global and sequence information to enable the holographic digital human to have natural voice response and holographic display effects in multi-modal interaction, significantly improving the real-time performance of the system.

[0062] Finally, through the application of self-supervised learning and contrastive learning strategies, the system can optimize parameters in real-time data feedback and adaptively adjust parameters through a federated learning mechanism, further improving the robustness and adaptability of the system in different scenarios. Overall, the present application effectively improves the accuracy, real-time performance, and intelligent adaptability of holographic display digital human voice recognition, making it more stable and efficient in complex multi-modal interaction scenarios, and solving the defects of existing technologies in multi-modal fusion depth, real-time weight adjustment, and emotion understanding. BRIEF DESCRIPTION OF DRAWINGS

[0063] The accompanying drawings are included to provide a further understanding of the present application, and constitute a part of the specification, illustrate embodiments of the present application, and are used to explain the present application, and do not constitute a limitation on the present application. In the drawings:

[0064] Figure 1 A flowchart of the holographic display digital human voice recognition enhancement method based on multi-modal interaction learning proposed by the present application;

[0065] Figure 2 A structure diagram of the emotion convolution block generating semantic-emotion mapping features in the holographic display digital human voice recognition enhancement method based on multi-modal interaction learning proposed by the present application. DETAILED DESCRIPTION

[0066] The application will be described in further detail below with reference to the drawings. These drawings are simplified schematic diagrams and only show the basic structure of the application in a schematic manner, and thus only show the components relevant to the application.

[0067] Reference Figure 1 and Figure 2 , the holographic display digital human voice recognition enhancement method based on multi-modal interaction learning includes the following steps:

[0068] S1, collect the voice data, visual data and environmental sound data of the user through a multi-sensor array, standardize and denoise the collected data, generate a multi-modal feature matrix, and the multi-modal feature matrix includes voice features, visual features and environmental features;

[0069] S2, input the multi-modal feature matrix into an adaptive multi-modal interaction kernel, use a dynamic graph convolution network to construct and dynamically update an interaction graph, adjust the weights of the interaction graph through a graph attention mechanism, and generate optimized multi-modal interaction features;

[0070] S3, input the optimized multi-modal interaction features into an emotion convolution block, extract visual emotion features and combine them with voice features to generate semantic-emotion mapping features;

[0071] S4, input the semantic-emotion mapping features into a cross-attention generator, perform feature cross-mapping and enhancement in a shared space, and output multi-modal fusion features with context awareness capability;

[0072] S5, input the multi-modal fusion features with context awareness capability into a multi-modal fusion-decomposition bidirectional network, perform deep fusion processing and generate decomposition path output, and generate decoupled and optimized multi-modal features;

[0073] S6, input the decoupled and optimized multi-modal features into a holographic-voice coexistence optimization module, use a hybrid architecture of transformer and bidirectional long short-term memory network for processing, and generate real-time synchronized voice recognition and holographic display data;

[0074] S7, based on real-time synchronization data feedback information, apply self-supervised learning and contrast learning strategies for parameter optimization, and realize adaptive adjustment of parameters through a federated learning mechanism.

[0075] In this embodiment, S1 specifically includes: performing spectral analysis on the voice data and extracting mel frequency cepstral coefficients to generate voice features; using a double-branch convolution attention network to extract visual features from the visual data; and performing background processing on the environmental sound data and extracting environmental features.

[0076] In this embodiment, S2 specifically includes:

[0077] S21, input the multi-modal feature matrix after standardization and noise reduction preprocessing into an adaptive multi-modal interaction kernel;

[0078] S22, construct an initial interaction graph G=V, E using a dynamic graph convolution network, where V represents a node set, each node corresponds to a feature of a different modality, E represents an edge set, and the weight of an edge is calculated according to the feature similarity and time dependence between nodes;

[0079] S23, in the initial interaction graph G, update the node features by the dynamic graph convolution network:

[0080]

[0081] wherein, represents the feature vector of node i in the l+1 layer, σ represents an activation function, N(i) represents the neighborhood set of node i, and α ij represents the attention weight between node i and node j, W (1) represents the weight matrix of the l layer, γ represents the neighborhood feature balance factor, represents a second-order neighborhood aggregation function, represents the feature vector of node p in the l layer, represents the feature vector of node j in the 1 layer, β represents a global feature balance coefficient, represents the pooling operation of node j, and b(1) represents the bias vector of the l layer;

[0082] S24, apply a graph attention mechanism to adjust the weights of the nodes and edges of the interaction graph;

[0083] S25, re-distribute the node features through the interaction graph adjusted by the graph attention mechanism, balance the weights of different modal features by using a multi-level feature aggregation and adaptive normalization mechanism, and generate multi-modal fusion features;

[0084] S26, output the multi-modal interaction features after multi-layer iteration optimization by the dynamic graph convolution network and the graph attention mechanism.

[0085] In the embodiment, the S3 specifically comprises:

[0086] S31, input the optimized multi-modal interaction features into an emotion convolution block, the emotion convolution block is composed of multiple convolution layers and nonlinear activation units, extract emotion features in visual data, the convolution layers use different convolution kernel sizes to capture emotion information of different scales, and generate a preliminary emotion feature map;

[0087] S32, performing multi-layer convolution and pooling operations on the generated preliminary emotion feature map, the convolution layer uses a nonlinear activation function for feature mapping, and the pooling layer is used to reduce the dimension of the preliminary emotion feature map, retain significant emotion features and reduce noise;

[0088] S33, applying an emotion attention mechanism to weight process the extracted emotion features:

[0089]

[0090] wherein β ij represents the attention weight of position (i, j) in the preliminary emotion feature map, a represents a learnable parameter vector, W e represents the weight matrix of the attention mechanism, E ij represents the feature value of position (i, j) in the preliminary emotion feature map, E pq represents the feature value of position (p, q) in the preliminary emotion feature map, E kl represents the feature value of position (k, l) in the preliminary emotion feature map, E st represents the feature value of position (s, t) in the preliminary emotion feature map, and ij -E pq ∥ 2 represents the square of the Euclidean distance between position (i, j) and position (p, q) in the preliminary emotion feature map, δ represents an adjustment factor, and b e represents a bias term, exp represents an exponential function, and kl -E st ∥ 2 represents the square of the Euclidean distance between position (k, l) and position (s, t) in the preliminary emotion feature map, M represents the height of the preliminary emotion feature map, N represents the width of the preliminary emotion feature map, and ReLU represents an activation function.

[0091] S34, applying the calculated attention weight to the emotion feature map to generate a weighted emotion feature;

[0092] S35, jointly modeling the weighted emotion feature and the input speech feature, combining the emotion feature and the speech feature through a multi-modal fusion function to generate a semantic-emotion mapping feature;

[0093] S36, outputting the generated semantic-emotion mapping feature as an input of the cross-attention generator.

[0094] In the embodiment, the S4 specifically comprises:

[0095] S41, inputting the semantic-emotion mapping feature into the cross-attention generator to uniformly embed different modal features, and the modal features have an aligned representation in a shared space.

[0096] S42, in the embedded shared space after the representation, the multi-level cross attention operation is performed on the input modal features, the potential correlation between the cross-modal features is recognized and strengthened through the parallel attention mechanism, and the synergistic relationship between multiple contexts is formed;

[0097] S43, the features subjected to the cross attention operation are subjected to hierarchical screening, the interactive features are subjected to hierarchical evaluation in different feature weight distribution strategies, the interactive features with the strongest correlation are retained, and noise and redundant information are suppressed;

[0098] S44, the screened interactive features are subjected to bidirectional enhancement processing, the global context information and the local context information are extracted from each modality by using a context association module;

[0099] S45, the interactive features subjected to bidirectional enhancement are subjected to normalization and adaptive adjustment strategies, the balance of the interactive feature expression is dynamically adjusted, and adaptive distribution in the shared space is realized;

[0100] S46, output the multi-modal fusion features with context awareness.

[0101] In the embodiment, the S5 specifically includes:

[0102] S51, input the multi-modal fusion features with context awareness into a multi-modal fusion-decomposition bidirectional network, the multi-modal fusion-decomposition bidirectional network includes a fusion path and a decomposition path, the fusion path is used for deep fusion processing between multi-modal data, and the decomposition path is used for separating and optimizing the fused features;

[0103] S52, a multi-head feature fusion mechanism is applied in the fusion path, the input features of different modalities are subjected to hierarchical processing, each head aggregates multi-modal features through different weight matrices and activation functions, and complex correlations between modalities are captured;

[0104] S53, in the last stage of the fusion path, a hierarchical feature synthesis module is introduced, the features of each modality are subjected to weighted summation and nonlinear transformation, the hierarchical feature synthesis module adaptively combines the features according to the weight adjustment of each modality and context information, and generates a deep fusion multi-modal feature representation;

[0105] S54, the deep fusion multi-modal feature representation is transmitted to the decomposition path, the decomposition path applies an adaptive decomposition module, and the deep fusion multi-modal feature representation is decomposed into a plurality of sub-feature sets with independent information through matrix decomposition and feature decomposition strategies;

[0106] S55, further processing the features using a context-dependent decoupling unit in the decomposition path, dynamically adjusting the weights and mutual relationships of each sub-feature set, optimizing the feature expression, and the decoupling unit maintains the balance of information flow when decomposing the features;

[0107] S56, outputting the decoupled and optimized multi-modal features, which have independence and recognition ability after decomposition and optimization.

[0108] In this embodiment, S6 specifically includes:

[0109] S61, inputting the decoupled and optimized multi-modal features into a holographic-voice co-evolution optimization module, which is composed of a transformer network and a bidirectional long short-term memory network, for realizing the combination of global and time sequence processing of multi-modal features;

[0110] S62, in the transformer network, the input decoupled and optimized multi-modal features are processed using a multi-head self-attention mechanism to identify the global dependency relationship between different positions in the feature sequence, and the relationship between features is established from different angles through the multi-head attention mechanism to capture rich context information and generate global feature representation;

[0111] S63, passing the global feature representation processed by the transformer network to the bidirectional long short-term memory network to further extract time sequence features, and the bidirectional long short-term memory network simultaneously captures the history and future information of the input features through the forward and backward paths to generate bidirectional time sequence features;

[0112] S64, combining the global feature representation output by the transformer with the bidirectional time sequence features output by the bidirectional long short-term memory network, and jointly optimizing through a feature weighting fusion module, which dynamically adjusts the weights of the global feature representation and the bidirectional time sequence features according to the context importance of the features;

[0113] S65, in the process of feature fusion, an adaptive feature balancing strategy is applied to make the global feature representation and the bidirectional time sequence features consistent in different scenarios, and the adaptive feature balancing strategy automatically adjusts the weight ratio of the features by monitoring their performance in real time;

[0114] S66, outputting the fused and optimized multi-modal real-time synchronous feature data, which is used for the decision-making process of the instant rendering of the speech recognition module and the holographic display system, and finally generates real-time synchronous speech recognition and holographic display data.

[0115] Example 1:

[0116] To verify the feasibility of the present application in implementation, the present application is applied to a certain intelligent conference system. The holographic display digital person as a virtual assistant participates in human-computer interaction, and needs to have quick and accurate speech recognition and emotion perception capabilities to improve the accuracy of conference records and interactivity with participants. The prior art often leads to high error rate and slow response when facing multi-modal data in a conference, especially in a multi-person conversation and a scene with large emotional fluctuations, because it cannot fully capture and fuse the complex relationship between speech, vision and environmental sound.

[0117] In this embodiment, a holographic display digital person speech recognition enhancement method based on multi-modal interaction learning is applied to solve the above problems. Various sensors are arranged in the conference room, including a high-precision microphone array, a camera and an environmental sound sensor, for collecting speech, facial expressions, body gestures and environmental sound data. The system inputs the collected multi-modal data into a data acquisition and preprocessing module for standardization and noise reduction processing to generate a multi-modal feature matrix. The feature matrix is inputted into an adaptive multi-modal interaction kernel and a dynamic graph convolution network to build and update an interaction graph, so that the system can dynamically identify the relationship between different modalities.

[0118] In the conference, when a participant speaks, the holographic display digital person not only recognizes the speech content, but also extracts the expression and posture information of the participant through the emotion convolution block, and combines the speech features to generate semantic-emotion mapping features. Subsequently, the cross-attention generator cross-maps and enhances the semantic-emotion mapping features with other modal features such as environmental sound in the shared space to output context-aware multi-modal fusion features. This enables the system to maintain high-accuracy speech recognition and smooth holographic display in a multi-person conversation, emotional changes and complex background noise.

[0119] To verify the effectiveness of the present application, multiple simulated conference experiments are conducted. The experiments are conducted in different situations, including normal conference, heated discussion, increased background noise and emotional excitement scenarios. By comparing the traditional single-modal processing method and the multi-modal interaction learning method of the present application, the accuracy of speech recognition, response time and naturalness of holographic display are recorded. The results show that in the heated discussion scenario, the speech recognition accuracy of the present application reaches 93.7%, while the traditional method is only 78.5%. In the case of increased background noise, the present application maintains an accuracy of 91.4% through adaptive multi-modal fusion and attention adjustment, while the traditional method drops to 74.2%. In addition, in the emotional excitement scenario, the system of the present application can quickly adjust the semantic-emotion mapping features, so that the response speed of the holographic display remains within 1.2 seconds, while the traditional method delays to 2.7 seconds.

[0120] Table 1 Comparison of holographic display digital person speech recognition performance experimental data

[0121]

[0122] From the analysis of Table 1, we can see that the performance of the present application is superior to the traditional method in various complex conference scenarios. In the normal conference scenario, the speech recognition accuracy of the present application reaches 95.3%, which is about 9.7 percentage points higher than the 85.6% of the traditional method, and the response time is 0.8 seconds, which is significantly better than the 1.3 seconds of the traditional method. This shows that the present application has higher recognition accuracy and reaction speed in the standard interactive environment.

[0123] In the intense discussion scenario, the speech recognition accuracy of the present application is 93.7%, while the traditional method is 78.5%, with a further gap of 15.2 percentage points. The response time in this scenario is 1.0 seconds, which is significantly lower than the 2.1 seconds of the traditional method, reflecting the efficiency of the present application in handling multiple speech inputs and complex interactions.

[0124] For the scenario with increased background noise, the accuracy of the present application is 91.4%, which remains at a high level, while the accuracy of the traditional method decreases to 74.2%. In this scenario, the response time of the present application is 1.1 seconds, while that of the traditional method is 2.5 seconds, indicating that the present application is more stable in dealing with external interference.

[0125] In the emotionally charged scenario, the speech recognition accuracy of the present application reaches 92.1%, far exceeding the 77.3% of the traditional method. The present application still maintains a shorter response time (1.2 seconds) in the context of changing emotions, while the traditional method takes 2.7 seconds, which shows that the present application has higher real-time performance and accuracy in dealing with complex emotional expressions.

[0126] Finally, in the multi-person conversation scenario, the accuracy of the present application is 90.5%, which is 16.7 percentage points higher than the 73.8% of the traditional method. Its response time is 1.0 seconds, while that of the traditional method is 2.4 seconds, further verifying the advantages of the present application in multi-modal data fusion and time series processing. These data comprehensively show that the present application has higher speech recognition accuracy and faster response speed in various complex scenarios, especially in dealing with background noise, multi-person communication and emotional changes.

[0127] Therefore, the speech recognition accuracy and response speed of the present application in different complex conference scenarios are significantly better than those of the traditional method, especially in challenging scenarios such as dealing with background noise, multi-person conversation and emotional excitement, showing higher robustness and real-time performance. Experimental data show that the present application effectively improves the speech recognition and interaction capabilities of the holographic display digital person through multi-modal interactive learning and innovative module design, has higher adaptability and stability, and can meet the complex and variable application requirements

[0128] The above description is only the preferred embodiment of the present application, but the protection scope of the present application is not limited to this. Any person skilled in the art can make equivalent substitutions or changes according to the technical solution and the inventive concept of the present application within the technical range disclosed by the present application, which should be covered in the protection scope of the present application.

Claims

1. A holographic display digital human speech recognition enhancement method based on multi-modal interaction learning, characterized in that, The method comprises the following steps: S1, collecting voice data, visual data and environmental sound data of a user through a multi-sensor array, performing standardization and noise reduction preprocessing on the collected data, and generating a multi-modal feature matrix, wherein the multi-modal feature matrix comprises voice features, visual features and environmental features; S2, inputting the multi-modal feature matrix into an adaptive multi-modal interaction kernel, constructing and dynamically updating an interaction graph using a dynamic graph convolution network, adjusting the weight of the interaction graph through a graph attention mechanism, and generating optimized multi-modal interaction features; S3, inputting the optimized multi-modal interaction features into an emotion convolution block, extracting visual emotion features and combining them with voice features to generate semantic-emotion mapping features; S4, inputting the semantic-emotion mapping features into a cross-attention generator to perform feature cross-mapping and enhancement in a shared space, and outputting multi-modal fusion features with context awareness; S5, inputting the multi-modal fusion features with context awareness into a multi-modal fusion-decomposition bidirectional network for deep fusion processing and generating decomposition path output to generate decoupled and optimized multi-modal features; S6, inputting the decoupled and optimized multi-modal features into a holographic-voice coexistence optimization module for processing using a hybrid architecture of a transformer and a bidirectional long short-term memory network to generate real-time synchronized voice recognition and holographic display data; S7, based on real-time synchronization data feedback information, applying self-supervised learning and contrast learning strategies for parameter optimization, and realizing adaptive adjustment of parameters through a federated learning mechanism; S1 specifically comprises: performing spectral analysis on the voice data and extracting mel frequency cepstral coefficients to generate voice features; extracting visual features from the visual data using a double-branch convolution attention network; and performing background processing on the environmental sound data and extracting environmental features; S3 specifically comprises: S31, inputting the optimized multi-modal interaction features into an emotion convolution block composed of multiple convolution layers and nonlinear activation units to extract emotion features from the visual data, wherein the convolution layers use different convolution kernel sizes to capture emotion information of different scales to generate a preliminary emotion feature map; S32, performing multi-layer convolution and pooling operations on the generated preliminary emotion feature map, wherein the convolution layer uses a nonlinear activation function for feature mapping, and the pooling layer is used to reduce the dimension of the preliminary emotion feature map, retain significant emotion features and reduce noise; S33, applying an emotion attention mechanism to weight process the extracted emotion features; wherein β ij represents the attention weight of position (i, j) in the preliminary emotion feature map, a represents a learnable parameter vector, W e represents a weight matrix of the attention mechanism, E ij represents the feature value of position (i, j) in the preliminary emotion feature map, E pq represents the feature value of position (p, q) in the preliminary emotion feature map, E kl represents the feature value of position (k, l) in the preliminary emotion feature map, E st represents the feature value of position (s, t) in the preliminary emotion feature map, ||E ij -E pq || 2 represents the square of the Euclidean distance between position (i, j) and position (p, q) in the preliminary emotion feature map, δ represents an adjustment factor, b e represents a bias term, exp represents an exponential function, ||E kl -E st || 2 represents the square of the Euclidean distance between position (k, l) and position (s, t) in the step emotion feature map, M represents the height of the preliminary emotion feature map, N represents the width of the preliminary emotion feature map, and ReLU represents an activation function. S34, applying the calculated attention weight to the emotion feature map to generate weighted emotion features; S35, jointly modeling the weighted emotion features and the input voice features to combine the emotion features and the voice features through a multi-modal fusion function to generate semantic-emotion mapping features; S36, outputting the generated semantic-emotion mapping features as input to the cross-attention generator.

2. The holographic display digital human speech recognition enhancement method based on multi-modal interaction learning according to claim 1, characterized in that, S2 specifically comprises: S21, inputting the multi-modal feature matrix after standardization and noise reduction preprocessing into an adaptive multi-modal interaction kernel; S22, an initial interaction graph G=(V, E) is constructed using a dynamic graph convolution network, where V represents a node set, each node corresponds to a feature of a different modality, E represents an edge set, and the weight of an edge is calculated according to the feature similarity and time dependence between nodes; S23, in the initial interaction graph G, node feature updating is performed by the dynamic graph convolution network; wherein, represents the feature vector of node i in the l+1th layer, σ represents an activation function, N(i) represents the neighborhood set of node i, and α ij represents the attention weight between node i and node j, W (l) represents the weight matrix of the lth layer, γ represents a neighborhood feature balancing factor, represents a second-order neighborhood aggregation function, represents the feature vector of node p in the lth layer, represents the feature vector of node j in the lth layer, β represents a global feature balancing coefficient, represents the pooling operation of node j, b (l) represents the bias vector of the lth layer; S24, a graph attention mechanism is applied to adjust the weights of the nodes and edges of the interaction graph; S25, the interaction graph adjusted by the graph attention mechanism is used to re-distribute the node features, and a multi-level feature aggregation and adaptive normalization mechanism is used to balance the weights of different modality features to generate multi-modal fusion features; S26, output the multi-modal interaction features optimized by the dynamic graph convolution network and the multi-layer iteration of the graph attention mechanism.

3. The holographic display digital human speech recognition enhancement method based on multi-modal interaction learning according to claim 1, characterized in that, The S4 specifically includes: S41, input the semantic-emotion mapping features into a cross-attention generator to uniformly embed and represent different modality features, and the modality features have an aligned representation in a shared space; S42, in the embedded shared space, perform multi-level cross-attention operations on the input modality features, identify and strengthen the potential associations between cross-modal features through parallel attention mechanisms, and form collaborative relationships between multiple contexts; S43, the features after the cross-attention operation are hierarchically filtered, the interaction features are hierarchically evaluated with different feature weight distribution strategies, the most relevant interaction features are retained, and noise and redundant information are suppressed; S44, the filtered interaction features are processed by bidirectional enhancement, and the context association module is used to extract global context information and local context information from each modality; S45, apply normalization and adaptive adjustment strategies to the bidirectional enhanced interaction features to dynamically adjust the balance of the interaction feature expression and achieve adaptive distribution in the shared space; S46, output the multi-modal fusion features with context awareness.

4. The holographic display digital human speech recognition enhancement method based on multi-modal interaction learning according to claim 1, characterized in that, The S5 specifically includes: S51, input the multi-modal fusion features with context awareness into a multi-modal fusion-decomposition bidirectional network, the multi-modal fusion-decomposition bidirectional network includes a fusion path and a decomposition path, the fusion path is used for deep fusion processing between multi-modal data, and the decomposition path is used for separating and optimizing the fused features; S52, in the fusion path, a multi-head feature fusion mechanism is applied to hierarchically process the input features of different modalities, each head aggregates multi-modal features through different weight matrices and activation functions to capture complex associations between modalities; S53, in the last stage of the fusion path, a hierarchical feature synthesis module is introduced to weight-sum and nonlinearly transform the features of each modality, the hierarchical feature synthesis module adaptively combines the features according to the weight adjustment and context information of each modality to generate a deep fusion multi-modal feature representation; S54, the deep fusion multi-modal feature representation is transmitted to the decomposition path, the decomposition path applies an adaptive decomposition module to decompose the deep fusion multi-modal feature representation into several sub-feature sets with independent information through matrix decomposition and feature decomposition strategies; S55, further processing the features using a context-dependent decoupling unit in the decomposition path, dynamically adjusting the weights and mutual relationships of each sub-feature set, optimizing feature expression, the decoupling unit maintaining the balance of information flow when decomposing features; S56, outputting the multi-modal features optimized by decoupling, the multi-modal features having independence and recognition ability after decomposition and optimization.

5. The holographic display digital human speech recognition enhancement method based on multi-modal interaction learning according to claim 1, characterized in that, The S6 specifically includes: S61, inputting the multi-modal features optimized by decoupling into a holographic-voice coexistence optimization module, the holographic-voice coexistence optimization module being composed of a transformer network and a bidirectional long short-term memory network, for realizing the combination of global and time sequence processing of multi-modal features; S62, in the transformer network, processing the input multi-modal features optimized by decoupling using a multi-head self-attention mechanism, identifying the global dependency relationship of different positions in the feature sequence, establishing the relationship between features from different angles through the multi-head attention mechanism, capturing rich context information and generating global feature representation; S63, transmitting the global feature representation processed by the transformer network to the bidirectional long short-term memory network, further extracting time sequence features, the bidirectional long short-term memory network simultaneously capturing the history and future information of the input features through the forward and backward paths, generating bidirectional time sequence features; S64, combining the global feature representation output by the transformer with the bidirectional time sequence features output by the bidirectional long short-term memory network, and jointly optimizing through a feature weighting fusion module, the feature weighting fusion module dynamically adjusting the weights of the global feature representation and the bidirectional time sequence features according to the context importance of the features; S65, in the process of feature fusion, applying an adaptive feature balancing strategy to make the global feature representation and the bidirectional time sequence features consistent in different scenarios, the adaptive feature balancing strategy automatically adjusting the weight ratio of the features by real-time monitoring of the performance of the features; S66, outputting the multi-modal real-time synchronous feature data optimized by fusion, the multi-modal real-time synchronous feature data being used for the decision-making process of the instant rendering of the speech recognition module and the holographic display system, and finally generating real-time synchronous speech recognition and holographic display data.

Images