Holographic display digital human speech recognition enhancement method based on multi-mode interactive learning

Abstract

The invention discloses a holographic display digital human speech recognition enhancement method based on multi-modal interactive learning, and the method comprises the following steps: S1, collecting and preprocessing data, and generating a multi-modal feature matrix; s2, inputting the multi-modal feature matrix into a self-adaptive multi-modal interaction kernel to generate multi-modal interaction features; s3, inputting the multi-modal interaction features into an emotion convolution block to generate semantic-emotion mapping features; s4, inputting the semantic-emotion mapping features into a cross attention generator, and outputting multi-modal fusion features; s5, inputting the multi-modal fusion features into a multi-modal fusion-decomposition two-way network, and generating multi-modal features after decoupling optimization; s6, inputting the multi-modal features into a holographic-speech symbiosis optimization module to generate speech recognition and holographic display data; and S7, performing parameter optimization by applying self-supervised learning and comparative learning strategies to realize self-adaptive adjustment. According to the invention, the enhancement of holographic display digital human speech recognition is realized by using a multi-mode interactive learning method.

Description

Technical Field

[0001] The present invention relates to the technical field of multimodal interactive learning and holographic display, and in particular to a holographic display digital human speech recognition enhancement method based on multimodal interactive learning. Background Art

[0002] In the field of multimodal interaction technology and holographic display systems, with the continuous development of artificial intelligence and human-computer interaction technology, the combination of speech recognition and holographic display plays a vital role in enhancing user experience and intelligent interaction. Current technology has achieved the performance of holographic display systems through multimodal data (such as speech, vision and ambient sound), enabling holographic digital people to provide more natural interactions in voice communication and visual presentation. However, in complex multimodal environments, there are still many challenges to achieving efficient, accurate and real-time multimodal data fusion and speech recognition.

[0003] Existing holographic display systems and speech recognition technologies mainly rely on a single modality or simple multimodal fusion methods, which makes the system lack sufficient flexibility and accuracy when facing complex situations and dynamic environments. Although some commonly used methods such as convolutional neural networks and recurrent neural networks perform well in specific tasks, they often perform poorly when dealing with the synchronization and association of multimodal data. These methods cannot fully capture the complex relationships and dependencies between multimodal information such as speech, vision, and ambient sound, especially in terms of weight adjustment and dynamic update of multimodal features, and it is difficult to cope with context changes during user interaction. Therefore, existing speech recognition technology is prone to recognition errors or delayed reactions in situations with complex emotions or dynamic changes in the environment, making it difficult for holographic digital humans to achieve natural and smooth interaction.

[0004] In recent years, some technologies have attempted to improve the interactivity and responsiveness of the system through multimodal deep learning models, such as using multi-head attention mechanisms and long short-term memory networks to enhance feature extraction and time series analysis. However, these methods have certain limitations in dealing with the depth of multimodal fusion and feature decoupling. The joint expression and optimization of multimodal data is a challenge. Traditional multimodal fusion techniques often achieve feature combination through simple splicing or weighted averaging, which leads to information redundancy and conflicts between features, reducing the robustness and generalization ability of the model. In addition, the existing system lacks sufficient perception of the context in the feature processing after fusion and decoupling, making it impossible for the holographic display and speech recognition modules to coordinate synchronously, thus affecting the overall interaction effect.

[0005] Existing multimodal interaction systems still have problems with dynamic modeling and updating of multimodal feature relationships when dealing with complex scenarios. Graph convolutional networks and graph attention networks perform well in processing static relationships between nodes, but cannot meet the needs of efficient interaction in terms of dynamic modeling and real-time weight adjustment of multimodal features. Due to the lack of a dynamic adaptive mechanism, the system is unable to update the interaction graph in a timely manner when processing real-time changes in multimodal data, resulting in limited responsiveness and accuracy of speech recognition. In addition, existing emotion understanding modules mostly use a single emotion analysis method and lack the ability to jointly model emotions and semantics, which limits the system's speech recognition performance in complex emotional scenarios.

[0006] In order to solve the above problems, existing technologies attempt to enhance the time series analysis of multimodal features by combining models such as convolutional neural networks and bidirectional long short-term memory networks. However, in the process of multimodal fusion and decomposition, there are still problems of information loss and feature conflict, and it is impossible to achieve deep fusion and fine-grained decoupling of multimodal features, which leads to the inability to accurately extract and maintain feature independence and correlation in complex multimodal interactions.

[0007] Therefore, although the existing technology has made certain progress in multimodal data fusion and speech recognition, it still has shortcomings in achieving real-time speech recognition, context perception capabilities, and emotional complexity processing of holographic display digital human.

[0008] Therefore, how to provide a holographic display digital human speech recognition enhancement method based on multimodal interactive learning is a problem that technical personnel in this field urgently need to solve. Summary of the invention

[0009] One purpose of the present invention is to propose a method for enhancing speech recognition of digital humans in holographic display based on multimodal interactive learning. The present invention integrates a dynamic graph convolutional network, an adaptive multimodal interactive kernel, an emotional convolution block, and a cross-attention generator through a multimodal interactive learning method, thereby achieving enhanced speech recognition of digital humans in holographic display. This method significantly improves the context perception and situational understanding of the system through deep fusion and decoupling optimization of multimodal data. Combining the global and time series processing advantages of the transformer and the bidirectional long short-term memory network, real-time synchronization of speech recognition and holographic display is ensured. The present invention has the advantages of strong real-time performance, high emotional adaptability, and natural and smooth interaction, which solves the shortcomings of the prior art in complex scenarios.

[0010] According to an embodiment of the present invention, a method for enhancing speech recognition of a holographically displayed digital human based on multimodal interactive learning comprises the following steps:

[0011] S1. Collecting the user's voice data, visual data and environmental sound data through a multi-sensor array, standardizing and pre-noising the collected data, and generating a multimodal feature matrix, wherein the multimodal feature matrix includes voice features, visual features and environmental features;

[0012] S2. Input the multimodal feature matrix into the adaptive multimodal interaction kernel, use the dynamic graph convolutional network to build and dynamically update the interaction graph, adjust the weight of the interaction graph through the graph attention mechanism, and generate optimized multimodal interaction features;

[0013] S3, input the optimized multimodal interaction features into the emotion convolution block, extract the visual emotion features and combine them with the speech features to generate semantic-emotion mapping features;

[0014] S4, input the semantic-emotion mapping features into the cross-attention generator, perform feature cross-mapping and enhancement in the shared space, and output multimodal fusion features with context-awareness;

[0015] S5, inputting the multimodal fusion features with context perception into the multimodal fusion-decomposition bidirectional network, performing deep fusion processing and generating decomposition path output, thereby generating decoupled and optimized multimodal features;

[0016] S6. Input the decoupled optimized multimodal features into the holographic-speech symbiosis optimization module, process them using the hybrid architecture of the transformer and the bidirectional long short-term memory network, and generate real-time synchronous speech recognition and holographic display data;

[0017] S7. Based on real-time synchronous data feedback information, self-supervised learning and contrastive learning strategies are applied to optimize parameters, and adaptive adjustment of parameters is achieved through the federated learning mechanism.

[0018] Optionally, S1 specifically includes: performing spectral analysis on the speech data and extracting Mel-frequency cepstral coefficients to generate speech features; extracting visual features from the visual data using a dual-branch convolutional attention network; performing background processing on the ambient sound data and extracting environmental features.

[0019] Optionally, the S2 specifically includes:

[0020] S21, inputting the multimodal feature matrix after standardization and noise reduction preprocessing into the adaptive multimodal interaction kernel;

[0021] S22. Use a dynamic graph convolutional network to construct an initial interaction graph G = (V, E), where V represents a set of nodes, each node corresponds to features of different modalities, and E represents a set of edges, and the weights of the edges are calculated based on the feature similarity and time dependency between nodes;

[0022] S23. In the initial interaction graph G, node features are updated through a dynamic graph convolutional network:

[0023]

[0024] in, represents the feature vector of node i in the l+1th layer, σ represents the activation function, N(i) represents the neighborhood set of node i, α ij represents the attention weight between node i and node j, W (l) represents the weight matrix of the lth layer, γ represents the neighborhood feature balance factor, represents the second-order neighborhood aggregation function, represents the feature vector of node p in layer l, represents the feature vector of node j in the lth layer, β represents the global feature balance coefficient, represents the pooling operation of node j, b(1) represents the bias vector of layer l;

[0025] S24. Apply the graph attention mechanism to adjust the weights of the nodes and edges of the interaction graph;

[0026] S25, through the interaction graph adjusted by the graph attention mechanism, node features are redistributed, and the weights of different modal features are balanced by multi-level feature aggregation and adaptive normalization mechanism to generate multi-modal fusion features;

[0027] S26. Output the multimodal interaction features after multi-layer iterative optimization of the dynamic graph convolutional network and graph attention mechanism.

[0028] Optionally, the S3 specifically includes:

[0029] S31, inputting the optimized multimodal interaction features into the emotion convolution block, wherein the emotion convolution block is composed of multiple convolution layers and nonlinear activation units, extracting emotion features from visual data, wherein the convolution layers use different convolution kernel sizes to capture emotion information of different scales, and generate a preliminary emotion feature map;

[0030] S32, performing multi-layer convolution and pooling operations on the generated preliminary emotion feature map, wherein the convolution layer uses a nonlinear activation function for feature mapping, and the pooling layer is used to reduce the dimension of the preliminary emotion feature map, retain significant emotion features and reduce noise;

[0031] S33. Apply the emotional attention mechanism to perform weighted processing on the extracted emotional features:

[0032]

[0033] Among them, β ijrepresents the attention weight of position (i, j) in the preliminary emotion feature map, a represents the learnable parameter vector, and W e Represents the weight matrix of the attention mechanism, E ij represents the feature value of position (i, j) in the preliminary emotion feature map, E pq represents the feature value of the position (p, q) in the preliminary emotion feature map, E kl represents the feature value of position (k, l) in the preliminary emotion feature map, E st represents the feature value of position (s, t) in the preliminary emotion feature map, ||E ij -E pq || 2 represents the square of the Euclidean distance between position (i, j) and position (p, q) in the preliminary emotion feature map, δ represents the adjustment factor, and b e represents the bias term, exp represents the exponential function, ||E kl -E st || 2 represents the square of the Euclidean distance between the position (k, l) and the position (s, t) in the first emotion feature map, M represents the height of the preliminary emotion feature map, N represents the width of the preliminary emotion feature map, and ReLU represents the activation function;

[0034] S34, applying the calculated attention weight to the emotion feature map to generate a weighted emotion feature;

[0035] S35, jointly modeling the weighted emotion feature and the input speech feature, combining the emotion feature and the speech feature through a multimodal fusion function, and generating a semantic-emotion mapping feature;

[0036] S36. Output the generated semantic-sentiment mapping features as input to the cross-attention generator.

[0037] Optionally, the S4 specifically includes:

[0038] S41, inputting the semantic-emotion mapping features into a cross-attention generator to uniformly embed the different modal features, wherein the modal features have aligned representations in a shared space;

[0039] S42. In the shared space after embedding, multi-level cross-attention operations are performed on the input modal features, and the potential correlation between cross-modal features is identified and strengthened through the parallel attention mechanism, forming a synergistic relationship between multiple contexts.

[0040] S43, hierarchical screening of features after cross-attention operation, hierarchical evaluation of interactive features with different feature weight distribution strategies, retaining the most relevant interactive features and suppressing noise and redundant information;

[0041] S44, performing bidirectional enhancement processing on the filtered interaction features, and using a context association module to extract global context information and local context information from each modality;

[0042] S45, applying normalization and adaptive adjustment strategies to the interactive features after bidirectional enhancement, dynamically adjusting the balance of interactive feature expression, and achieving adaptive distribution in the shared space;

[0043] S46. Output multimodal fusion features with context-awareness.

[0044] Optionally, the S5 specifically includes:

[0045] S51, inputting the multimodal fusion features with context perception capability into a multimodal fusion-decomposition bidirectional network, wherein the multimodal fusion-decomposition bidirectional network includes a fusion path and a decomposition path, wherein the fusion path is used to perform deep fusion processing between multimodal data, and the decomposition path is used to separate and optimize the fused features;

[0046] S52. In the fusion path, a multi-head feature fusion mechanism is applied to process the input features of different modalities in layers. Each head aggregates multimodal features through different weight matrices and activation functions to capture the complex associations between the modalities.

[0047] S53, at the final stage of the fusion path, a hierarchical feature synthesis module is introduced to perform weighted summation and nonlinear transformation on the features of each modality, wherein the hierarchical feature synthesis module adaptively combines features according to weight adjustment of each modality and context information to generate a deeply fused multimodal feature representation;

[0048] S54, passing the deeply fused multimodal feature representation to a decomposition path, wherein the decomposition path applies an adaptive decomposition module to decompose the deeply fused multimodal feature representation into a plurality of sub-feature sets with independent information through matrix decomposition and feature decomposition strategies;

[0049] S55, further processing the features using a context-dependent decoupling unit in the decomposition path, dynamically adjusting the weights and relationships of each sub-feature set, and optimizing feature expression, wherein the decoupling unit maintains a balance of information flow when decomposing the features;

[0050] S56. Outputting decoupled and optimized multimodal features, wherein the multimodal features have independence and recognition capabilities after decomposition and optimization.

[0051] Optionally, the S6 specifically includes:

[0052] S61, inputting the decoupled optimized multimodal features into a holographic-speech symbiosis optimization module, wherein the holographic-speech symbiosis optimization module is composed of a transformer network and a bidirectional long short-term memory network, and is used to realize the combination of global and time series processing of multimodal features;

[0053] S62. In the transformer network, the multi-head self-attention mechanism is used to process the decoupled and optimized multimodal features of the input, identify the global dependencies at different positions in the feature sequence, establish the relationship between features from different angles through the multi-head self-attention mechanism, capture rich contextual information and generate global feature representation;

[0054] S63, passing the global feature representation processed by the transformer network to a bidirectional long short-term memory network to further extract time series features, wherein the bidirectional long short-term memory network simultaneously captures historical and future information of the input features through forward and backward paths to generate bidirectional time series features;

[0055] S64, combining the global feature representation output by the transformer and the bidirectional time series features output by the bidirectional long short-term memory network, and performing joint optimization through a feature weighted fusion module, wherein the feature weighted fusion module dynamically adjusts the weights of the global feature representation and the bidirectional time series features according to the contextual importance of the features;

[0056] S65. In the process of feature fusion, an adaptive feature balancing strategy is applied to keep the global feature representation and the bidirectional time series features consistent in different scenarios, wherein the adaptive feature balancing strategy automatically adjusts the weight ratio of the features by monitoring their performance in real time;

[0057] S66. Output the fused and optimized multimodal real-time synchronous feature data, which is used in the decision-making process of instant rendering of the speech recognition module and the holographic display system, and finally generates real-time synchronous speech recognition and holographic display data.

[0058] The beneficial effects of the present invention are:

[0059] First, the present invention uses multimodal data collected by a multi-sensor array, and after standardization and noise reduction preprocessing, generates a high-quality multimodal feature matrix to ensure the reliability and accuracy of data input. The introduction of adaptive multimodal interaction kernels and dynamic graph convolutional networks enables multimodal features to be fully expressed in the process of constructing and dynamically updating interaction graphs. The interaction graph is weighted through the graph attention mechanism, achieving real-time feature optimization and enhancing the adaptability of the system in different scenarios.

[0060] In addition, the application of emotion convolution blocks and the construction of semantic-emotion mapping layers can effectively extract visual emotion features in complex emotional scenes and combine them with speech features, thereby generating more context-aware semantic-emotion mapping features, improving the performance of speech recognition in emotionally variable environments. The cross-attention generator outputs multimodal fusion features with context-awareness through cross-mapping and enhancement of features in a shared space, which can accurately understand and respond to multimodal input signals.

[0061] The design of the multimodal fusion-decomposition bidirectional network provides an efficient solution for the deep fusion and fine-grained decoupling of multimodal features. Through this network, the system avoids information redundancy and feature conflict while maintaining feature independence and information integrity, ensuring the fluency of multimodal data in the fusion and decomposition process. The holographic-speech symbiosis optimization module combines the advantages of the transformer and the bidirectional long short-term memory network, realizing the efficient combination of global feature extraction and time series analysis, thereby generating real-time synchronized speech recognition and holographic display data. By comprehensively processing global and sequence information, this module enables the holographic digital human to have natural voice response and holographic display effects in multimodal interaction, significantly improving the real-time performance of the system.

[0062] Finally, through the application of self-supervised learning and contrastive learning strategies, the system can optimize parameters in real-time data feedback, and realize adaptive adjustment of parameters through the federated learning mechanism, further improving the robustness of the system and its adaptability in different scenarios. Overall, the present invention effectively improves the accuracy, real-time and intelligent adaptability of holographic display digital human voice recognition, making it more stable and efficient in complex multimodal interaction scenarios, and solves the defects of existing technologies in multimodal fusion depth, real-time weight adjustment and emotion understanding. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0064] Figure 1 This is a flow chart of the method for enhancing speech recognition of holographic display digital human based on multimodal interactive learning proposed by the present invention;

[0065] Figure 2 This is a structural schematic diagram of the generation of semantic-emotion mapping features by the emotion convolution block of the holographic display digital human speech recognition enhancement method based on multimodal interactive learning proposed in the present invention. DETAILED DESCRIPTION

[0066] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, which only illustrate the basic structure of the present invention in a schematic manner, and therefore only show the components related to the present invention.

[0067] refer to Figure 1 and Figure 2 , a holographic display digital human speech recognition enhancement method based on multimodal interactive learning includes the following steps:

[0068] S1. Collecting the user's voice data, visual data and environmental sound data through a multi-sensor array, standardizing and pre-noising the collected data, and generating a multimodal feature matrix, wherein the multimodal feature matrix includes voice features, visual features and environmental features;

[0069] S2. Input the multimodal feature matrix into the adaptive multimodal interaction kernel, use the dynamic graph convolutional network to build and dynamically update the interaction graph, adjust the weight of the interaction graph through the graph attention mechanism, and generate optimized multimodal interaction features;

[0070] S3, input the optimized multimodal interaction features into the emotion convolution block, extract the visual emotion features and combine them with the speech features to generate semantic-emotion mapping features;

[0071] S4, input the semantic-emotion mapping features into the cross-attention generator, perform feature cross-mapping and enhancement in the shared space, and output multimodal fusion features with context-awareness;

[0072] S5, inputting the multimodal fusion features with context perception into the multimodal fusion-decomposition bidirectional network, performing deep fusion processing and generating decomposition path output, thereby generating multimodal features after decoupling optimization;

[0073] S6. Input the decoupled optimized multimodal features into the holographic-speech symbiosis optimization module, process them using the hybrid architecture of the transformer and the bidirectional long short-term memory network, and generate real-time synchronous speech recognition and holographic display data;

[0074] S7. Based on real-time synchronous data feedback information, self-supervised learning and contrastive learning strategies are applied to optimize parameters, and adaptive adjustment of parameters is achieved through the federated learning mechanism.

[0075] In this embodiment, S1 specifically includes: performing spectral analysis on the voice data and extracting Mel-frequency cepstral coefficients to generate voice features; extracting visual features from the visual data using a dual-branch convolutional attention network; performing background processing on the ambient sound data and extracting environmental features.

[0076] In this implementation, S2 specifically includes:

[0077] S21, inputting the multimodal feature matrix after standardization and noise reduction preprocessing into the adaptive multimodal interaction kernel;

[0078] S22. Use the dynamic graph convolutional network to construct the initial interaction graph G = V, E, where V represents a set of nodes, each node corresponds to features of different modalities, and E represents a set of edges, and the weights of the edges are calculated based on the feature similarity and time dependency between nodes;

[0079] S23. In the initial interaction graph G, node features are updated through a dynamic graph convolutional network:

[0080]

[0081] in, represents the feature vector of node i in the l+1th layer, σ represents the activation function, N(i) represents the neighborhood set of node i, α ij represents the attention weight between node i and node j, W (1) represents the weight matrix of the lth layer, γ represents the neighborhood feature balance factor, represents the second-order neighborhood aggregation function, represents the feature vector of node p in layer l, represents the feature vector of node j in layer 1, β represents the global feature balance coefficient, represents the pooling operation of node j, b(1) represents the bias vector of layer l;

[0082] S24. Apply the graph attention mechanism to adjust the weights of the nodes and edges of the interaction graph;

[0083] S25, through the interaction graph adjusted by the graph attention mechanism, node features are redistributed, and the weights of different modal features are balanced by multi-level feature aggregation and adaptive normalization mechanism to generate multi-modal fusion features;

[0084] S26. Output the multimodal interaction features after multi-layer iterative optimization of the dynamic graph convolutional network and graph attention mechanism.

[0085] In this implementation, S3 specifically includes:

[0086] S31, inputting the optimized multimodal interaction features into the emotion convolution block, wherein the emotion convolution block is composed of multiple convolution layers and nonlinear activation units, extracting emotion features from visual data, wherein the convolution layers use different convolution kernel sizes to capture emotion information of different scales, and generate a preliminary emotion feature map;

[0087] S32, performing multi-layer convolution and pooling operations on the generated preliminary emotion feature map, wherein the convolution layer uses a nonlinear activation function for feature mapping, and the pooling layer is used to reduce the dimension of the preliminary emotion feature map, retain significant emotion features and reduce noise;

[0088] S33. Apply the emotional attention mechanism to perform weighted processing on the extracted emotional features:

[0089]

[0090] Among them, β ij represents the attention weight of position (i, j) in the preliminary emotion feature map, a represents the learnable parameter vector, and W e Represents the weight matrix of the attention mechanism, E ij represents the feature value of position (i, j) in the preliminary emotion feature map, E pq represents the feature value of position (p,q) in the preliminary emotion feature map, E kl represents the feature value of position (k, l) in the preliminary emotion feature map, E st Represents the feature value of position (s, t) in the preliminary emotion feature map, ∥E ij -E pq ∥ 2 represents the square of the Euclidean distance between position (i, j) and position (p, q) in the preliminary emotion feature map, δ represents the adjustment factor, and b e represents the bias term, exp represents the exponential function, ∥E kl -E st ∥ 2 represents the square of the Euclidean distance between the position (k, l) and the position (s, t) in the first emotion feature map, M represents the height of the preliminary emotion feature map, N represents the width of the preliminary emotion feature map, and ReLU represents the activation function;

[0091] S34, applying the calculated attention weight to the emotion feature map to generate a weighted emotion feature;

[0092] S35, jointly modeling the weighted emotion feature and the input speech feature, combining the emotion feature and the speech feature through a multimodal fusion function, and generating a semantic-emotion mapping feature;

[0093] S36. Output the generated semantic-sentiment mapping features as input to the cross-attention generator.

[0094] In this implementation, S4 specifically includes:

[0095] S41, inputting the semantic-emotion mapping features into a cross-attention generator to uniformly embed the different modal features, wherein the modal features have aligned representations in a shared space;

[0096] S42. In the shared space after embedding, multi-level cross-attention operations are performed on the input modal features, and the potential correlation between cross-modal features is identified and strengthened through the parallel attention mechanism, forming a synergistic relationship between multiple contexts.

[0097] S43, hierarchical screening of features after cross-attention operation, hierarchical evaluation of interactive features with different feature weight distribution strategies, retaining the most relevant interactive features and suppressing noise and redundant information;

[0098] S44, performing bidirectional enhancement processing on the filtered interaction features, and using a context association module to extract global context information and local context information from each modality;

[0099] S45, applying normalization and adaptive adjustment strategies to the interactive features after bidirectional enhancement, dynamically adjusting the balance of interactive feature expression, and achieving adaptive distribution in the shared space;

[0100] S46. Output multimodal fusion features with context-awareness.

[0101] In this implementation manner, S5 specifically includes:

[0102] S51, inputting the multimodal fusion features with context perception capability into a multimodal fusion-decomposition bidirectional network, wherein the multimodal fusion-decomposition bidirectional network includes a fusion path and a decomposition path, wherein the fusion path is used to perform deep fusion processing between multimodal data, and the decomposition path is used to separate and optimize the fused features;

[0103] S52. In the fusion path, a multi-head feature fusion mechanism is applied to process the input features of different modalities in layers. Each head aggregates multimodal features through different weight matrices and activation functions to capture the complex associations between the modalities.

[0104] S53, at the final stage of the fusion path, a hierarchical feature synthesis module is introduced to perform weighted summation and nonlinear transformation on the features of each modality, wherein the hierarchical feature synthesis module adaptively combines features according to weight adjustment of each modality and context information to generate a deeply fused multimodal feature representation;

[0105] S54, passing the deeply fused multimodal feature representation to a decomposition path, wherein the decomposition path applies an adaptive decomposition module to decompose the deeply fused multimodal feature representation into a plurality of sub-feature sets with independent information through matrix decomposition and feature decomposition strategies;

[0106] S55, further processing the features using a context-dependent decoupling unit in the decomposition path, dynamically adjusting the weights and relationships of each sub-feature set, and optimizing feature expression, wherein the decoupling unit maintains a balance of information flow when decomposing the features;

[0107] S56. Outputting decoupled and optimized multimodal features, wherein the multimodal features have independence and recognition capabilities after decomposition and optimization.

[0108] In this implementation manner, S6 specifically includes:

[0109] S61, inputting the decoupled optimized multimodal features into a holographic-speech symbiosis optimization module, wherein the holographic-speech symbiosis optimization module is composed of a transformer network and a bidirectional long short-term memory network, and is used to realize the combination of global and time series processing of multimodal features;

[0110] S62. In the transformer network, the multi-head self-attention mechanism is used to process the decoupled and optimized multimodal features of the input, identify the global dependencies at different positions in the feature sequence, establish the relationship between features from different angles through the multi-head self-attention mechanism, capture rich contextual information and generate global feature representation;

[0111] S63, passing the global feature representation processed by the transformer network to a bidirectional long short-term memory network to further extract time series features, wherein the bidirectional long short-term memory network simultaneously captures historical and future information of the input features through forward and backward paths to generate bidirectional time series features;

[0112] S64, combining the global feature representation output by the transformer and the bidirectional time series features output by the bidirectional long short-term memory network, and performing joint optimization through a feature weighted fusion module, wherein the feature weighted fusion module dynamically adjusts the weights of the global feature representation and the bidirectional time series features according to the contextual importance of the features;

[0113] S65. In the process of feature fusion, an adaptive feature balancing strategy is applied to keep the global feature representation and the bidirectional time series features consistent in different scenarios, wherein the adaptive feature balancing strategy automatically adjusts the weight ratio of the features by monitoring their performance in real time;

[0114] S66. Output the fused and optimized multimodal real-time synchronous feature data, which is used in the decision-making process of instant rendering of the speech recognition module and the holographic display system, and finally generates real-time synchronous speech recognition and holographic display data.

[0115] Embodiment 1:

[0116] In order to verify the feasibility of the present invention in implementation, the present invention is applied to a certain intelligent conference system. As a virtual assistant participating in human-computer interaction, the holographic display digital human needs to have fast and accurate voice recognition and emotion perception capabilities to improve the accuracy of meeting records and interactivity with participants. When faced with multimodal data in meetings, the existing technology often leads to high recognition error rates and slow response speeds due to the inability to fully capture and integrate the complex relationship between voice, vision, and ambient sound. It performs poorly, especially in scenarios where multiple people are talking and emotions fluctuate greatly.

[0117] In this embodiment, a holographic display digital human voice recognition enhancement method based on multimodal interactive learning is applied to solve the above problems. A variety of sensors are arranged in the conference room, including a high-precision microphone array, a camera, and an ambient sound sensor, which are used to collect voice, facial expressions, body postures, and ambient sound data. The system inputs the collected multimodal data into the data acquisition and preprocessing module for standardization and noise reduction to generate a multimodal feature matrix. The feature matrix is ​​used as input, and the interaction graph is constructed and updated through the adaptive multimodal interaction kernel and the dynamic graph convolution network, so that the system can dynamically identify the relationship between different modalities.

[0118] In a meeting, when a participant speaks, the holographic digital human not only recognizes the speech content, but also extracts the person's facial expression and posture information through the emotion convolution block, and combines it with the speech features to generate semantic-emotion mapping features. Subsequently, the cross-attention generator cross-maps and enhances the semantic-emotion mapping features with other modal features such as ambient sound in a shared space, and outputs context-aware multimodal fusion features. This enables the system to maintain high-accuracy speech recognition and smooth holographic display in situations where multiple people are talking, emotions change, and background noise is complex.

[0119] In order to verify the effectiveness of the present invention, several simulated meeting experiments were conducted. The experiments were conducted in different situations, including normal meetings, heated discussions, scenes with increased background noise, and emotional excitement. By comparing the traditional single-modal processing method and the multimodal interactive learning method of the present invention, the accuracy of speech recognition, response time, and naturalness of holographic display were recorded. The results show that in the scene of intense discussion, the speech recognition accuracy of the present invention reached 93.7%, while the traditional method was only 78.5%. In the case of increased background noise, the present invention maintains the accuracy at 91.4% through adaptive multimodal fusion and attention adjustment, while the traditional method drops to 74.2%. In addition, in emotionally excited scenes, the system of the present invention can quickly adjust the semantic-emotional mapping features to keep the response speed of the holographic display within 1.2 seconds, while the traditional method delays up to 2.7 seconds.

[0120] Table 1. Experimental data table of holographic display digital human speech recognition performance comparison

[0121]

[0122] By analyzing Table 1, we can see that the present invention outperforms traditional methods in a variety of complex conference scenarios. In a normal conference scenario, the speech recognition accuracy of the present invention reaches 95.3%, which is about 9.7 percentage points higher than the 85.6% of the traditional method, and the response time is 0.8 seconds, which is significantly better than the 1.3 seconds of the traditional method. This shows that the present invention has higher recognition accuracy and response speed in a standard interactive environment.

[0123] In the intense discussion scenario, the speech recognition accuracy of the present invention is 93.7%, while the traditional method is 78.5%, and the gap is further widened to 15.2 percentage points. The response time in this scenario is 1.0 second, which is significantly lower than the 2.1 seconds of the traditional method, reflecting the efficiency of the present invention in processing multiple voice inputs and complex interactions.

[0124] For the scenario with increased background noise, the accuracy of the present invention is 91.4%, which remains at a high level, while the accuracy of the traditional method drops to 74.2%. In this scenario, the response time of the present invention is 1.1 seconds, while the traditional method is 2.5 seconds, indicating that the present invention is more stable in dealing with external interference.

[0125] In emotionally agitated scenes, the speech recognition accuracy of the present invention reaches 92.1%, far exceeding the 77.3% of the traditional method. The present invention can still maintain a short response time (1.2 seconds) in emotionally changeable situations, while the traditional method takes 2.7 seconds, which shows that the present invention has higher real-time performance and accuracy when dealing with complex emotional expressions.

[0126] Finally, in the multi-person conversation scenario, the accuracy of the present invention is 90.5%, 16.7 percentage points higher than the 73.8% of the traditional method. Its response time is 1.0 second, while the traditional method is 2.4 seconds, which further verifies the advantages of the present invention in multimodal data fusion and time series processing. These data comprehensively illustrate that the present invention has higher speech recognition accuracy and faster response speed in various complex scenarios, especially in dealing with background noise, multi-person communication and emotional changes.

[0127] It can be seen that the speech recognition accuracy and response speed of the present invention in different complex conference scenarios are significantly better than traditional methods, especially when dealing with challenging scenarios such as background noise, multi-person conversations, and emotional excitement, it shows higher robustness and real-time performance. Experimental data show that the present invention effectively improves the speech recognition and interaction capabilities of holographic display digital people through multimodal interactive learning and innovative module design, has higher adaptability and stability, and can meet complex and changing application needs.

[0128] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A holographic display digital human speech recognition enhancement method based on multimodal interactive learning, characterized in that: The steps include: S1. Collecting the user's voice data, visual data and environmental sound data through a multi-sensor array, standardizing and pre-noising the collected data, and generating a multimodal feature matrix, wherein the multimodal feature matrix includes voice features, visual features and environmental features; S2. Input the multimodal feature matrix into the adaptive multimodal interaction kernel, use the dynamic graph convolutional network to build and dynamically update the interaction graph, adjust the weight of the interaction graph through the graph attention mechanism, and generate optimized multimodal interaction features; S3, input the optimized multimodal interaction features into the emotion convolution block, extract the visual emotion features and combine them with the speech features to generate semantic-emotion mapping features; S4, input the semantic-emotion mapping features into the cross-attention generator, perform feature cross-mapping and enhancement in the shared space, and output multimodal fusion features with context-awareness; S5, inputting the multimodal fusion features with context perception into the multimodal fusion-decomposition bidirectional network, performing deep fusion processing and generating decomposition path output, thereby generating multimodal features after decoupling optimization; S6. Input the decoupled optimized multimodal features into the holographic-speech symbiosis optimization module, process them using the hybrid architecture of the transformer and the bidirectional long short-term memory network, and generate real-time synchronous speech recognition and holographic display data; S7. Based on real-time synchronous data feedback information, self-supervised learning and contrastive learning strategies are applied to optimize parameters, and adaptive adjustment of parameters is achieved through the federated learning mechanism.

2. The method for enhancing speech recognition of holographic digital human based on multimodal interactive learning according to claim 1 is characterized in that: The S1 specifically includes: performing spectral analysis on the speech data and extracting Mel-frequency cepstral coefficients to generate speech features; extracting visual features from the visual data using a dual-branch convolutional attention network; performing background processing on the ambient sound data and extracting environmental features.

3. The method for enhancing speech recognition of holographic digital human based on multimodal interactive learning according to claim 1 is characterized in that: The S2 specifically includes: S21, inputting the multimodal feature matrix after standardization and noise reduction preprocessing into the adaptive multimodal interaction kernel; S22. Use the dynamic graph convolutional network to construct the initial interaction graph G = (V, E), where V represents a set of nodes, each node corresponds to features of different modalities, and E represents a set of edges, and the weights of the edges are calculated based on the feature similarity and time dependency between nodes; S23. In the initial interaction graph G, node features are updated through a dynamic graph convolutional network: in, represents the feature vector of node i in the l+1th layer, σ represents the activation function, N(i) represents the neighborhood set of node i, α ij represents the attention weight between node i and node j, W (l) represents the weight matrix of the lth layer, γ represents the neighborhood feature balance factor, represents the second-order neighborhood aggregation function, represents the feature vector of node p in layer l, represents the feature vector of node j in the lth layer, β represents the global feature balance coefficient, represents the pooling operation of node j, b (l) Represents the bias vector of the lth layer; S24. Apply the graph attention mechanism to adjust the weights of the nodes and edges of the interaction graph; S25, through the interaction graph adjusted by the graph attention mechanism, node features are redistributed, and the weights of different modal features are balanced by multi-level feature aggregation and adaptive normalization mechanism to generate multi-modal fusion features; S26. Output the multimodal interaction features after multi-layer iterative optimization of the dynamic graph convolutional network and graph attention mechanism.

4. The method for enhancing speech recognition of holographic display digital human based on multimodal interactive learning according to claim 1, characterized in that: The S3 specifically includes: S31, inputting the optimized multimodal interaction features into the emotion convolution block, wherein the emotion convolution block is composed of multiple convolution layers and nonlinear activation units, extracting emotion features from visual data, wherein the convolution layers use different convolution kernel sizes to capture emotion information of different scales, and generate a preliminary emotion feature map; S32, performing multi-layer convolution and pooling operations on the generated preliminary emotion feature map, wherein the convolution layer uses a nonlinear activation function for feature mapping, and the pooling layer is used to reduce the dimension of the preliminary emotion feature map, retain significant emotion features and reduce noise; S33. Apply the emotional attention mechanism to perform weighted processing on the extracted emotional features: Among them, β ij represents the attention weight of position (i, j) in the preliminary emotion feature map, a represents the learnable parameter vector, and W e Represents the weight matrix of the attention mechanism, E ij represents the feature value of position (i, j) in the preliminary emotion feature map, E pq represents the feature value of position (p,q) in the preliminary emotion feature map, E kl represents the feature value of position (k, l) in the preliminary emotion feature map, E st Represents the feature value of position (s, t) in the preliminary emotion feature map, ∥E ij -E pq ∥ 2 represents the square of the Euclidean distance between position (i, j) and position (p, q) in the preliminary emotion feature map, δ represents the adjustment factor, and b e represents the bias term, exp represents the exponential function, ∥E kl -E st ∥ 2 represents the square of the Euclidean distance between the position (k, l) and the position (s, t) in the first emotion feature map, M represents the height of the preliminary emotion feature map, N represents the width of the preliminary emotion feature map, and ReLU represents the activation function; S34, applying the calculated attention weight to the emotion feature map to generate a weighted emotion feature; S35, jointly modeling the weighted emotion feature and the input speech feature, combining the emotion feature and the speech feature through a multimodal fusion function, and generating a semantic-emotion mapping feature; S36. Output the generated semantic-sentiment mapping features as input to the cross-attention generator.

5. The method for enhancing speech recognition of holographic display digital human based on multimodal interactive learning according to claim 1, characterized in that: The S4 specifically includes: S41, inputting the semantic-emotion mapping features into a cross-attention generator to uniformly embed the different modal features, wherein the modal features have aligned representations in a shared space; S42. In the shared space after embedding, multi-level cross-attention operations are performed on the input modal features, and the potential correlation between cross-modal features is identified and strengthened through the parallel attention mechanism, forming a synergistic relationship between multiple contexts. S43, hierarchical screening of features after cross-attention operation, hierarchical evaluation of interactive features with different feature weight distribution strategies, retaining the most relevant interactive features and suppressing noise and redundant information; S44, performing bidirectional enhancement processing on the filtered interaction features, and using a context association module to extract global context information and local context information from each modality; S45, applying normalization and adaptive adjustment strategies to the interactive features after bidirectional enhancement, dynamically adjusting the balance of interactive feature expression, and achieving adaptive distribution in the shared space; S46. Output multimodal fusion features with context-awareness.

6. The method for enhancing speech recognition of holographic display digital human based on multimodal interactive learning according to claim 1, characterized in that: The S5 specifically includes: S51, inputting the multimodal fusion features with context perception capability into a multimodal fusion-decomposition bidirectional network, wherein the multimodal fusion-decomposition bidirectional network includes a fusion path and a decomposition path, wherein the fusion path is used to perform deep fusion processing between multimodal data, and the decomposition path is used to separate and optimize the fused features; S52. In the fusion path, a multi-head feature fusion mechanism is applied to process the input features of different modalities in layers. Each head aggregates multimodal features through different weight matrices and activation functions to capture the complex associations between the modalities. S53, at the final stage of the fusion path, a hierarchical feature synthesis module is introduced to perform weighted summation and nonlinear transformation on the features of each modality, wherein the hierarchical feature synthesis module adaptively combines features according to weight adjustment of each modality and context information to generate a deeply fused multimodal feature representation; S54, passing the deeply fused multimodal feature representation to a decomposition path, wherein the decomposition path applies an adaptive decomposition module to decompose the deeply fused multimodal feature representation into a plurality of sub-feature sets with independent information through matrix decomposition and feature decomposition strategies; S55, further processing the features using a context-dependent decoupling unit in the decomposition path, dynamically adjusting the weights and relationships of each sub-feature set, and optimizing feature expression, wherein the decoupling unit maintains a balance of information flow when decomposing the features; S56. Outputting decoupled and optimized multimodal features, wherein the multimodal features have independence and recognition capabilities after decomposition and optimization.

7. The method for enhancing speech recognition of holographic display digital human based on multimodal interactive learning according to claim 1, characterized in that: The S6 specifically includes: S61, inputting the decoupled optimized multimodal features into a holographic-speech symbiosis optimization module, wherein the holographic-speech symbiosis optimization module is composed of a transformer network and a bidirectional long short-term memory network, and is used to realize the combination of global and time series processing of multimodal features; S62. In the transformer network, the multi-head self-attention mechanism is used to process the decoupled and optimized multimodal features of the input, identify the global dependencies at different positions in the feature sequence, establish the relationship between features from different angles through the multi-head self-attention mechanism, capture rich contextual information and generate global feature representation; S63, passing the global feature representation processed by the transformer network to a bidirectional long short-term memory network to further extract time series features, wherein the bidirectional long short-term memory network simultaneously captures historical and future information of the input features through forward and backward paths to generate bidirectional time series features; S64, combining the global feature representation output by the transformer and the bidirectional time series features output by the bidirectional long short-term memory network, and performing joint optimization through a feature weighted fusion module, wherein the feature weighted fusion module dynamically adjusts the weights of the global feature representation and the bidirectional time series features according to the contextual importance of the features; S65. In the process of feature fusion, an adaptive feature balancing strategy is applied to keep the global feature representation and the bidirectional time series features consistent in different scenarios, wherein the adaptive feature balancing strategy automatically adjusts the weight ratio of the features by monitoring their performance in real time; S66. Output the fused and optimized multimodal real-time synchronous feature data, which is used in the decision-making process of instant rendering of the speech recognition module and the holographic display system, and finally generates real-time synchronous speech recognition and holographic display data.

Images