Virtual interaction response system and method based on voice semantic recognition

Abstract

This application provides a virtual interactive response system and method based on speech and semantic recognition, relating to the field of speech and semantic recognition technology. Responding to a acquired real-time speech stream, it extracts acoustic feature vectors, performs real-time speech recognition, and generates a first text sequence; it extracts word embedding features, determines nonlinear coupling weights, and generates a fused semantic vector; it performs sequence editing operations on the first text sequence based on the fused semantic vector to generate a second text sequence; it performs domain-knowledge-enhanced semantic parsing on the second text sequence to generate intent encoding and entity encoding; based on the intent encoding, it retrieves structured response data from a preset business knowledge graph; and it drives the virtual interactive object to perform corresponding action feedback based on the structured response data. This application can dynamically locate spoken language correction boundaries using acoustic paralinguistic features, and improves the robustness and accuracy of virtual interactive responses through closed-loop linkage of cross-modal information and semantic reconstruction.

Description

Technical Field

[0001] This application relates to the field of speech and semantic recognition technology, and more specifically, to a virtual interactive response system and method based on speech and semantic recognition. Background Technology

[0002] In the field of virtual interactive response technology, interaction methods based on speech and semantic recognition typically employ a speech recognition module to convert user speech into a text sequence, then use a natural language understanding model to perform semantic parsing on the text sequence to obtain the user's intent. Finally, based on the intent, a preset response library or knowledge base is retrieved to generate feedback content and drive the virtual object to output. In existing technologies, speech recognition and semantic understanding modules are mostly processed serially in a cascaded manner. Some improved solutions introduce a speech emotion recognition module, which independently analyzes acoustic features to obtain user emotion tags and overlays these emotion tags as auxiliary information into the final response strategy to adjust the tone or expression of the virtual object's feedback.

[0003] Existing speech-semantic recognition-based interaction methods typically cascade speech recognition and semantic parsing units sequentially. After speech recognition, only the text sequence is retained, discarding acoustic features. This results in the semantic parsing stage failing to perceive paralinguistic information such as prosodic variations and vocal physiological states contained within the speech signal. When users exhibit hesitation, repetition, or self-correction in their spoken expression, pure text processing methods struggle to accurately identify the user's corrective intent and true semantic boundaries. They may mistakenly treat redundant content that the user has actively discarded as valid instructions, leading to erroneous responses from virtual interactive objects. Therefore, how to dynamically locate spoken correction boundaries using acoustic paralinguistic features and improve the robustness and accuracy of virtual interactive responses through closed-loop linkage of cross-modal information and semantic reconstruction has become a challenge for the industry. Summary of the Invention

[0004] This application provides a virtual interactive response system and method based on speech semantic recognition, which can dynamically locate spoken language correction boundaries by utilizing acoustic paralinguistic features, and improve the robustness and accuracy of virtual interactive response through closed-loop linkage of cross-modal information and semantic reconstruction.

[0005] In a first aspect, this application provides a virtual interactive response method based on speech semantic recognition, the response method comprising the following steps:

[0006] In response to the acquired real-time speech stream, acoustic feature vectors are extracted and real-time speech recognition is performed to generate a first text sequence with time-series markers;

[0007] Extract word embedding features from the first text sequence, determine the nonlinear coupling weights between the acoustic feature vector and the word embedding features based on cross-modal attention, and generate a fused semantic vector;

[0008] Based on preset semantic reconstruction rules, sequence editing is performed on the first text sequence to generate a second text sequence that removes non-fluent redundancy and is temporally coherent;

[0009] The second text sequence is subjected to domain-knowledge-enhanced semantic parsing to generate intent encoding and entity encoding, and the business knowledge graph is retrieved to generate structured response data, driving the virtual object to perform action feedback.

[0010] In this embodiment, extracting acoustic feature vectors in response to the acquired real-time speech stream specifically includes:

[0011] After pre-emphasis and frame-segmentation windowing processing of the real-time audio stream, multi-dimensional basic acoustic parameters are extracted.

[0012] Based on the aforementioned multidimensional basic acoustic parameters, inverse filtering of the glottal wave is performed to separate the glottal excitation source signal and the vocal tract impulse response signal.

[0013] The glottal closure quotient parameter of the glottal excitation source signal is extracted, and the glottal closure quotient parameter is used as a secondary language feature channel to characterize the physiological state of vocalization.

[0014] The multidimensional basic acoustic parameters and the secondary language feature channels are concatenated along the time dimension to generate an acoustic feature vector.

[0015] In this embodiment, performing real-time speech recognition and generating a first text sequence with time-series markers specifically includes:

[0016] The real-time speech stream is input into the encoder of the streaming end-to-end speech recognition model to extract the acoustic hidden layer representation sequence.

[0017] The acoustic hidden layer representation sequence is input into the connected temporal classification decoder to predict the posterior probability distribution of phonemes frame by frame, and a recognition token sequence carrying a timestamp is generated based on the dynamic programming bundle search algorithm.

[0018] A first text sequence with time-series markers is generated based on the identification token sequence.

[0019] In this embodiment, generating a first text sequence with a time-series marker based on the identification token sequence specifically includes:

[0020] Confidence assessment is performed on each token in the identified token sequence;

[0021] If the confidence level of the target token is lower than the preset semantic validity threshold, an empty token is generated to replace the target token, and the original timestamp interval marker corresponding to the target token is retained;

[0022] Based on the identified token sequence and the inserted empty token, a first text sequence with a time sequence marker is generated.

[0023] In this embodiment, extracting word embedding features from the first text sequence and determining the nonlinear coupling weights between the acoustic feature vector and the word embedding features based on cross-modal attention to generate a fused semantic vector specifically includes:

[0024] The acoustic feature vector is divided into an acoustic feature frame sequence along the time axis, and the word embedding features of the first text sequence are divided into a text token sequence.

[0025] Construct a temporal offset alignment matrix to establish a mapping relationship between each acoustic feature frame in the acoustic feature frame sequence and the text token sequence based on the temporal marker;

[0026] The acoustic feature frame sequence and the text token sequence are input into a multi-head cross-modal attention network to determine the text-dominated attention distribution with the text modality as the query vector and the acoustic modality as the key vector, and the acoustic-dominated attention distribution with the acoustic modality as the query vector and the text modality as the key vector, respectively.

[0027] The text-dominant attention distribution and the acoustic-dominant attention distribution are weighted and fused through a gated adaptive fusion layer to generate a fused semantic vector.

[0028] In this embodiment, according to preset semantic reconstruction rules, performing sequence editing on the first text sequence to generate a second text sequence that removes non-fluent redundancy and is temporally coherent specifically includes:

[0029] The fused semantic vector is concatenated with the position encoding vector of each token in the first text sequence to generate a token-level joint representation sequence;

[0030] The token-level joint representation sequence is input into an edit operation classifier based on a pointer generation network to predict the edit operation label token by token. The edit operation label includes: retain label, delete label, and replace label.

[0031] In response to the prediction that the edit operation tag of the target token is a delete tag and the target token belongs to a preset set of spoken language correction trigger words, the target token and its associated context correction fragments are marked as discarded semantic fragments and removed.

[0032] In response to the target token's edit operation tag being predicted as a replacement tag, the target token is masked based on contextual semantics to generate a replacement token;

[0033] The second text sequence is generated by arranging the retention token corresponding to the retention tag, the replacement token generated by the replacement tag, and the remaining tokens after removing the discarded semantic fragments in chronological order.

[0034] In this embodiment, performing domain-knowledge-enhanced semantic parsing on the second text sequence to generate intent encoding and entity encoding specifically includes:

[0035] Perform semantic parsing on the second text sequence to extract an explicit entity mention list;

[0036] Get a snapshot of the historical dialogue state of the current interaction session, and extract the implicit context entities and their corresponding timeliness decay weights;

[0037] The explicit entity mention list and the implicit context entity weighted by the timeliness decay weight are linked together to the corresponding entity node in the preset business knowledge graph. The adjacency relationship is extracted with the corresponding entity node as the center according to the preset association hop number to construct a dynamic knowledge subgraph.

[0038] The dynamic knowledge subgraph is graph embedded and encoded to generate a knowledge enhancement vector. The knowledge enhancement vector is then gated and fused with the semantic encoding vector of the second text sequence to generate intent encoding and entity encoding.

[0039] Secondly, this application provides a virtual interactive response system based on speech semantic recognition for executing a virtual interactive response method based on speech semantic recognition, the virtual interactive response system based on speech semantic recognition comprising:

[0040] The first text generation module, in response to the acquired real-time speech stream, extracts acoustic feature vectors and performs real-time speech recognition to generate a first text sequence with time-series markers.

[0041] The semantic vector generation module extracts word embedding features from the first text sequence, determines the nonlinear coupling weights between the acoustic feature vector and the word embedding features based on cross-modal attention, and generates a fused semantic vector.

[0042] The second text generation module performs sequence editing on the first text sequence according to preset semantic reconstruction rules to generate a second text sequence that removes non-fluent redundancy and is temporally coherent.

[0043] The structured response module performs domain-knowledge-enhanced semantic parsing on the second text sequence to generate intent and entity codes, and retrieves business knowledge graphs to generate structured response data, driving virtual objects to perform action feedback.

[0044] Thirdly, this application provides a computer device, the computer device including a memory and a processor, the memory for storing a computer program, and the processor for calling and running the computer program from the memory, so that the computer device executes the above-described virtual interactive response method based on speech semantic recognition.

[0045] Fourthly, this application provides a computer-readable storage medium storing instructions or code that, when executed on a computer, cause the computer to implement the aforementioned virtual interactive response method based on speech semantic recognition.

[0046] The technical solutions provided by the embodiments disclosed in this application have the following beneficial effects:

[0047] In response to the acquired real-time speech stream, acoustic feature vectors are extracted and real-time speech recognition is performed to generate a first text sequence with temporal tags. Word embedding features of the first text sequence are extracted, and the nonlinear coupling weights between the acoustic feature vectors and the word embedding features are determined based on cross-modal attention to generate a fused semantic vector. According to preset semantic reconstruction rules, sequence editing is performed on the first text sequence to generate a second text sequence that removes non-fluent redundancy and is temporally coherent. The second text sequence is subjected to domain knowledge-enhanced semantic parsing to generate intent encoding and entity encoding, and business knowledge graphs are retrieved to generate structured response data to drive virtual objects to perform action feedback.

[0048] Therefore, this application firstly, by performing streaming speech recognition and generating a first text sequence with temporal tags while extracting acoustic feature vectors, provides a precise frame-level time alignment benchmark for subsequent cross-modal fusion. By preserving the start and end time information of each text unit in the original audio stream, it avoids the modal misalignment problem caused by the loss of time information in traditional asynchronous processing, laying a reliable data foundation for improving the semantic understanding accuracy in spoken language interaction scenarios. Secondly, by using a cross-modal attention mechanism to non-linearly couple acoustic feature vectors with text word embedding features, the fused semantic vector carries the explicit semantic content of the text sequence and also incorporates paralinguistic information such as prosodic features and vocal physiological states contained in the speech signal. This allows for dynamic compensation of the text modality by the acoustic modality when the user hesitates, makes corrections, or experiences emotional fluctuations, capturing the nuances that cannot be reflected at the text level. The system implicitly identifies user intent, thereby enhancing the depth of perception and robustness of semantic understanding of natural spoken language. Then, a semantic reconstruction model performs sequence editing on the first text sequence, utilizing cross-modal information carried in the fused semantic vectors to identify and remove non-fluent redundant content from spoken language while preserving the original temporal relationships of effective semantic segments, generating a second text sequence. This improves adaptability to real-world spoken language scenarios and the accuracy of semantic parsing. Finally, domain-knowledge-enhanced semantic parsing of the second text sequence, combined with business knowledge graph retrieval, generates structured response data and drives virtual objects to perform action feedback. Because the second text sequence has eliminated redundant interference from spoken language correction, the text content input to the semantic parsing stage is more accurate and concise, thus improving the accuracy of intent recognition and entity linking, and enhancing the robustness and accuracy of virtual interactive responses.

[0049] In summary, the technical solution adopted in this application can dynamically locate spoken language correction boundaries using acoustic paralinguistic features, and improve the robustness and accuracy of virtual interactive responses through closed-loop linkage of cross-modal information and semantic reconstruction. Attached Figure Description

[0050] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only for this embodiment of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0051] Figure 1 This is an exemplary flowchart of the virtual interactive response method based on speech semantic recognition provided in this application;

[0052] Figure 2 This is a schematic diagram of the network structure based on the cross-modal attention mechanism provided in this application;

[0053] Figure 3 This is a flowchart illustrating the sequence editing operations provided in this application;

[0054] Figure 4 This is a module structure diagram of a virtual interactive response system based on speech semantic recognition provided in this application;

[0055] Figure 5 This is a schematic diagram of the structure of a computer device that implements a virtual interactive response method based on speech semantic recognition according to the present application. Detailed Implementation

[0056] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0057] This application provides a virtual interactive response system and method based on speech semantic recognition. Its core is to respond to a captured real-time speech stream, extract acoustic feature vectors, perform real-time speech recognition, and generate a first text sequence with temporal markers. Next, it extracts word embedding features from the first text sequence, determines the non-linear coupling weights between the acoustic feature vectors and the word embedding features based on cross-modal attention, and generates a fused semantic vector. Following preset semantic reconstruction rules, it performs sequence editing on the first text sequence to generate a second text sequence that eliminates non-fluent redundancy and is temporally coherent. Finally, it performs domain-knowledge-enhanced semantic parsing on the second text sequence to generate intent encoding and entity encoding, and retrieves business knowledge graphs to generate structured response data, driving virtual objects to perform action feedback. This approach utilizes acoustic paralinguistic features to dynamically locate spoken language correction boundaries, and through the closed-loop linkage of cross-modal information and semantic reconstruction, improves the robustness and accuracy of virtual interactive responses.

[0058] Example 1: To better understand the above technical solution, the following will provide a detailed description of the technical solution in conjunction with the accompanying drawings and specific implementation methods. (Refer to...) Figure 1 As shown in the figure, this is an exemplary flowchart of a virtual interactive response method based on speech semantic recognition according to this embodiment of the present application. The response method includes the following steps:

[0059] In step S1, in response to the acquired real-time speech stream, acoustic feature vectors are extracted and real-time speech recognition is performed to generate a first text sequence with time-series markers.

[0060] In this embodiment, the extraction of acoustic feature vectors in response to the acquired real-time speech stream can be achieved through the following steps:

[0061] After pre-emphasis and frame-segmentation windowing processing of the real-time audio stream, multi-dimensional basic acoustic parameters are extracted.

[0062] Based on the aforementioned multidimensional basic acoustic parameters, inverse filtering of the glottal wave is performed to separate the glottal excitation source signal and the vocal tract impulse response signal.

[0063] The glottal closure quotient parameter of the glottal excitation source signal is extracted, and the glottal closure quotient parameter is used as a secondary language feature channel to characterize the physiological state of vocalization.

[0064] The multidimensional basic acoustic parameters and the secondary language feature channels are concatenated along the time dimension to generate an acoustic feature vector.

[0065] In practical implementation, firstly, after pre-emphasis and frame-by-frame windowing processing of the real-time speech stream, multi-dimensional fundamental acoustic parameters can be extracted. Specifically, the real-time speech stream is pre-emphasized to enhance high-frequency components, the processed signal is framed according to a preset frame length and frame shift, and a window function is applied to each frame. Multi-dimensional fundamental acoustic parameters, including Mel-frequency cepstral coefficients, fundamental envelope, short-time energy, and formant frequencies, are extracted from each windowed frame. Secondly, inverse glottal filtering can be performed based on the multi-dimensional fundamental acoustic parameters to separate the glottal excitation source signal from the vocal tract impulse response signal. Specifically, linear predictive coding technology is used to estimate the vocal tract impulse response, and an inverse filter is constructed to inversely filter the original speech, separating the glottal excitation source signal. Then, The glottal closure quotient parameter of the glottal excitation source signal is extracted and used as a secondary language feature channel to characterize the physiological state of vocalization. Specifically, the fundamental frequency of the glottal excitation source signal is estimated to determine the boundary of the glottal vibration period. Within each period, the opening and closing times of the glottis are identified by differentiation, and the ratio of the closing duration to the period duration is calculated to obtain the glottal closure quotient parameter. The glottal closure quotient parameter reflects the degree of vocal cord tension, and it exhibits an abnormally high characteristic when the user hesitates or is tense. Finally, the multidimensional basic acoustic parameters and the secondary language feature channel can be concatenated along the time dimension to generate an acoustic feature vector. That is, the multidimensional basic acoustic parameters and the glottal closure quotient parameter of each speech frame are concatenated in time sequence to form the final acoustic feature vector.

[0066] In this embodiment, real-time speech recognition and generation of a first text sequence with time-series markers can be achieved through the following steps:

[0067] The real-time speech stream is input into the encoder of the streaming end-to-end speech recognition model to extract the acoustic hidden layer representation sequence.

[0068] The acoustic hidden layer representation sequence is input into the connected temporal classification decoder to predict the posterior probability distribution of phonemes frame by frame, and a recognition token sequence carrying a timestamp is generated based on the dynamic programming bundle search algorithm.

[0069] A first text sequence with time-series markers is generated based on the identification token sequence.

[0070] In practical implementation, firstly, the real-time speech stream can be input into the encoder of a streaming end-to-end speech recognition model to extract the acoustic hidden layer representation sequence. That is, the real-time speech stream after front-end signal processing is input into the encoder network of a streaming end-to-end speech recognition model to extract the acoustic hidden layer representation sequence. Here, the streaming end-to-end speech recognition model refers to a neural network model architecture that can process audio streams incrementally and with low latency. The encoder can adopt a neural network structure based on causal convolution or unidirectional long short-term memory network. Causal convolution, by introducing a temporal mask into the convolution kernel, ensures that the output at the current moment depends only on the input at the current and past moments, and not on information from future moments, thus satisfying the causal constraints of streaming processing. Unidirectional Long Short-Term Memory (LSTM) networks, through their inherent unidirectional temporal dependency characteristics, also satisfy the requirements of streaming processing. Real-time speech streams are input to the encoder frame-by-frame or block-by-block in the form of audio frame sequences. The encoder performs nonlinear transformations and high-level semantic abstractions on the acoustic features of each input frame, outputting a sequence of acoustic hidden layer representation vectors corresponding to each input audio frame in the temporal dimension. Each vector in the acoustic hidden layer representation sequence is a compressed semantic representation of its corresponding audio frame, capturing the phoneme distribution, articulation state, and local contextual information of the speech signal at that moment. Then, the acoustic hidden layer representation sequence can be input to a connected temporal classification decoder to predict the posterior probability distribution of phonemes frame-by-frame, and based on... The dynamic programming bundle search algorithm generates a sequence of identification tokens carrying timestamps. That is, the connected temporal classification decoder consists of one or more fully connected layers and a Softmax activation layer. The fully connected layer maps the acoustic latent representation vector output by the encoder to a feature space with a dimension equal to the size of the phoneme vocabulary. The phoneme vocabulary includes all basic articulators in the target language and a blank symbol representing no speech output or phoneme boundary, such as the initial consonant, final vowel, and tone combination in Chinese. The Softmax activation layer performs a normalized exponential operation on the mapped feature vector and outputs the posterior probability value of the current audio frame belonging to each phoneme category and the blank symbol. For each frame of input acoustic latent representation, the connected temporal classification decoder outputs a probability distribution vector, representing the most likely corresponding articulator in that frame. Arranging the probability distribution vectors corresponding to all frames in temporal order yields the phoneme posterior probability distribution sequence.

[0071] In this embodiment, generating a first text sequence with a time-series marker based on the identification token sequence can be achieved through the following steps:

[0072] Confidence assessment is performed on each token in the identified token sequence;

[0073] If the confidence level of the target token is lower than the preset semantic validity threshold, an empty token is generated to replace the target token, and the original timestamp interval marker corresponding to the target token is retained;

[0074] Based on the identified token sequence and the inserted empty token, a first text sequence with a time sequence marker is generated.

[0075] In practical implementation, firstly, the confidence assessment can be calculated based on the posterior probability information generated during the dynamic programming bundle search decoding process. For each recognized token, the average phoneme posterior probability value of its corresponding time frame interval in the bundle search decoding path is obtained as the confidence score of the token. The purpose of the confidence assessment is to quantify the reliability of each recognized text token. Then, if the confidence score of the target token is lower than a preset semantic validity threshold, an empty token is generated to replace the target token, and the original timestamp interval marker corresponding to the target token is retained. That is, the confidence score of each token is compared with the preset semantic validity threshold. The semantic validity threshold is an empirical parameter preset based on experimental data analysis, used to distinguish the confidence boundary between reliable and unreliable recognition results. If the confidence score of the target token is greater than or equal to the semantic validity threshold, the recognition result of the target token is determined to be reliable and retained. The text characters of the token and its corresponding original timestamp interval marker remain unchanged. If the confidence score of the target token is lower than the threshold, the recognition result of the target token is determined to be reliable and retained. If the semantic validity threshold is exceeded, the recognition result of the target token is determined to be unreliable. In this case, an empty token is generated to replace the target token, while retaining the original timestamp interval marker corresponding to the target token. The empty token is a special placeholder symbol that occupies a character position in the text sequence but does not carry any semantic content information. The empty token completely inherits and retains the original timestamp interval marker corresponding to the target token it replaces, that is, its start timestamp and end timestamp are completely consistent with the replaced low-confidence token. Finally, based on the recognized token sequence and the inserted empty token, a first text sequence with time sequence markers can be generated. That is, the recognized token sequence is traversed. For each token position, if its confidence score meets the retention condition, the original text token is retained at that position; if its confidence score does not meet the retention condition, the original text token is replaced by an empty token at that position. All tokens are arranged in their original time sequence to form the final token sequence. At the same time, each token position is accompanied by its corresponding start and end timestamp interval markers, thereby generating a first text sequence with time sequence markers.

[0076] In step S2, word embedding features of the first text sequence are extracted, and nonlinear coupling weights between the acoustic feature vector and the word embedding features are determined based on cross-modal attention to generate a fused semantic vector.

[0077] Preferably, in this embodiment, extracting word embedding features from the first text sequence and determining the nonlinear coupling weights between the acoustic feature vector and the word embedding features based on cross-modal attention to generate a fused semantic vector can be achieved through the following steps:

[0078] In step S21, the acoustic feature vector is divided into an acoustic feature frame sequence along the time axis, and the word embedding features of the first text sequence are divided into a text token sequence.

[0079] In step S22, a temporal offset alignment matrix is ​​constructed to establish a mapping relationship between each acoustic feature frame in the acoustic feature frame sequence and the text token sequence based on the temporal marker;

[0080] In step S23, the acoustic feature frame sequence and the text token sequence are input into a multi-head cross-modal attention network to determine the text-dominated attention distribution with the text modality as the query vector and the acoustic modality as the key vector, and the acoustic-dominated attention distribution with the acoustic modality as the query vector and the text modality as the key vector, respectively.

[0081] In step S24, the text-dominant attention distribution and the acoustic-dominant attention distribution are weighted and fused through a gated adaptive fusion layer to generate a fused semantic vector.

[0082] In specific implementation, firstly, the acoustic feature vector is divided into an acoustic feature frame sequence along the time axis, and the word embedding features of the first text sequence are divided into a text token sequence. That is, the acoustic feature vectors corresponding to all speech frames are arranged in their original time order to form the acoustic feature frame sequence. For example, if a speech segment is divided into T audio frames, and D-dimensional acoustic features are extracted from each frame, then the acoustic feature frame sequence can be represented as a feature matrix of dimension T×D, where the elements represent the D-dimensional acoustic feature vector of the t-th audio frame. Word embedding processing is then performed on the first text sequence to generate a text token sequence. This token sequence, or word embedding processing, refers to mapping discrete text tokens to a continuous low-dimensional vector space. Pre-trained word embedding models, such as Word2Vec, GloVe, or BERT's static vocabulary embedding layer, can be used. If the first text sequence contains M tokens, each mapped to an E-dimensional word embedding vector, then the text token sequence can be represented as a feature matrix of dimension M×E. The position encoding vector of each token is element-wise added to its word embedding vector to form a text token sequence incorporating positional information. Secondly, a temporal offset alignment matrix can be constructed to integrate acoustic feature frames. Each acoustic feature frame in the sequence is mapped to the text token sequence based on a time-series marker. That is, the time-series marker is the original audio start and end timestamp information annotated for each text token in the aforementioned speech recognition step. The construction process of the time-series offset alignment matrix is ​​as follows: Create a two-dimensional matrix of dimension T×M, where T is the total number of audio frames and M is the total number of text tokens. For each element in the matrix, its row and column indices correspond to an audio frame and a text token, respectively. Traverse each text token in the text token sequence, obtain the start and end timestamps of the token, and align the acoustic feature frames... All audio frames whose frame timestamps fall within the interval formed by the start and end timestamps are marked as having an alignment relationship with the text token. For matrix element positions with an alignment relationship, a value of 1 is assigned, and for positions without an alignment relationship, a value of 0 is assigned. For empty tokens generated in the aforementioned steps, although they do not carry semantic text content, their corresponding original timestamp interval markings are completely preserved. When constructing the temporal offset alignment matrix, empty tokens are treated as ordinary tokens participating in the alignment mapping, and their corresponding acoustic feature frames will be normally marked as having an alignment relationship with the empty token.

[0083] In practical implementation, the acoustic feature frame sequence and text token sequence can be input into a multi-head cross-modal attention network to determine the text-dominated attention distribution with the text modality as the query vector and the acoustic modality as the key-value vector, and the acoustic-dominated attention distribution with the acoustic modality as the query vector and the text modality as the key-value vector. That is, the acoustic feature frame sequence and the text token sequence are mapped to the same-dimensional hidden layer representation space through linear projection to obtain the acoustic hidden layer representation sequence and the text hidden layer representation sequence. The hidden layer dimension is evenly divided into multiple subspaces according to the preset number of attention heads. Each attention head is configured with an independent query, key, and value projection matrix. Each head performs attention calculation in parallel. With the text modality as the query and the acoustic modality as the key-value, for each attention head, the corresponding projection matrix is ​​used to generate a text query matrix, an acoustic key matrix, and an acoustic value matrix. The dot product of the text query matrix and the transpose of the acoustic key matrix is ​​calculated, scaled, and multiplied element-wise with the temporal offset alignment matrix to constrain the attention calculation within the temporal alignment range. The constrained score matrix is ​​row-normalized to obtain the text-dominant attention distribution matrix, which is then multiplied by the acoustic value matrix to obtain the text-dominant attention representation for that head. This branch focuses each text token on the acoustic features within its pronunciation period, aiding semantic disambiguation. Next, using the acoustic modality as the query and the text modality as the key, a symmetric computation process is employed: generating the acoustic query matrix, text key matrix, and text value matrix; calculating the dot product score and scaling it before multiplying it by the transpose of the temporal offset alignment matrix; normalizing this result to obtain the acoustic-dominant attention distribution matrix, which is then multiplied by the text value matrix to obtain the acoustic-dominant attention representation. This branch focuses the attention of each frame's acoustic features on the text tokens of its corresponding time period. When the audio frame is in a period of user hesitation or self-correction, the attention weights are tilted towards tokens such as correction trigger words, thereby conveying the implicit correction intention to the subsequent fusion representation. Then, the two branches concatenate the outputs of each attention head along the feature dimension and linearly fuse them through the output projection matrix to obtain the final output matrix of each branch. Among them, the output matrix of the acoustic dominant branch is aggregated to the time interval corresponding to each text token according to the temporal offset alignment matrix, so that its dimension is aligned with the number of text tokens. Finally, the acoustic feature frame sequence is statistically pooled in the time dimension to extract global acoustic statistics such as fundamental frequency envelope variance, short-time energy peak-to-average ratio, glottal closure quotient mean, and abnormal frame proportion. These are then used by a state coding network to generate a global acoustic state vector. After expanding the vector to the same dimension as the attention output, it is concatenated with the text-dominated output matrix and the acoustic-dominated output matrix, and input into the gating network to generate a token-by-token gating coefficient. The outputs of the two branches are then weighted and fused based on this gating coefficient: the larger the gating coefficient, the more the fusion result is biased towards text semantics; when a correction precursor feature such as throat tightening or fundamental frequency drop is detected, the gating coefficient is automatically reduced, allowing the fused vector to incorporate more acoustic correction information, and finally generating a fused semantic vector corresponding to the text token sequence.

[0084] like Figure 2 The diagram illustrates the network structure of the cross-modal attention mechanism, showcasing the complete data flow from input acoustic feature frame sequences and text token sequences to the generation of a fused semantic vector: After establishing a frame-level mapping relationship between the acoustic feature frame sequences and the text token sequences through a temporal offset alignment matrix, they are jointly input into a multi-head cross-modal attention layer; the multi-head cross-modal attention layer contains two parallel branches: a text-dominant attention branch and an acoustic-dominant attention branch, which respectively calculate the attention distribution for text modality queries against acoustic modality and the attention distribution for acoustic modality queries against text modality; simultaneously, the acoustic feature frame sequences, after having their global acoustic state extracted by a statistical pooling module, are input into a gating network to generate gating coefficients; the outputs of the text-dominant attention branch and the acoustic-dominant attention branch are jointly input into the weighted fusion node in the gating fusion layer, where they are adaptively weighted and combined under the adjustment of the gating coefficients, ultimately generating a fused semantic vector.

[0085] It should be noted that by using a cross-modal attention mechanism to non-linearly couple acoustic feature vectors with text word embedding features, the fused semantic vector carries the explicit semantic content of the text sequence and also incorporates paralinguistic information such as prosodic features and vocal physiological states contained in the speech signal. When users hesitate, make corrections, or experience emotional fluctuations, the acoustic modality can dynamically compensate for the text modality to capture the user's implicit intentions that cannot be reflected at the text level, thereby enhancing the perception depth of natural spoken language and the robustness of semantic understanding.

[0086] In step S3, according to the preset semantic reconstruction rules, the first text sequence is edited to generate a second text sequence that removes non-fluent redundancy and is temporally coherent.

[0087] In this embodiment, the first text sequence is edited according to preset semantic reconstruction rules to generate a second text sequence that removes non-fluent redundancy and is temporally coherent. This can be achieved through the following steps:

[0088] The fused semantic vector is concatenated with the position encoding vector of each token in the first text sequence to generate a token-level joint representation sequence;

[0089] The token-level joint representation sequence is input into an edit operation classifier based on a pointer generation network to predict the edit operation label token by token. The edit operation label includes: retain label, delete label, and replace label.

[0090] In response to the prediction that the edit operation tag of the target token is a delete tag and the target token belongs to a preset set of spoken language correction trigger words, the target token and its associated context correction fragments are marked as discarded semantic fragments and removed.

[0091] In response to the target token's edit operation tag being predicted as a replacement tag, the target token is masked based on contextual semantics to generate a replacement token;

[0092] The second text sequence is generated by arranging the retention token corresponding to the retention tag, the replacement token generated by the replacement tag, and the remaining tokens after removing the discarded semantic fragments in chronological order.

[0093] In specific implementation, firstly, the fused semantic vector can be concatenated with the position encoding vector of each token in the first text sequence to generate a token-level joint representation sequence. That is, the position encoding vector is generated using a sinusoidal position encoding method. For the p-th token in the sequence, the i-th dimension of its position encoding vector is calculated according to a preset sine function. Tokens at different positions have different position encoding vectors. By combining the position encoding vector with the token's semantic vector, the fused semantic vector corresponding to each token and its position encoding vector are concatenated end-to-end along the feature dimension direction. That is, the components of the fused semantic vector are arranged first, followed by the components of the position encoding vector, forming a joint representation vector with a dimension equal to the sum of the two. The joint representation vectors of the tokens are arranged in their original order to obtain the token-level joint representation sequence. Next, this sequence can be input into an edit operation classifier based on a pointer generation network to predict the edit operation label for each token. The pointer generation network-based edit operation classifier is a neural network structure specifically designed for sequence editing tasks. It consists of a feature extraction network composed of multiple layers of bidirectional long short-term memory networks or self-attention-based transformer encoding layers, and a linear classification layer. The token-level joint representation sequence is processed by the feature extraction network, which performs a context-aware nonlinear transformation on the joint representation vector of each token, thus integrating the token's joint representation vector with its original meaning. After feature extraction, the context information of adjacent tokens is used to output a context representation sequence of the same length as the input sequence. Each token representation vector in this sequence is then fed into a linear classification layer, which consists of a fully connected layer and a normalized exponential function layer. The fully connected layer maps the input token representation vector to a three-dimensional score vector, where the three dimensions correspond to the original scores of the retain, delete, and replace labels, respectively. The normalized exponential function layer converts these three scores into a probability distribution, representing the probability values ​​for each of the three labels: retain, delete, and replace. For each token, the label with the highest probability value is selected as the predicted edit operation label. Here, retaining the label indicates that the token is... The text content of the token is accurate and necessary, and should be retained as is in the final output second text sequence. The deletion tag indicates that the token or its associated fragments belong to non-fluent redundancy in spoken expression and should be removed from the sequence. Non-fluent redundancy includes: spoken correction trigger words, semantic fragments that are actively abandoned, repeated filler words, and text corresponding to meaningless sounds caused by hesitation. The replacement tag indicates that the text content of the token has recognition errors or inappropriate expression and needs to be replaced with more appropriate words. Then, in response to the target token's edit operation tag being predicted as a deletion tag and the target token belonging to the preset spoken correction trigger word set, the target token and its associated context correction fragments are marked as discarded semantic fragments and removed.

[0094] Furthermore, in specific implementation, in response to the target token's edit operation tag being predicted as a delete tag and the target token belonging to a preset set of spoken language correction trigger words, the target token and its associated context correction fragments can be marked as discarded semantic fragments and removed. That is, the boundary determination rule for associated context correction fragments is: starting from the first token after the target token, scanning sequentially until a syntactic pause boundary is encountered. Syntactic pause boundaries include tokens corresponding to punctuation marks such as commas, periods, semicolons, question marks, and exclamation marks, as well as empty tokens representing longer silent segments. All consecutive tokens following the target token and preceding the syntactic pause boundary, along with the target token itself, are marked as discarded semantic segments. After marking, all tokens contained in the discarded semantic segments are removed from the first text sequence. After the removal operation, tokens originally following the discarded semantic segments in the sequence are shifted forward to fill the gaps created by the removal. Then, in response to the edit operation tag of the target token being predicted as a replacement tag, the target token is masked based on the context semantics to generate a replacement token. That is, the rest of the token sequence in the first text sequence, except for the position of the target token, is input into the... The pre-trained masked language model sets the target token's location as a mask marker. Through its internal multi-layered self-attention mechanism, the pre-trained masked language model integrates bidirectional contextual semantic information before and after the mask position. At the output layer, it predicts the word that should be filled at the mask position. The masked language model outputs a probability distribution covering all words in the vocabulary, selecting the word with the highest probability value as the replacement token. After generating the replacement token, this replacement token replaces the original target token in the first text sequence. Finally, based on the retained tokens corresponding to the retained labels, the replacement tokens generated by the replacement labels, and the remaining tokens after removing discarded semantic segments, the second text sequence is generated in chronological order. That is, each token position in the first text sequence is traversed. For tokens predicted as retained labels, their original text content is directly copied to the corresponding position in the output sequence. For tokens predicted as replacement labels, their original text content is replaced by the replacement token. For tokens marked as discarded semantic segments and removed, they do not occupy any position in the output sequence. Subsequent tokens are shifted forward sequentially. All retained or replaced tokens are arranged in chronological order according to their original chronological positions in the first text sequence to form the second text sequence.

[0095] like Figure 3The flowchart shown illustrates the sequence editing operation, illustrating the complete data flow from the input of a first text sequence and a fused semantic vector, through processing by the editing operation classifier, to the generation of a second text sequence: The first text sequence and the fused semantic vector are input to the editing operation classifier, which predicts the editing operation label for each token in the first text sequence, outputting three editing operation types: retain, delete, and replace. The retain branch directly passes the corresponding token to the temporal arrangement node; the delete branch marks the corresponding token and its associated context correction fragment as discarded semantic fragments and removes them from the sequence, then passes the remaining tokens to the temporal arrangement node; the replace branch generates a mask marker at the corresponding token position, calls a pre-trained language model to fill in the replacement token, and then passes it to the temporal arrangement node. The temporal arrangement node arranges the tokens from the three branches in the original temporal order, ultimately generating a second text sequence that removes non-fluency redundancy and maintains temporal coherence.

[0096] It should be noted that by performing sequence editing operations on the first text sequence through the semantic reconstruction model, the non-fluency and redundant content in spoken expression is identified and removed by utilizing the cross-modal information carried in the fused semantic vector, while retaining the original temporal relationship of effective semantic segments, thus generating the second text sequence. This improves the adaptability to real spoken language scenarios and the accuracy of semantic parsing.

[0097] In step S4, the second text sequence is subjected to semantic parsing enhanced with domain knowledge to generate intent encoding and entity encoding, and the business knowledge graph is retrieved to generate structured response data, driving the virtual object to perform action feedback.

[0098] In this embodiment, the second text sequence is subjected to domain-knowledge-enhanced semantic parsing to generate intent encoding and entity encoding, which can be achieved through the following steps:

[0099] Perform semantic parsing on the second text sequence to extract an explicit entity mention list;

[0100] Get a snapshot of the historical dialogue state of the current interaction session, and extract the implicit context entities and their corresponding timeliness decay weights;

[0101] The explicit entity mention list and the implicit context entity weighted by the timeliness decay weight are linked together to the corresponding entity node in the preset business knowledge graph. The adjacency relationship is extracted with the corresponding entity node as the center according to the preset association hop number to construct a dynamic knowledge subgraph.

[0102] The dynamic knowledge subgraph is graph embedded and encoded to generate a knowledge enhancement vector. The knowledge enhancement vector is then gated and fused with the semantic encoding vector of the second text sequence to generate intent encoding and entity encoding.

[0103] In practical implementation, firstly, semantic parsing can be performed on the second text sequence to extract an explicit entity mention list. That is, named entity recognition can be used to perform sequence labeling on the second text sequence, merging tokens consecutively labeled with the same entity category into explicit entity mentions, and recording their entity text, category, and location information. Secondly, a snapshot of the historical dialogue state of the current interactive session can be obtained to extract implicit context entities and their corresponding timeliness decay weights. That is, the historical dialogue state snapshot refers to the state record obtained after structured extraction of the dialogue history within the same interactive session before the current round. The state record includes: the user intent identified by the system in each historical round, and the state of each historical round. The entity mention list extracted from the database, along with the dialogue round number or timestamp corresponding to each entity mention, is used to extract previously appearing entities from historical dialogue state snapshots as implicit context entities. A timeliness decay weight is calculated based on the distance between the mention's current round and its previous round; the greater the distance, the lower the weight. Entities below a truncation threshold are removed. Then, the explicit entity mention list and the implicit context entities weighted by the timeliness decay weight are linked to corresponding entity nodes in a pre-defined business knowledge graph. Adjacency relationships are extracted from these corresponding entity nodes according to a pre-defined number of association hops, constructing a dynamic knowledge subgraph. In other words, the pre-defined business knowledge graph is a structure pre-built for a specific interaction domain. A structured knowledge base is constructed, with entities as nodes and semantic relationships between entities as edges, forming a directed or undirected graph structure. For example, in the navigation domain, entity nodes in the knowledge graph include: location names, road names, point-of-interest categories, administrative divisions, etc.; edges between entities include: location relationships, adjacency relationships, affiliation relationships, distance relationships, etc. Explicit entities and implicit entities (after attenuation weighting) are linked to corresponding nodes in the business knowledge graph. Explicit entities are directly matched, while implicit entities require a weighted score exceeding a confidence threshold to be linked. Using the successfully linked anchor node as the center, adjacency relationships are extracted according to a preset association hop count, constructing a knowledge graph containing associated entities and semantic relationships. The dynamic knowledge subgraph is then used for edge-based encoding. Finally, graph embedding encoding is performed on the dynamic knowledge subgraph to generate knowledge enhancement vectors. These knowledge enhancement vectors are then gated and fused with the semantic encoding vectors of the second text sequence to generate intent encoding and entity encoding. Specifically, semantic encoding is performed on the second text sequence to obtain semantic encoding vectors, and graph convolution or graph attention encoding is performed on the dynamic knowledge subgraph to obtain knowledge enhancement vectors. The two vectors are concatenated and input into a gating network to generate gating coefficients. The two vectors are then weighted and fused based on these gating coefficients. The fusion results are input into the intent classifier and entity classifier, respectively, to generate intent encoding and entity encoding. Entity encoding includes the unique identifier of the entity in the knowledge graph and its entity category information.

[0104] In practical implementation, structured response data is generated by retrieving information from a pre-defined business knowledge graph based on intent encoding. This structured response data then drives the virtual interactive object to perform corresponding action feedback. Specifically, firstly, the target relation type is determined based on intent encoding. Starting from the entity node in the knowledge graph corresponding to the entity encoding, a path traversal is performed along the edges of the target relation type, extracting node and edge attribute information. The retrieved knowledge information is then organized according to a pre-defined format to generate structured response data. This structured response data includes response type fields, text content fields, entity information fields, operation instruction fields, and interaction control fields. Secondly, the action feedback of the virtual interactive object is divided into three levels: the first level is voice-synchronized feedback, driving the virtual interactive object's lip movements to remain synchronized with the audio stream; the second level is semantic-accompanied feedback, driving the virtual interactive object to perform actions such as nodding, shaking its head, and gesture instructions based on the semantics and emotion of the response content. The third level of coordinated action is state adaptation feedback, which dynamically adjusts the action style and rhythm of the virtual interactive object based on the user's acoustic state. Next, a pre-defined action mapping rule library is constructed, storing the correspondence between response data types and action identifiers. When the response data contains spatial directional instructions, the rotation angle of the virtual interactive object's skeletal nodes is calculated, driving it to execute orientation deflection and gesture pointing animations. When the response data contains emotional polarity labels, a matching texture map package is called to perform blending deformation processing on the facial mesh, presenting the corresponding expression state. When the response data contains interactive waiting instructions, the virtual interactive object is driven to execute a loop waiting action. Finally, the speech synthesis engine is called to convert the response text content into an audio data stream and synchronously output phoneme-level time alignment information. After aligning the action control instructions with the speech data stream on the timeline, the output is synchronously sent to the graphics rendering engine and audio playback device, achieving precise synchronization between the virtual interactive object's actions and speech.

[0105] Therefore, this application firstly, by performing streaming speech recognition and generating a first text sequence with temporal tags while extracting acoustic feature vectors, provides a precise frame-level time alignment benchmark for subsequent cross-modal fusion. By preserving the start and end time information of each text unit in the original audio stream, it avoids the modal misalignment problem caused by the loss of time information in traditional asynchronous processing, laying a reliable data foundation for improving the semantic understanding accuracy in spoken language interaction scenarios. Secondly, by using a cross-modal attention mechanism to non-linearly couple acoustic feature vectors with text word embedding features, the fused semantic vector carries the explicit semantic content of the text sequence and also incorporates paralinguistic information such as prosodic features and vocal physiological states contained in the speech signal. This allows for dynamic compensation of the text modality by the acoustic modality when the user hesitates, makes corrections, or experiences emotional fluctuations, capturing the nuances that cannot be reflected at the text level. The system implicitly identifies user intent, thereby enhancing the depth of perception and robustness of semantic understanding of natural spoken language. Then, a semantic reconstruction model performs sequence editing on the first text sequence, utilizing cross-modal information carried in the fused semantic vectors to identify and remove non-fluent redundant content from spoken language while preserving the original temporal relationships of effective semantic segments, generating a second text sequence. This improves adaptability to real-world spoken language scenarios and the accuracy of semantic parsing. Finally, domain-knowledge-enhanced semantic parsing of the second text sequence, combined with business knowledge graph retrieval, generates structured response data and drives virtual objects to perform action feedback. Because the second text sequence has eliminated redundant interference from spoken language correction, the text content input to the semantic parsing stage is more accurate and concise, thus improving the accuracy of intent recognition and entity linking, and enhancing the robustness and accuracy of virtual interactive responses.

[0106] In summary, the technical solution adopted in this application can dynamically locate spoken language correction boundaries using acoustic paralinguistic features, and improve the robustness and accuracy of virtual interactive responses through closed-loop linkage of cross-modal information and semantic reconstruction.

[0107] Example 2: This application provides a reference for a virtual interactive response system based on speech semantic recognition. Figure 4 As shown, this figure is a block structure diagram of the response method according to this embodiment of the present application. The response method includes:

[0108] The first text generation module, in response to the acquired real-time speech stream, extracts acoustic feature vectors and performs real-time speech recognition to generate a first text sequence with time-series markers.

[0109] The semantic vector generation module extracts word embedding features from the first text sequence, determines the nonlinear coupling weights between the acoustic feature vector and the word embedding features based on cross-modal attention, and generates a fused semantic vector.

[0110] The second text generation module performs sequence editing on the first text sequence according to preset semantic reconstruction rules to generate a second text sequence that removes non-fluent redundancy and is temporally coherent.

[0111] The structured response module performs domain-knowledge-enhanced semantic parsing on the second text sequence to generate intent and entity codes, and retrieves business knowledge graphs to generate structured response data, driving virtual objects to perform action feedback.

[0112] The foregoing has detailed examples of a virtual interactive response system and method based on speech semantic recognition provided in this application. It is understood that the corresponding apparatus, in order to achieve the above functions, includes hardware structures and / or software modules corresponding to the execution of each function. Those skilled in the art should readily recognize that, based on the units and algorithm steps of the examples described in conjunction with the embodiments disclosed herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed by hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0113] In embodiment three, this application also provides a computer device, the computer device including a memory and a processor, the memory for storing a computer program, and the processor for calling and running the computer program from the memory, so that the computer device executes the above-described virtual interactive response method based on speech semantic recognition.

[0114] In this embodiment, reference Figure 5 The dashed lines in the figure indicate that the unit or module is optional. This figure is a structural schematic diagram of a computer device for a virtual interactive response system based on speech semantic recognition, according to an embodiment of this application. The virtual interactive response method based on speech semantic recognition described in the above embodiment can be... Figure 5 The computer device shown is used to implement this, and the computer device includes at least one processor 501, a memory 502 and at least one communication unit 505. The computer device may be a terminal device, a server or a chip.

[0115] Processor 501 can be a general-purpose processor or a special-purpose processor. For example, processor 501 can be a central processing unit (CPU), which can be used to control computer devices, execute software programs, and process data from software programs. The computer device may also include a communication unit 505 to realize signal input (reception) and output (transmission).

[0116] For example, the computer device may be a chip, and the communication unit 505 may be the input and / or output circuit of the chip, or the communication unit 505 may be the communication interface of the chip, which may be a component of a terminal device, network device or other device.

[0117] For example, the computer device may be a terminal device or a server, and the communication unit 505 may be a transceiver of the terminal device or the server, or the communication unit 505 may be a transceiver circuit of the terminal device or the server.

[0118] The computer device may include one or more memories 502 storing a program 504. The program 504 can be executed by a processor 501 to generate instructions 503, causing the processor 501 to perform the methods described in the above method embodiments according to the instructions 503. Optionally, the memory 502 may also store data (such as a target audit model). Optionally, the processor 501 may also read data stored in the memory 502, which may be stored at the same storage address as the program 504, or the data may be stored at a different storage address than the program 504.

[0119] The processor 501 and memory 502 can be configured separately or integrated together, for example, integrated on the system-on-chip (SOC) of the terminal device.

[0120] It should be understood that each step of the above method embodiment can be completed by hardware logic circuits or software instructions in processor 501. Processor 501 can be a central processing unit, digital signal processor (DSP), application specific integrated circuit (ASIC), field programmable gate array (FPGA), or other programmable logic device, such as discrete gate, transistor logic device, or discrete hardware component.

[0121] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0122] In embodiment four, this application also provides a computer-readable storage medium storing instructions or code that, when executed on a computer, cause the computer to implement the above-described virtual interactive response method based on speech semantic recognition.

[0123] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0124] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of the invention. Therefore, if these modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include these modifications and variations.

Claims

1. A method for responding to virtual interaction based on voice semantic recognition, characterized in that, The response method includes the following steps: In response to the acquired real-time speech stream, acoustic feature vectors are extracted and real-time speech recognition is performed to generate a first text sequence with time-series markers; Extract word embedding features from the first text sequence, determine the nonlinear coupling weights between the acoustic feature vector and the word embedding features based on cross-modal attention, and generate a fused semantic vector; Based on preset semantic reconstruction rules, sequence editing is performed on the first text sequence to generate a second text sequence that removes non-fluent redundancy and is temporally coherent; The second text sequence is subjected to domain-knowledge-enhanced semantic parsing to generate intent encoding and entity encoding, and the business knowledge graph is retrieved to generate structured response data, driving the virtual object to perform action feedback. 2.The virtual interactive response method based on voice semantic recognition of claim 1, wherein, In response to the acquired real-time speech stream, the extraction of acoustic feature vectors specifically includes: After pre-emphasis and frame-segmentation windowing processing of the real-time audio stream, multi-dimensional basic acoustic parameters are extracted. Based on the aforementioned multidimensional basic acoustic parameters, inverse filtering of the glottal wave is performed to separate the glottal excitation source signal and the vocal tract impulse response signal. The glottal closure quotient parameter of the glottal excitation source signal is extracted, and the glottal closure quotient parameter is used as a secondary language feature channel to characterize the physiological state of vocalization. The multidimensional basic acoustic parameters and the secondary language feature channels are concatenated along the time dimension to generate an acoustic feature vector. 3.The virtual interactive response method based on voice semantic recognition of claim 1, wherein, Real-time speech recognition, specifically generating a first text sequence with time-stamped tags, includes: The real-time speech stream is input into the encoder of the streaming end-to-end speech recognition model to extract the acoustic hidden layer representation sequence. The acoustic hidden layer representation sequence is input into the connected temporal classification decoder to predict the posterior probability distribution of phonemes frame by frame, and a recognition token sequence carrying a timestamp is generated based on the dynamic programming bundle search algorithm. A first text sequence with time-series markers is generated based on the identification token sequence.

4. The virtual interactive response method based on voice semantic recognition of claim 3, wherein, Generating a first text sequence with a time-series marker based on the identified token sequence specifically includes: Confidence assessment is performed on each token in the identified token sequence; If the confidence level of the target token is lower than the preset semantic validity threshold, an empty token is generated to replace the target token, and the original timestamp interval marker corresponding to the target token is retained; Based on the identified token sequence and the inserted empty token, a first text sequence with a time sequence marker is generated.

5. The virtual interactive response method based on speech semantic recognition according to claim 1, characterized in that, Extracting word embedding features from the first text sequence, determining the non-linear coupling weights between the acoustic feature vector and the word embedding features based on cross-modal attention, and generating a fused semantic vector specifically includes: The acoustic feature vector is divided into an acoustic feature frame sequence along the time axis, and the word embedding features of the first text sequence are divided into a text token sequence. Construct a temporal offset alignment matrix to establish a mapping relationship between each acoustic feature frame in the acoustic feature frame sequence and the text token sequence based on the temporal marker; The acoustic feature frame sequence and the text token sequence are input into a multi-head cross-modal attention network to determine the text-dominated attention distribution with the text modality as the query vector and the acoustic modality as the key vector, and the acoustic-dominated attention distribution with the acoustic modality as the query vector and the text modality as the key vector, respectively. The text-dominant attention distribution and the acoustic-dominant attention distribution are weighted and fused through a gated adaptive fusion layer to generate a fused semantic vector.

6. The virtual interactive response method based on speech semantic recognition according to claim 1, characterized in that, Based on preset semantic reconstruction rules, sequence editing is performed on the first text sequence to generate a second text sequence that removes non-fluent redundancy and is temporally coherent. Specifically, this includes: The fused semantic vector is concatenated with the position encoding vector of each token in the first text sequence to generate a token-level joint representation sequence; The token-level joint representation sequence is input into an edit operation classifier based on a pointer generation network to predict the edit operation label token by token. The edit operation label includes: retain label, delete label, and replace label. In response to the prediction that the edit operation tag of the target token is a delete tag and the target token belongs to a preset set of spoken language correction trigger words, the target token and its associated context correction fragments are marked as discarded semantic fragments and removed. In response to the target token's edit operation tag being predicted as a replacement tag, the target token is masked based on contextual semantics to generate a replacement token; The second text sequence is generated by arranging the retention token corresponding to the retention tag, the replacement token generated by the replacement tag, and the remaining tokens after removing the discarded semantic fragments in chronological order.

7. The virtual interactive response method based on speech semantic recognition according to claim 1, characterized in that, The second text sequence is subjected to domain-knowledge-enhanced semantic parsing to generate intent encoding and entity encoding, specifically including: Perform semantic parsing on the second text sequence to extract an explicit entity mention list; Get a snapshot of the historical dialogue state of the current interaction session, and extract the implicit context entities and their corresponding timeliness decay weights; The explicit entity mention list and the implicit context entity weighted by the timeliness decay weight are linked together to the corresponding entity node in the preset business knowledge graph. The adjacency relationship is extracted with the corresponding entity node as the center according to the preset association hop number to construct a dynamic knowledge subgraph. The dynamic knowledge subgraph is graph embedded and encoded to generate a knowledge enhancement vector. The knowledge enhancement vector is then gated and fused with the semantic encoding vector of the second text sequence to generate intent encoding and entity encoding.

8. A virtual interactive response system based on speech semantic recognition, used to execute a virtual interactive response method based on speech semantic recognition as described in any one of claims 1 to 7, characterized in that, The virtual interactive response system based on speech semantic recognition includes: The first text generation module is used to extract acoustic feature vectors in response to the acquired real-time speech stream, and perform real-time speech recognition to generate a first text sequence with time-series markers. The semantic vector generation module is used to extract word embedding features from the first text sequence, determine the nonlinear coupling weight between the acoustic feature vector and the word embedding features based on cross-modal attention, and generate a fused semantic vector. The second text generation module is used to perform sequence editing on the first text sequence according to preset semantic reconstruction rules, and generate a second text sequence that removes non-fluent redundancy and is temporally coherent. The structured response module is used to perform domain-knowledge-enhanced semantic parsing on the second text sequence, generate intent encoding and entity encoding, retrieve business knowledge graph to generate structured response data, and drive virtual objects to perform action feedback.

9. A computer device, characterized in that, The computer device includes a memory and a processor. The memory is used to store computer programs, and the processor is used to call and run the computer programs from the memory, so that the computer device performs a virtual interactive response method based on speech semantic recognition as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores instructions or code that, when executed on a computer, cause the computer to implement a virtual interactive response method based on speech semantic recognition as described in any one of claims 1 to 7.

Images