A voice interaction method and system based on 3D virtualization

Abstract

The present invention relates to the field of intelligent voice interaction technology, and discloses a 3D virtual-based voice interaction method and system, including: receiving a user's real-time voice and preprocessing it, establishing a voice recognition model, and analyzing the user's pronunciation characteristics and clarity; optimizing the voice recognition model based on the user's pronunciation characteristics and clarity; combining the user's historical commands and the current 3D environment context, and using the optimized voice recognition model to generate a set of candidate predicted instructions for the user's next step; the candidate predicted instruction set includes multiple candidate instructions; identifying the priority of each candidate instruction, selecting the candidate instruction with the highest priority as the user's next predicted instruction, prioritizing the loading of related resources for the predicted instructions, and quickly responding to user needs. This method more accurately predicts user needs in complex scenarios, significantly improving the accuracy and real-time responsiveness of voice interaction.

Description

Technical Field

[0001] The present invention relates to the field of intelligent voice interaction technology, and specifically to a 3D virtual-based voice interaction method and system. Background Art

[0002] With the rapid development of computer technology and artificial intelligence, speech recognition technology has become a key means of human-computer interaction. Especially in immersive environments such as virtual reality (VR) and augmented reality (AR), speech recognition not only provides users with a more natural way of interaction but also significantly enhances the user experience. However, existing speech recognition systems still face many challenges in real-world applications, such as varying speech clarity, background noise interference, adaptability issues for users with articulation disorders, and understanding context in complex environments.

[0003] Traditional speech recognition systems are typically based on static models, utilizing fixed features and algorithms to process speech signals. While these systems can handle standard speech input, they often struggle with variations in pronunciation, accent, speech rate, and ambient noise. Furthermore, existing speech recognition models lack sufficient adaptability to handle complex scenarios and contextual changes. For example, information about a user's behavior, context, and historical commands in a virtual environment is often not effectively utilized, limiting the speed of command generation and response, impacting the user's interactive experience.

[0004] While some current optimization methods have introduced technologies like deep learning to improve speech recognition accuracy, most are limited to analyzing single speech input and neglect the combined use of historical user commands and environmental context. Specifically, existing technologies have yet to effectively combine historical user interactions with real-time environmental context to dynamically optimize speech recognition models and the generation of candidate command sets, further improving the accuracy and responsiveness of speech recognition systems in practical applications. Summary of the Invention

[0005] In view of the above-mentioned problems, the present invention is proposed.

[0006] Therefore, the technical problem solved by the present invention is that the existing speech recognition system cannot effectively combine the user's historical commands and 3D environment context in a complex environment, resulting in inaccurate command generation and slow response speed.

[0007] To solve the above technical problems, the present invention provides the following technical solutions: a 3D virtual-based voice interaction method, comprising: receiving and preprocessing a user's real-time voice, establishing a voice recognition model, and analyzing the user's pronunciation characteristics and clarity;

[0008] Optimize the speech recognition model based on the user's pronunciation characteristics and clarity;

[0009] Combining the user's historical commands and the current 3D environment context, using the optimized speech recognition model to generate a set of candidate predicted instructions for the user's next step; the candidate predicted instruction set includes multiple candidate instructions;

[0010] Identify the priority of each candidate instruction, use the candidate instruction with the highest priority as the user's next predicted instruction, prioritize loading related resources of the predicted instruction, and quickly respond to user needs; identifying the priority of each candidate instruction includes calculating the priority of each candidate instruction based on the voice clarity score, the execution frequency of historical commands, and the matching degree of the instruction with the current environment.

[0011] As a preferred solution of the 3D virtual voice interaction method described in the present invention, the preprocessing includes using the Wiener filter algorithm to remove the noise of the real-time voice, dividing the denoised real-time voice into multiple continuous frames of audio, and the overlapping part between each frame is ; Use Mel frequency cepstral coefficients to extract the time-frequency features of each frame of audio and establish a time-frequency feature matrix; where, Indicates the overlap length;

[0012] The time-frequency features include the spectrum, pitch, volume, fundamental frequency and spectrum flatness of the audio.

[0013] As a preferred solution of the 3D virtual voice interaction method described in the present invention, wherein: the voice recognition model includes a convolutional neural network layer, a pooling layer and a fully connected layer;

[0014] The time-frequency feature matrix is ​​input into the convolutional neural network layer for convolution operation to extract local features and form a local feature map; the pooling layer uses the maximum pooling method to reduce the dimension of the local features to obtain the key local feature map; the fully connected layer converts the local feature map and the key local feature map into feature vectors to construct a global feature map.

[0015] As a preferred solution of the 3D virtual voice interaction method described in the present invention, wherein: analyzing the clarity of the user's pronunciation includes extracting high-frequency coefficients and low-frequency coefficients from the global feature map using a wavelet transform method; calculating the energy of the high-frequency coefficients and the energy of the low-frequency coefficients; determining the clarity score of the speech based on the ratio of the high-frequency energy to the low-frequency energy; setting a clarity score threshold, when the clarity score When the clarity score threshold is reached, the speech clarity is judged to be insufficient and the error tolerance rate is increased;

[0016] Analyzing the user's pronunciation characteristics includes using the sliding window method to calculate the local energy of the high-frequency detail coefficient, finding the local energy peak, and determining it as the stressed part; calculating the average time interval between adjacent local energy peaks; setting the fast speech rate threshold and the slow speech rate threshold, and when the average time interval is When the speaking speed threshold is high, the speaking speed is judged to be too fast, and the weight of the high-frequency filter in the convolutional neural network is increased to enhance the ability to capture fast-changing syllables; when the average time interval is The slow speech threshold determines the speech speed is slow, increases the weight of the low-frequency filter in the convolutional neural network, and enhances the recognition ability of slow-paced speech.

[0017] As a preferred embodiment of the 3D virtual reality-based voice interaction method of the present invention, generating a set of candidate predicted commands for the user's next step includes receiving a historical command sequence of the user; converting the historical command sequence into time series features using a long short-term memory network to generate a hidden state of the historical commands; and recursively updating the hidden state to learn information from the historical commands to obtain a historical command state;

[0018] Using the optimized speech recognition model, we analyze the real-time voice input received to obtain real-time voice features. We build a multi-layer deep model to deduce the user's possible operation intentions layer by layer and generate a set of candidate predicted instructions for the user's next step.

[0019] As a preferred solution of the 3D virtual voice interaction method of the present invention, wherein: the multi-layer depth model includes an LSTM layer, a GCN layer and an output layer;

[0020] In the LSTM layer, combining the historical command status and the real-time speech features, using LSTM to generate a preliminary candidate command set; combining the historical command status and the real-time speech features, using LSTM to generate a real-time candidate command set;

[0021] The GCN layer includes: based on the user's historical command sequence, using the meta-learning framework to pre-train the rule engine offline and establish a user-specific rule model; when the user starts voice interaction, based on the current command voice clarity score and the average value of all syllable intervals in the current command, sampling subsamples with the same command voice clarity score and the average value of all syllable intervals in the current command from the historical interaction data to construct a support set, quickly fine-tune the meta-model parameters, and generate a rule engine personalized for the current user; for each candidate command, using the support set to extract a semantic-behavior joint feature vector; based on the semantic-behavior joint feature vector, construct a heterogeneous graph; the node set of the constructed heterogeneous graph includes: command node, target object node and context semantic node; let any two nodes i, j in the heterogeneous graph have the corresponding semantic-behavior joint feature vector as and ,when When , an edge is established between i and j, and the weight of the edge is ;in, Represents feature similarity, Indicates the set structural connection threshold.

[0022] As a preferred solution of the 3D virtual voice interaction method described in the present invention, wherein: structural propagation is performed on the heterogeneous graph through a multi-layer graph convolutional network to obtain a graph structure embedding representation of the candidate instructions, and the instruction prediction result is output through a prediction head structure; the prediction result is compared with the actual interaction behavior label to identify the prediction error. When the prediction error exceeds a set threshold, the structural risk factor of the candidate instruction is extracted and a structural attribution matrix is ​​constructed;

[0023] The structural risk factor includes a semantic ambiguity factor and a path conflict factor; the semantic ambiguity factor is a quantitative indicator of the degree of semantic concentration when a candidate instruction node points to multiple objects or semantic nodes; all edges of the candidate instruction node and their corresponding edge weights are extracted, and after normalizing the edge weight set into a probability distribution, the uncertainty degree of the probability distribution is calculated based on Shannon entropy, and the obtained entropy value is normalized as the value of the semantic ambiguity factor; the path conflict factor represents the degree of overlap between the path where the candidate instruction node is located and the adjacent candidate path in the graph structure on the structural edge set, defining the node The path is ,extract The edge set ; According to the figure and There are directly connected instruction nodes to construct neighbor path sets , extract each neighbor path separately The edge set , and calculate with The structural Jaccard similarity between the two is taken as the maximum value of the structural Jaccard similarity. ;

[0024] The structural attribution matrix is ​​introduced as a regulating factor into the parameter update path of the personalized rule engine. The updated personalized rule engine is used to calculate the personalized preference score of each candidate instruction, guiding the meta-model to avoid structural high-risk areas in subsequent tasks; the personalized preference score is integrated with the semantic adaptation score to generate a comprehensive score for the candidate instructions, and the candidate instructions are sorted by score, and the top M candidate instructions are selected as the environmental context candidate instruction set; the semantic adaptation score is the embedding probability value output by the prediction head after the graph convolutional network structure propagation, which is used to measure the degree of structural semantic fit between the candidate instructions and the target objects and context nodes in the graph; the output layer uses the attention mechanism to weightedly fuse the preliminary candidate instruction set, the real-time candidate instruction set and the environmental context candidate instruction set to generate a candidate predicted instruction set for the user's next step.

[0025] A 3D virtual voice interaction system, wherein:

[0026] The data module receives the user's real-time voice and performs preprocessing, builds a speech recognition model, and analyzes the user's pronunciation characteristics and clarity;

[0027] The optimization module optimizes the speech recognition model based on the user's pronunciation characteristics and clarity;

[0028] The prediction module combines the user's historical commands and the current 3D environment context and uses the optimized speech recognition model to generate a set of candidate predicted instructions for the user's next step; the candidate predicted instruction set includes multiple candidate instructions;

[0029] The loading module identifies the priority of each candidate instruction, takes the candidate instruction with the highest priority as the user's next predicted instruction, and prioritizes loading related resources of the predicted instruction to quickly respond to user needs.

[0030] A computer device comprises: a memory and a processor; the memory stores a computer program, wherein the processor implements the steps of any one of the methods of the present invention when executing the computer program.

[0031] A computer-readable storage medium stores a computer program thereon, wherein the computer program, when executed by a processor, implements the steps of any one of the methods of the present invention.

[0032] Beneficial effects of the present invention: The 3D virtual-based voice interaction method provided by the present invention proposes a candidate instruction generation method of a multi-layer deep model by combining the optimized voice recognition model, user historical commands and 3D environmental context, which significantly improves the accuracy and real-time response capability of voice interaction. Through the gradual reasoning and integration of voice input, historical data and environmental information by a multi-layer deep learning structure, the present invention can dynamically optimize the instruction generation process to ensure that the system can more accurately predict user needs in complex scenarios. In addition, the use of a fault tolerance adjustment mechanism and a clarity-based optimization strategy enables the system to effectively deal with problems such as unclear pronunciation or voice interference, thereby improving the robustness of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0034] Figure 1 An overall flow chart of a 3D virtual voice interaction method provided for the first embodiment of the present invention. DETAILED DESCRIPTION

[0035] To make the above-mentioned objects, features, and advantages of the present invention more clearly understood, the following detailed description of the specific embodiments of the present invention is given in conjunction with the accompanying drawings. It is obvious that the described embodiments are only part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary persons in this field without creative work should fall within the scope of protection of the present invention.

[0036] Example 1, with reference to Figure 1 , as an embodiment of the present invention, provides a 3D virtual voice interaction method, comprising:

[0037] S1: Receive the user's real-time voice and pre-process it, establish a voice recognition model, analyze the user's pronunciation characteristics and clarity; optimize the voice recognition model based on the user's pronunciation characteristics and clarity.

[0038] The Wiener filter algorithm is used to remove the noise of the received real-time speech, and then the denoised real-time speech is divided into multiple time periods of The overlapping part between each frame is ;in Indicates the frame length, represents the overlap length, and for This ensures that the signal within each frame can be considered a stationary signal, facilitating time-frequency feature extraction. After the time-frequency transformation of each frame of audio signal, the relevant time-frequency features are extracted to provide accurate input for the subsequent speech recognition process.

[0039] The speech recognition model includes convolutional neural network layers, pooling layers, and fully connected layers.

[0040] The time-frequency feature matrix is ​​input into the convolutional neural network layer, and the time-frequency feature matrix is ​​convolved by the convolution kernel to extract local features. The formula is:

[0041] After the convolution operation, the maximum pooling is used through the pooling layer to reduce the dimension of the local features, retain the most important feature information, and obtain the key local feature map. The formula is expressed as:

[0042]

[0043] in, Represents the local feature map output by the convolutional layer at position The eigenvalue at ; Represents the input time-frequency feature matrix Middle, location The elements at Represents the convolution kernel Middle, location The weight of the place; is the convolution kernel; Represents the index of the convolution kernel. The key local feature map representing the output of the pooling layer is at position The eigenvalue at ; Represents the local feature map output by the convolutional layer Middle, location The eigenvalue at It is the local feature map after convolutional layer processing; Indicates the offset of the pooling window. and Indicates a position index.

[0044] The fully connected layer converts local feature maps and key local feature maps into feature vectors and constructs global features. The fully connected layer connects all other neurons through multiple neurons to map high-dimensional feature information to the final output space.

[0045]

[0046] Will and Flattened into a single feature vector , as the input of the fully connected layer.

[0047]

[0048]

[0049] Where v represents the flattened eigenvector; Represents the local feature map output by the convolutional layer; Represents the key local feature map output by the pooling layer; Flatten represents the flattening operation; z represents the linear output of the fully connected layer; represents the weight of the fully connected layer, represents the bias term; represents the global feature map, Represents the activation function.

[0050] For the global feature map conduct Layer discrete wavelet transform, get the Low-frequency coefficients of the layer and high frequency coefficients , calculate the The energy of the high-frequency detail part and the low-frequency approximation part of the layer:

[0051]

[0052]

[0053] Calculate the clarity score based on high-frequency energy and low-frequency energy:

[0054]

[0055] in, A very small positive number to prevent the denominator from being zero. Set the clarity score threshold ,when When the speech clarity is insufficient, the recognition model's fault tolerance adjustment mechanism is activated.

[0056] in, Indicates the The low-frequency approximation coefficients after layer wavelet transform contain the stationary components of the speech signal and usually represent the basic contour or main energy of the speech; Indicates the The high-frequency detail coefficients after wavelet transform contain the changing parts of speech, rapid changes or detailed information, such as stress, intonation changes, and pronunciation mutations; Represents the global feature map; Indicates the input signal Carry out the Layer discrete wavelet transform operations. Indicates the The energy of the high-frequency details of the layer; Indicates the High-frequency coefficients after layer wavelet transform The elements; Indicates the The energy of the low-frequency approximation of the layer; Indicates the Low-frequency coefficients after layer wavelet transform The elements; Indicates the speech intelligibility score.

[0057] In the high frequency detail coefficient Set sliding window , calculate the The local high-frequency energy in a window:

[0058]

[0059] The energy peak is detected by sliding comparison method when: 、 and When , it is marked as a valid peak. Extract all valid peaks and get the peak time point sequence , and use this to calculate the time interval between adjacent syllables:

[0060]

[0061] Further calculate the average of all syllable intervals:

[0062]

[0063] Set the fast speech threshold and slow speech threshold , judge the speaking speed according to the following rules: If , judge that the speaking speed is too fast; if , judge that the speaking speed is slow; if , judging the speaking speed as normal. When the speaking speed is too fast or too slow, the weight optimization mechanism is triggered. Indicates the total number of syllables detected in this speech segment. Time interval number index; Indicates the The time interval between one syllable and the next.

[0064] in, Indicates the Sliding window Local high-frequency energy within; Indicates the A sliding window; Indicates the After the layer wavelet transform, the position High-frequency detail coefficient at ; Indicates the The local high-frequency energy of a sliding window; Indicates the The local high-frequency energy of a sliding window; represents the energy threshold; Indicates the The time point corresponding to the effective peak; Indicates the The time point corresponding to the effective peak; Indicates the syllable and The time interval between syllables. Indicates the number of valid peaks; represents the average value of all syllable intervals; Indicates the threshold for fast speech rate. Indicates the threshold for slow speech rate.

[0065] During the training process, the backpropagation algorithm is used to calculate the gradient of the loss function and the network parameters are optimized by gradient descent. The cross entropy loss function is used to measure the gap between the model output and the actual label, and the recognition accuracy is improved by optimizing the loss function.

[0066] The error tolerance adjustment mechanism includes increasing the convolution kernel size for low-definition speech so that the model can capture more local features when processing low-definition speech.

[0067] in, is the original convolution kernel, is the magnification, is the new convolution kernel.

[0068] The step size of the pooling layer is adjusted to 1 / 2 of the original step size, so that the pooling window can capture more fine-grained features and improve the recognition ability of low-definition speech.

[0069]

[0070] in, is the new pooling stride, is the original step size.

[0071] The weight optimization mechanism involves using the backpropagation algorithm to focus on high-error areas during training, particularly those characteristic of fast or slow speech. Gradient descent is used to update the convolution kernel weights, making the model more sensitive to these areas.

[0072]

[0073] in, is the updated convolution kernel weight, is the current convolution kernel weight, is the learning rate, is the gradient of the loss function with respect to the weights.

[0074] Adjust the window size of the pooling layer according to the speed of speech. When the speech speed is too fast, increase the pooling window to ensure that the features of fast speech can be effectively extracted. When the speech speed is too slow, reduce the pooling window to enable more accurate extraction of the detailed features of slow speech. The formula for adjusting the window size of the pooling layer is expressed as:

[0075]

[0076] in, is the adjusted pooling window size, is the original pooling window size.

[0077] By introducing a multi-layer deep learning model and optimization mechanism, we have significantly improved the robustness and accuracy of the speech recognition system. Compared to traditional speech recognition systems, our improvements are mainly reflected in the following aspects: First, by combining a convolutional neural network (CNN) with time-frequency feature extraction, we perform fine-grained analysis of the input speech, effectively extracting local features of the audio, and ensuring that the system can accurately recognize even ambiguous and unclear speech. Second, by introducing a multi-layer wavelet transform, we can dynamically assess speech clarity and pronunciation characteristics, further adjusting the system's fault tolerance to cope with complex speech input. This innovation enables the system to maintain high-precision recognition capabilities despite non-standard pronunciation, speech disorders, or noise interference.

[0078] Furthermore, speech preprocessing and recognition optimization have significant practical value in 3D speech interaction systems. Wiener filtering removes background noise from real-time speech, and the processed speech is then divided into overlapping frames to ensure local signal stationarity within the frames, facilitating the extraction of more representative time-frequency features. Based on this, a convolutional neural network architecture extracts and compresses local features, obtaining key speech patterns through pooling. A fully connected layer further embeds the local graph into the global semantic space to form a unified feature vector. Speech clarity is measured by the energy ratio of high- and low-frequency coefficients decomposed by the discrete wavelet transform. When the score falls below a set threshold, an adjustment mechanism for the recognition model structure is automatically triggered. This mechanism expands the receptive field of the convolution kernel and compresses the pooling step size to capture ambiguous speech information in detail, significantly enhancing the model's adaptability to non-standard pronunciation and speech with low signal-to-noise ratio, effectively improving the fault tolerance and robustness of speech recognition.

[0079] In terms of speech rate feature recognition, a sliding window is used to analyze high-frequency energy changes, extract syllable boundaries corresponding to local peaks, and further calculate the time interval between adjacent peaks to evaluate the speech rate status. Based on the comparison of the average time interval with the preset threshold, the system dynamically determines whether the speech rate is fast or slow. When the speech rate is fast, the pooling window is enlarged to ensure the effective extraction of high-speed variation features; when the speech rate is slow, the pooling window is reduced to capture subtle speech details. During the training process, the recognition model uses the backpropagation algorithm, focusing on the recognition error areas generated under fast and slow speech rates, and dynamically adjusting the convolution kernel weights. This optimization strategy improves the model's responsiveness to speech rate fluctuations, enabling it to maintain high recognition accuracy under different speech rate conditions. The overall structure has an adaptive rhythm adjustment function, which can effectively reduce the error recognition rate caused by changes in pronunciation rhythm, and exhibits higher stability and adaptability in multi-round interactive tasks.

[0080] S2: Combining the user's historical commands and the current 3D environment context, using the optimized speech recognition model to generate a set of candidate predicted instructions for the user's next step; the candidate predicted instruction set includes multiple candidate instructions.

[0081] Extract the user's historical instruction sequence from the user's interaction history, including the instruction's timestamp, execution frequency, and execution context information.

[0082] LSTM is used to convert historical command sequences into time series features. The formula is expressed as:

[0083] in, Indicates the The hidden state of the moment, Indicates the The historical instructions input at each moment. LSTM learns and stores information from historical instructions by recursively updating the hidden state. Indicates the The hidden state of the moment.

[0084] Use the optimized speech recognition model to analyze real-time speech input to obtain real-time speech features .

[0085] The multi-layer deep model includes an LSTM layer, a GCN layer and an output layer.

[0086] At the LSTM layer, the system analyzes the relationship between historical command states and the current voice input, and combines the voice input to generate a preliminary set of candidate commands:

[0087]

[0088] in, Represents the historical command state, which is the final state of LSTM after learning the historical command sequence; represents real-time speech features, Represents a preliminary candidate instruction set.

[0089] Combine historical command status with real-time speech features and use LSTM to generate a real-time candidate command set:

[0090]

[0091] The GCN layer includes: obtaining real-time 3D environment context information, including user location, virtual targets, interactive objects and scene status; providing the system with background about the current environment to help the system make more intelligent predictions.

[0092] Offline, sequence each user's historical instructions into a learning task Perform meta-training:

[0093]

[0094] Learning tasks The internal part is divided into two parts, including support set Used to quickly adapt to user rule preferences; query set Used to evaluate whether the adapted model generalizes well.

[0095] During the support set training process of each meta-task: the historical instruction sequence is encoded to generate a temporal state representation representing the user behavior path; the pronunciation clarity score, speech rhythm vector and language feature embedding corresponding to each instruction are integrated; at the same time, the environmental state (position, object, scene state) of each instruction is embedded as a context vector; the above multi-source data are mapped to the same feature space through a shared embedding network, and jointly optimized to obtain the semantic-behavior joint feature vector , the formula is:

[0096]

[0097]

[0098] in, represents the metamodel embedding function; Represents the current 3D environment context vector; ups represents the user's current coordinates or spatial position label in the virtual scene; Indicates the coding information of the object (such as pump station, button, valve, etc.) associated with the current interactive instruction; Indicates the overall state of the scene, such as space labels, task stages, exception markers, etc.

[0099] In the MAML framework, the optimization goal of meta-training is to enable the model to learn from multiple user tasks simultaneously after receiving a small amount of individual user data. After that, the inner loop obtains the personalized parameters for the user, and the outer loop updates the global metamodel parameters, quickly generating the current user-specific rule engine capabilities, thereby achieving efficient generalization and rapid adaptation. Represents the latest user small sample training set in the online stage; Includes speech clarity score, semantic label, and context object for each instruction; The target instructions selected by the actual user as recorded by the system; the data time window is limited to the most recent 5-10 sessions to ensure adaptability and real-time performance; the total number of data items N is generally set to 8-16.

[0100] Inner loop: for each user task , in the support set Perform gradient update to obtain personalized parameters for the user:

[0101]

[0102] in, Represents the current global model parameters; represents the inner loop learning rate; represents the loss function on the support set. Indicates that the user Fine-tuned personalized model parameters; Indicates the parameters Gradient operation of ; Indicates The rule engine model with parameters.

[0103] Outer loop: adapt the model for each task Applied to query sets , calculate the generalization error, and update the meta-model parameters by aggregating the errors of all tasks back propagating :

[0104]

[0105] in, represents the meta-update learning rate; Represents the loss function value on the query set; Historical support set for users The user's historical query set. : Loss function for parameters gradient; Indicates The rule engine model with parameters.

[0106] When the user starts voice interaction, the system reads the most recent rounds of interaction data and builds a small sample training set. The training set does not directly use all historical data, but introduces a feature guidance mechanism based on the current speech state and environmental context. From the historical interaction records, subsamples with the same average value of the speech clarity score and all syllable intervals in the instructions as the current round of interaction are sampled as the support set to adjust the meta-model parameters. Perform a quick fine-tuning to obtain a personalized rule engine for the current user for real-time reasoning:

[0107]

[0108] in, Represents a user A user-specific rules engine. Represents the parameter update path of the user personalized fine-tuning process in the meta-learning framework; with the global meta-model parameters As a starting point, first in the user Support set Computational Model The loss function , then the loss function is related to the parameters Find the gradient and get the direction that best optimizes the model performance under the current data.

[0109] Construct a heterogeneous graph containing instruction entities, scene objects, and semantic labels ; Nodes include: instruction nodes, target object nodes and context semantic nodes. Among them, Represents a collection of nodes; Represents a set of edges.

[0110] The command node represents the semantic-behavioral joint feature vector of each candidate voice command; the target object node represents the interactive entity in the current 3D scene; and the context semantic node represents the context clues in the user's recent interaction, such as the current scene state, user location label, and previous operation path.

[0111] The connection between nodes is not a predefined fixed edge type, but is dynamically established by calculating the similarity between their semantic-behavioral feature vectors; suppose any two nodes i, j, their corresponding feature vectors are and , then the condition for establishing an edge between two nodes is:

[0112]

[0113] in, To improve feature similarity, Indicates the structural connection threshold set by the system.

[0114] Edges include: instruction-object edge: indicating that the action intention contained in the candidate instruction is strongly correlated with the interaction with a certain object; instruction-instruction edge: indicating that there is high behavioral similarity or contextual semantic commonality between two candidate instructions (such as sequential operations or synonymous instructions); instruction-context edge: indicating that the current instruction and the user's recent behavioral context are highly adaptable in the meta-learning feature space; object-context edge: indicating that a certain object is frequently operated or paid attention to only in a specific scene state.

[0115] Use multi-layer GCN to propagate structural context semantics and transform the graph structure into an embedding space initialization vector :

[0116]

[0117] Get candidate instructions Graph structure embedding , Represents the last layer of GCN.

[0118] in, Represents a node in the graph The initial embedding vector of Representation node In the The embedding vector of the layer; Representation node The set of neighbor nodes of Indicates the The weights of the layer GCN. represents the normalization coefficient, Represents the activation function. Indicates an instruction node Embedding vector after the Lth layer of GCN.

[0119] Then, the prediction head structure is introduced to embed the vector Input to the fully connected layer and output the predicted instruction label , and then with the real instruction label Compare and calculate the prediction error , set the significant forecast error threshold based on historical experience, when When it is greater than the significant prediction error threshold, it is determined to be an error instruction node.

[0120] The structural risk factors of each candidate instruction are extracted to construct a structural attribution matrix; the structural risk factors include: the semantic ambiguity factor is that the candidate instruction has multiple equal-weight paths in the graph structure or the pointing nodes are discretely distributed, resulting in the semantic concentration being lower than the set threshold; the path conflict factor is that the candidate instruction path has a competitive relationship with other candidate paths, indicating the risk of selection conflict.

[0121] The semantic ambiguity factor Indicates candidate instruction nodes When pointing to multiple objects or semantic nodes, the quantitative result of the semantic concentration degree; get the node The set of all edges of , and extract the weight of each edge , normalize the edge set to a probability distribution , and then use Shannon entropy to calculate uncertainty, and we get ;factor It is a continuous variable and its value range is [0,1].

[0122] The path conflict factor Indicates candidate instruction nodes The degree of overlap between the path and the adjacent candidate paths in the graph structure; define the node The path is ,extract The edge set ; Then according to the figure and There are directly connected instruction nodes to construct neighbor path sets , extract each neighbor path separately The edge set , and calculate with The structural Jaccard similarity between the two is taken as the maximum value of the structural Jaccard similarity. .

[0123] Each candidate instruction node The structural feature vector of Measurable semantic fuzziness factor in graph structure Path conflict factor Composition, defined as:

[0124]

[0125] Each node The attribution vector Its error Multiply together to form an attribution matrix , the formula is:

[0126] in, represents the index of the candidate instruction node, Represents a set of error instruction nodes.

[0127] The structural attribution matrix is ​​not used directly for loss calculation, but is introduced into the meta-learning model parameter update path. In the outer loop, the system uses right The gradient is directional adjusted:

[0128]

[0129] in, Amplify the gradient adjustment weight of the structural risk area, weaken the update direction of highly ambiguous and conflicting paths, and form an instruction embedding space with stronger structural stability and better generalization ability.

[0130] Using Parameters Updated user-specific rules engine Calculate the preference score for each instruction:

[0131]

[0132] in, The output is a continuous value in the range [0,1], which indicates the probability that the user is inclined to execute the instruction in the current situation. The higher the score, the more consistent the instruction is with the user's behavioral preferences in similar historical situations. Represents the candidate instruction preference score.

[0133] Use MLP (Multi-layer Perceptron) to transform the environment state vector Handle embedding of the current environment context state :

[0134]

[0135] Get each candidate instruction Graph structure semantic adaptation score in the current context:

[0136]

[0137] For each candidate instruction, the meta-learning rule score and graph structure semantic adaptation score are integrated:

[0138]

[0139] in, , which can be adjusted dynamically. Indicates candidate instructions The context adaptation score in the current graph structure. Represents the final score of the candidate instructions after fusion.

[0140] Sort the final scores by high to low, and take the top M output environment context candidate instruction sets:

[0141]

[0142] The output layer includes: through the attention mechanism, the historical commands, voice input and environmental context information are finally integrated to generate an accurate set of candidate instructions.

[0143]

[0144] in, Represents a set of candidate instructions. Indicates selecting the first M instructions in descending order of scores; Represents a set of candidate instructions for an environment context.

[0145] During the candidate command generation phase, an innovative multi-layer neural inference mechanism was constructed that integrates meta-learning feature fine-tuning and graph structure context modeling, significantly improving the voice interaction system's ability to respond to personalized behavior patterns and complex 3D environmental changes. On the one hand, through a historical sample guidance mechanism based on the current voice state and syllable interval characteristics, precise fine-tuning of meta-model parameters is achieved, enabling the system to quickly generate a personalized rule engine that meets the current user's preferences. On the other hand, by combining structural semantic propagation in heterogeneous graphs with graph neural network embedding representations, a structure-aware pathway is established between command nodes, target objects, and contextual information, effectively eliminating the prediction offset problem caused by the lack of contextual understanding in traditional static prediction models in multi-semantic, multi-path scenarios. This dual-path fusion mechanism not only improves the accuracy of candidate command generation, but also maintains the generalization stability of the model under conditions of sudden changes in user behavior or environmental switching.

[0146] Furthermore, by introducing a structural attribution mechanism for GCN prediction errors, constructing a structural feature vector of candidate instructions (including semantic ambiguity factors and path conflict factors) and generating an attribution matrix, dynamic adjustment of gradient direction during model training is achieved. This mechanism combines the prediction errors of nodes in the graph with their structural risk factors to minimize the interference of structurally ambiguous regions on the model learning path, thus forming a closed-loop optimization feedback process of "error identification-factor extraction-direction regulation." Compared with existing methods, this mechanism not only effectively alleviates the coupling interference problem between complex instruction paths, but also improves the model's robustness and interpretation ability for regions with low semantic concentration and high path overlap in complex graph structures during the task adaptation phase. Ultimately, it enhances the actual deployment performance of voice interaction systems in highly dynamic virtual environments and the level of human-machine collaborative intelligence.

[0147] Existing methods generally rely on fixed templates or simplified context processing mechanisms, and lack the ability to adapt to the user's real-time pronunciation characteristics and speaking speed rhythm. This leads to problems such as response lag, command recognition deviation, and high repetition rate of candidate predicted commands in dynamically changing 3D virtual scenes. It is also unable to effectively take into account structural factors such as semantic ambiguity and path conflict.

[0148] By introducing a historical sample feature sampling mechanism that integrates speech clarity and rhythmic features, combined with a MAML-based meta-learning outer-loop tuning process, a highly personalized rule-based inference engine is constructed at the initial stage of task initiation. Furthermore, a structural attribution matrix is ​​constructed based on the graph structure risk factors of candidate instructions, guiding the model to avoid high-risk paths during the structural embedding phase, significantly improving generalization prediction capabilities in complex instruction networks. Experimental results show that on a 3D interactive simulation platform, the system outperforms existing models in terms of semantic accuracy, instruction response speed, and structural stability, demonstrating its system adaptability and practical application value in multi-round voice interaction tasks.

[0149] S3: Identify the priority of each candidate instruction, use the highest-priority candidate instruction as the user's next predicted instruction, prioritize loading related resources for the predicted instruction, and quickly respond to user needs.

[0150] The priority of each candidate command is calculated based on the speech clarity score, the execution frequency of historical commands, and the matching degree between the command and the current environment. The formula is expressed as:

[0151] in, It is The comprehensive priority of candidate instructions; is the speech intelligibility score; is the frequency of historical commands; is the relevance of the current 3D environment context; 、 and is the weight coefficient, which is adjusted according to the importance of different factors.

[0152] Calculate the priority of all candidate instructions and sort them by priority. The instruction with the highest priority will be selected as the next predicted instruction. Sorting formula:

[0153]

[0154] in, It is the sorted candidate instruction set, arranged from high to low priority. Indicates the size of the candidate instruction set, that is, a total of Candidate instructions: From the sorted candidate instruction set, select the instruction with the highest priority as the next predicted instruction.

[0155]

[0156] in, Indicates the next predicted instruction, that is, the instruction with the highest priority. It means that after the candidate instruction set is sorted by comprehensive scores, the candidate instruction with the highest score is selected.

[0157] Prioritize loading relevant virtual resources for the next predicted instruction, including 3D models, UI interfaces, sound effects, animations, etc. All these resources will be loaded according to the requirements of the current instruction.

[0158] For example, in a virtual power distribution scenario, the user vaguely issued the command "Turn off that switch" through voice, and the system identified three candidate commands: turning off the main switch of distribution cabinet A1, turning off the backup switch of distribution cabinet B3, and turning off the console screen display.

[0159] For these three candidate commands, the system comprehensively evaluated their corresponding speech clarity scores, historical command execution frequency, and match with the current context. For example, the speech clarity score for turning off the main switch of distribution cabinet A1 was 0.92, with a historical execution frequency of 15 times and a match with the current context of 0.88. The clarity score for turning off the backup switch of distribution cabinet B3 was 0.78, with a frequency of 3 times and a match with 0.65. The clarity score for turning off the console screen display was 0.85, with a frequency of 1 time and a match with 0.40.

[0160] The system calculated the overall priority of each of the three candidate commands, setting weights of 0.4 for voice clarity, 0.3 for historical frequency, and 0.3 for environmental compatibility. Ultimately, turning off the main switch of distribution cabinet A1 scored the highest, and the system prioritized it as the predicted command in the current context, preloading related execution resources to achieve more efficient user command feedback.

[0161] By introducing a priority calculation method based on speech clarity scores, historical command execution frequency, and the relevance of the current 3D environment context, the accuracy of candidate command generation and response has been significantly improved. By assigning different weight coefficients to the importance of each factor, the priority calculation method achieves dynamic adjustment of priorities, enabling the system to better understand and predict user intent. This priority calculation method based on multi-dimensional input makes the selection of candidate commands more intelligent and can adapt to complex usage scenarios.

[0162] Furthermore, by prioritizing the loading of virtual resources related to predicted commands (such as 3D models, UI interfaces, sound effects, and animations), the smoothness and user experience of user interaction are enhanced. Compared to traditional methods, this approach dynamically optimizes the command generation and resource loading process based on real-time user needs, significantly improving interaction efficiency and user experience in virtual environments. This is particularly advantageous in complex 3D environments and multi-tasking scenarios.

[0163] Embodiment 2, an embodiment of the present invention, provides a 3D virtual voice interaction system, including:

[0164] The data module receives the user's real-time voice and performs preprocessing, establishes a speech recognition model, and analyzes the user's pronunciation characteristics and clarity.

[0165] The optimization module optimizes the speech recognition model based on the user's pronunciation characteristics and clarity.

[0166] The prediction module combines the user's historical commands and the current 3D environment context and uses the optimized speech recognition model to generate a set of candidate predicted instructions for the user's next step; the candidate predicted instruction set includes multiple candidate instructions.

[0167] The loading module identifies the priority of each candidate instruction, takes the candidate instruction with the highest priority as the user's next predicted instruction, and prioritizes loading related resources of the predicted instruction to quickly respond to user needs.

[0168] Example 3, an embodiment of the present invention, is different from the previous two embodiments in that:

[0169] If the functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or the portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage media include various media that can store program code, such as USB flash drives, mobile hard drives, read-only memories (ROMs), random access memories (RAMs), magnetic disks, or optical disks.

[0170] The logic and / or steps represented in the flowcharts or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing the logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (e.g., a computer-based system, a system including a processor, or other system that can fetch and execute instructions from an instruction execution system, apparatus, or device). For purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0171] More specific examples (a non-exhaustive list) of computer-readable media include the following: an electrical connection with one or more wires (electronic devices), a portable computer disk cartridge (magnetic devices), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disc read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, and then editing, interpreting, or processing in another suitable manner as necessary, and then storing it in a computer memory.

[0172] It should be understood that various components of the present invention may be implemented using hardware, software, firmware, or a combination thereof. In the above-described embodiments, multiple steps or methods may be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof may be used: a discrete logic circuit having logic gate circuits for implementing logic functions on data signals, an application-specific integrated circuit having suitable combinational logic gate circuits, a programmable gate array (PGA), a field-programmable gate array (FPGA), etc.

[0173] Example 4 is an embodiment of the present invention, which provides a 3D virtual voice interaction method and system. In order to verify the beneficial effects of the present invention, scientific demonstration is carried out through simulation experiments.

[0174] The experiment simulated a typical 3D virtual operation scenario, in which four users (numbered U01~U04) interacted in the virtual power equipment maintenance system and issued commands such as switching, monitoring, and switching.

[0175] During the experimental preparation phase, real-time user voice input during simulated operations was collected. After Wiener filtering and noise reduction, the Mel-Frequency Cepstral Coefficients (MFCCs) were extracted to form a time-frequency feature matrix with a frame length of 25ms and an overlap of 10ms. This matrix was fed into a deep speech recognition model comprising a CNN, maximum pooling, and fully connected architecture. Wavelet transforms were used to extract high and low frequency energy to construct an intelligibility score. A sliding window was also used to analyze speech rate and rhythm, dynamically assess the degree of pronunciation deviation, and establish a dual fault tolerance mechanism for both speech rate and intelligibility.

[0176] The system then calls upon the user's past command sequences, extracts behavioral path states using LSTM, and constructs the current context vector based on real-time speech features. The system utilizes a fine-tuned meta-learning model to output a joint semantic-behavioral embedding for each candidate command, constructing a heterogeneous graph consisting of command nodes, object nodes, and context nodes. The graph structure is propagated through GCN to obtain a graph-structured semantic embedding for each candidate command. The system then incorporates historical user tags to perform attribution modeling of structural errors. Finally, the structural attribution matrix $A$ is used to adjust the meta-model update path, completing the adaptive fine-tuning of the user-specific rule engine.

[0177] The system ultimately outputs a personalized preference score and graph structure semantic adaptation score for each candidate instruction, which are then combined into a comprehensive score. After sorting, the top M instructions are selected as the priority response set. The following are the main scoring results of the four users in the experiment:

[0178] In the experiment, user U01's speech clarity score was 0.64, speech speed feature score was 0.81, historical command matching score was 0.58, graph structure adaptation score was 0.64, personalized preference score was 0.72, and the final comprehensive score was 0.68.

[0179] In the experiment, user U02's speech clarity score was 0.93, speech speed feature score was 0.78, historical command matching score was 0.54, graph structure adaptation score was 0.78, personalized preference score was 0.71, and the final comprehensive score was 0.74.

[0180] In the experiment, user U03's speech clarity score was 0.83, speech speed feature score was 0.73, historical command matching score was 0.59, graph structure adaptation score was 0.69, personalized preference score was 0.75, and the final comprehensive score was 0.72.

[0181] In the experiment, user U04's speech clarity score was 0.86, speech speed feature score was 0.46, historical command matching score was 0.74, graph structure adaptation score was 0.70, personalized preference score was 0.60, and the final comprehensive score was 0.65.

[0182] Analysis of the experimental data shows that user U02's final command received a comprehensive score of 0.74 due to her high speech clarity score (0.93) and good structural fit score (0.78), allowing the system to accurately infer the most appropriate action intention for her current scenario. In contrast, user U04, while achieving a speech clarity score of 0.86, experienced a slight shift in model preference recognition due to her lower speech rate score (0.46), resulting in a final comprehensive score of 0.65, lower than that of other users. This phenomenon demonstrates that in low-speed speech scenarios, the system's speech rate adjustment mechanism can still maintain recognition accuracy, but the response score decreases slightly, verifying that the system's adaptive optimization mechanism is stable but still subject to input anomalies.

[0183] Compared with existing recognition systems based solely on speech or rule templates, the present invention forms a closed-loop logic chain from command generation to prediction feedback by introducing semantic-behavioral joint feature modeling, GCN structure propagation, and structural attribution adjustment paths. This not only improves prediction accuracy, but also enhances fault tolerance for speech anomalies, ambiguous expressions, historical behavior jumps, and other situations. When faced with input with ambiguous speech or abnormal speaking speed, traditional model recognition results often offset or lose key command information. The multiple adjustment mechanisms and graph modeling framework embedded in the present invention can effectively locate structural uncertainties and complete real-time fine-tuning, effectively improving the accuracy of command prediction and the response speed of the interactive system.

[0184] Furthermore, the personalized rule engine of this invention, using only 8 to 16 recent user voice recordings, can achieve rapid adaptation in its inner loop and fine-tune its update path in its outer loop through structural attribution feedback, demonstrating generalization capabilities far exceeding those of traditional speech recognition systems. Overall experimental results demonstrate that this method is particularly suitable for applications requiring rapid response and high-precision recognition in complex 3D virtual scenes, demonstrating strong adaptability, stability, and potential for widespread adoption.

[0185] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. A 3D virtual voice interaction method, characterized in that: include: Receive the user's real-time speech and pre-process it, build a speech recognition model, and analyze the user's pronunciation characteristics and clarity; Optimize the speech recognition model based on the user's pronunciation characteristics and clarity; Combining the user's historical commands and the current 3D environment context, using the optimized speech recognition model to generate a set of candidate predicted instructions for the user's next step; the candidate predicted instruction set includes multiple candidate instructions; Identifying the priority of each candidate command, selecting the highest-priority candidate command as the user's next predicted command, prioritizing the loading of related resources for the predicted command, and quickly responding to user needs; identifying the priority of each candidate command includes calculating the priority of each candidate command based on the speech clarity score, the execution frequency of historical commands, and the matching degree of the command with the current environment; Analyzing the user's pronunciation clarity includes extracting high-frequency coefficients and low-frequency coefficients from the global feature map using a wavelet transform method; Calculating the energy of the high-frequency coefficient and the energy of the low-frequency coefficient; Determine the speech intelligibility score based on the ratio of high-frequency energy to low-frequency energy; Set the clarity score threshold. When the clarity score threshold is reached, the speech clarity is judged to be insufficient and the error tolerance rate is increased; Analyzing the user's pronunciation characteristics includes using the sliding window method to calculate the local energy of the high-frequency detail coefficient, finding the local energy peak, and determining it as the stressed part; calculating the average time interval between adjacent local energy peaks; setting the fast speech rate threshold and the slow speech rate threshold, and when the average time interval is When the speaking speed threshold is high, the speaking speed is judged to be too fast, and the weight of the high-frequency filter in the convolutional neural network is increased to enhance the ability to capture fast-changing syllables; when the average time interval is The slow speech threshold determines the speech speed is slow, increases the weight of the low-frequency filter in the convolutional neural network, and enhances the recognition ability of slow-paced speech.

2. The 3D virtual voice interaction method according to claim 1, wherein: The preprocessing includes using the Wiener filter algorithm to remove the noise of the real-time speech, dividing the real-time speech after denoising into multiple continuous frames of audio, and the overlapping part between each frame is ver; use Mel frequency cepstral coefficients to extract the time-frequency features of each frame of audio and establish a time-frequency feature matrix; where, Indicates the overlap length; The time-frequency features include the spectrum, pitch, volume, fundamental frequency and spectrum flatness of the audio.

3. The 3D virtual voice interaction method according to claim 2, wherein: The speech recognition model includes a convolutional neural network layer, a pooling layer and a fully connected layer; The time-frequency feature matrix is ​​input into the convolutional neural network layer for convolution operation to extract local features and form a local feature map; the pooling layer uses the maximum pooling method to reduce the dimension of the local features to obtain the key local feature map; the fully connected layer converts the local feature map and the key local feature map into feature vectors to construct a global feature map.

4. The 3D virtual voice interaction method according to claim 3, wherein: Generating a set of candidate predicted instructions for the user's next step includes receiving a historical instruction sequence of the user; converting the historical instruction sequence into a time series feature using a long short-term memory network, generating a hidden state of the historical instructions, and recursively updating the hidden state to learn information in the historical instructions to obtain a historical command state; Utilize the optimized speech recognition model to analyze the received real-time speech input and obtain real-time speech features; Build a multi-layer deep model to deduce the user's possible operation intentions layer by layer and generate a set of candidate predicted instructions for the user's next step.

5. The 3D virtual voice interaction method according to claim 4, wherein: The multi-layer deep model includes an LSTM layer, a GCN layer and an output layer; In the LSTM layer, the historical command status and real-time speech features are combined to generate a preliminary candidate command set using LSTM; Combining the historical command state with real-time speech features, and using LSTM to generate a real-time candidate command set; The GCN layer includes pre-training the rule engine offline using a meta-learning framework based on the user's historical command sequence to establish a user-specific rule model. When the user starts voice interaction, based on the current command speech clarity score and the average value of all syllable intervals in the current command, a subsample with the same command speech clarity score and the average value of all syllable intervals in the current command is sampled from the historical interaction data to construct a support set, and the meta-model parameters are quickly fine-tuned to generate a rule engine personalized for the current user. For each candidate command, the support set is used to extract a semantic-behavioral joint feature vector. Constructing a heterogeneous graph based on the semantic-behavioral joint feature vector; The node set for constructing heterogeneous graph includes: instruction node, target object node and context semantic node; let any two nodes i, j in the heterogeneous graph have the corresponding semantic-behavior joint feature vector as and ,when When , an edge is established between i and j, and the weight of the edge is ;in, Represents feature similarity, Indicates the set structural connection threshold.

6. The 3D virtual voice interaction method according to claim 5, wherein: Perform structural propagation on the heterogeneous graph through a multi-layer graph convolutional network to obtain a graph structure embedding representation of the candidate instructions, and output the instruction prediction results through a prediction head structure; compare the prediction results with the actual interaction behavior labels to identify the prediction error; when the prediction error exceeds a set threshold, extract the structural risk factor of the candidate instruction and construct a structural attribution matrix; The structural risk factor includes a semantic ambiguity factor and a path conflict factor; the semantic ambiguity factor is a quantitative indicator of the degree of semantic concentration when a candidate instruction node points to multiple objects or semantic nodes: all edges of the candidate instruction node and their corresponding edge weights are extracted, the edge weight set is normalized into a probability distribution, and the uncertainty degree of the probability distribution is calculated based on Shannon entropy. The obtained entropy value is normalized and used as the value of the semantic ambiguity factor; The path conflict factor represents the degree of overlap between the path where the candidate instruction node is located and the adjacent candidate path in the graph structure on the structural edge set. The path is ,extract The edge set ; According to the figure and There are directly connected instruction nodes to construct neighbor path sets , extract each neighbor path separately The edge set , and calculate with The structural Jaccard similarity between the two is taken as the maximum value of the structural Jaccard similarity. ; The structural attribution matrix is ​​introduced as a regulating factor into the parameter update path of the personalized rule engine. The updated personalized rule engine is used to calculate the personalized preference score of each candidate instruction, guiding the meta-model to avoid high-risk structural areas in subsequent tasks. The personalized preference score and the semantic adaptation score are integrated to generate a comprehensive score for the candidate instructions, which are sorted by score, and the top M candidate instructions are selected as the environmental context candidate instruction set; the semantic adaptation score is the embedding probability value output by the prediction head after the graph convolutional network structure is propagated, which is used to measure the degree of structural semantic fit between the candidate instructions and the target objects and context nodes in the graph; the output layer uses the attention mechanism to weightedly fuse the preliminary candidate instruction set, the real-time candidate instruction set and the environmental context candidate instruction set to generate the user's next step candidate prediction instruction set.

7. A 3D virtual voice interaction system using the method according to any one of claims 1 to 6, characterized in that: The data module receives the user's real-time voice and performs preprocessing, builds a speech recognition model, and analyzes the user's pronunciation characteristics and clarity; The optimization module optimizes the speech recognition model based on the user's pronunciation characteristics and clarity; The prediction module combines the user's historical commands and the current 3D environment context and uses the optimized speech recognition model to generate a set of candidate predicted instructions for the user's next step; the candidate predicted instruction set includes multiple candidate instructions; The loading module identifies the priority of each candidate instruction, takes the candidate instruction with the highest priority as the user's next predicted instruction, and prioritizes loading related resources of the predicted instruction to quickly respond to user needs.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the 3D virtual-based voice interaction method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the 3D virtual voice interaction method according to any one of claims 1 to 6 are implemented.

Images