AR and VR based language control interaction method and system

Abstract

This invention discloses a language control interaction method and system based on AR and VR, relating to the AR and VR fields. It collects user voice commands through a microphone array structure, preprocesses them using a hybrid noise reduction framework, and then performs text conversion and semantic analysis using a modular collaborative architecture natural language processing model to extract interaction intent and related parameters. The model undergoes hierarchical training through a dynamic incremental training system, achieving accurate learning by combining comprehensive interaction complexity. A multi-dimensional feature matching algorithm is used to match with an AR / VR scene database to generate structured control commands to drive virtual scene operations. The system includes modules for voice acquisition and conversion, signal preprocessing, intelligent semantic parsing, scene matching decision-making, and interaction execution. This invention overcomes the limitations of traditional interaction methods, improves the anti-interference capability of voice acquisition, the accuracy of semantic parsing, and the efficiency of scene matching, achieving a natural and efficient AR / VR immersive interactive experience.

Description

Technical Field

[0001] This invention belongs to the field of AR and VR technology, specifically relating to a language control interaction method and system based on AR and VR. Background Technology

[0002] With the increasing application of AR and VR technologies in fields such as gaming, education, healthcare, and industry, users' demand for natural and efficient interaction in virtual environments is becoming more and more urgent.

[0003] Existing AR / VR interaction methods have obvious limitations. Gesture recognition is easily affected by lighting and the regularity of movements, resulting in unstable accuracy. Eye-tracking technology is costly and lacks positioning accuracy in complex scenes. Handheld operation requires holding the device, making the interaction less natural and complex command input cumbersome.

[0004] Language, as the most natural means of human communication, has enormous potential in AR / VR interaction. However, existing language interaction technologies face key bottlenecks: speech acquisition is severely affected by environmental noise, resulting in poor speech signal quality in complex scenarios. Traditional single-microphone or simple array acquisition methods struggle to cope with multi-source noise in AR / VR scenes, leading to speech signal distortion. General natural language processing models lack adaptability to the AR / VR domain, exhibiting low accuracy in intent recognition and parameter extraction for specialized terminology and complex commands. Furthermore, the training process does not categorize corpus complexity, resulting in low model learning efficiency. Existing matching methods often rely on a single feature dimension, failing to consider the correlation between interaction intent, parameter constraints, and scene attributes, leading to cross-scene mismatches and a lack of fuzzy command processing mechanisms. These issues severely restrict the improvement of immersive experiences in AR / VR technology.

[0005] Therefore, there is an urgent need for an innovative language-controlled interaction method and system to solve the above pain points. Summary of the Invention

[0006] This invention aims to solve at least one of the technical problems existing in the prior art; to this end, this invention proposes a language control interaction method and system based on AR and VR, to solve the following technical problem:

[0007] Voice acquisition is severely affected by environmental noise, resulting in poor voice signal quality in complex scenarios. Traditional single-microphone or simple array acquisition methods are unable to cope with multi-source noise in AR / VR scenarios, leading to voice signal distortion. General natural language processing models lack adaptability to the AR / VR domain, with low accuracy in intent recognition and parameter extraction for technical terms and complex commands. Furthermore, the training process does not classify the complexity of the corpus, resulting in low model learning efficiency. Existing matching methods mostly rely on a single feature dimension and do not consider the correlation between interaction intent, parameter constraints, and scene attributes, which easily leads to cross-scene mismatches. The lack of a mechanism for handling fuzzy commands severely restricts the improvement of immersive experiences in AR / VR technology.

[0008] To address the above problems, a first aspect of the present invention provides a language control interaction method based on AR and VR, comprising the following steps:

[0009] S1: Collects the user's voice commands and converts the collected voice signals into electrical signals;

[0010] S2: Preprocess the converted electrical signal, including noise reduction, speech enhancement and endpoint detection;

[0011] S3: A deep learning-based natural language processing model is used to perform text conversion and semantic analysis on the preprocessed speech signal, extracting the user's interaction intent and related parameters. The natural language processing model is trained through AR / VR scene corpus, a dynamic incremental training system is constructed, the comprehensive interaction complexity is calculated and graded, and training is carried out in stages according to the increasing order of complexity level.

[0012] S4: Employ a multi-dimensional feature matching algorithm to perform matching calculations with the pre-stored AR / VR scene database from three dimensions: interaction intent similarity, parameter matching degree, and scene relevance, to determine the corresponding interaction object and operation type, and generate control commands based on the interaction object and operation type;

[0013] S5: Based on the generated control instructions, perform corresponding operations on the interactive objects in the virtual scene.

[0014] Preferably, in step S1, collecting the user's voice commands includes the following steps:

[0015] Voice commands are captured using a microphone array.

[0016] The microphone array contains at least four high-sensitivity microphones with a signal-to-noise ratio of ≥65dB. The microphone array adopts a concentric ring distribution layout with a distance of 2 to 3 centimeters between the centers of adjacent microphones. The ring diameter is adapted to the industry standard wearing size of AR / VR devices. Beamforming technology based on spatial spectrum estimation is used to dynamically focus the user's voice direction.

[0017] Preferably, in step S2, the noise reduction process employs a hybrid noise reduction framework. First, an adaptive RLS filtering algorithm is used to adaptively eliminate periodic noise, with a filtering convergence speed ≤30ms. Then, an improved wavelet thresholding noise reduction algorithm is used to process non-periodic impulse noise, employing a soft-hard threshold tradeoff function based on Bayes estimation. The threshold calculation is specifically as follows:

[0018]

[0019] in, Indicates the threshold. Indicates the standard deviation of noise. Indicates the signal length.

[0020] Preferably, in step S3, the natural language processing model includes the following steps:

[0021] The natural language processing model adopts a modular collaborative architecture, including a speech-to-text sub-model, a consciousness recognition sub-model, and a parameter extraction sub-model.

[0022] The speech-to-text sub-model adopts a Transformer architecture with attention mechanism, which includes 6 layers of encoder and 6 layers of decoder, and the word vector dimension is set to 512 dimensions.

[0023] The consciousness recognition sub-model is constructed using a BERT pre-trained model combined with a bidirectional LSTM network. The LSTM network has a hidden layer dimension of 256, and AR / VR domain-specific word vectors are initialized through transfer learning.

[0024] The parameter extraction sub-model adopts an attention sequence labeling model supported by a CRF layer;

[0025] The three sub-models perform cross-membrane joint learning through a feature sharing layer, which dynamically fuses low-level acoustic features and high-level semantic features, and uses a gated gradient propagation mechanism to optimize the joint training process.

[0026] Among them, AR / VR domain-specific word vectors are obtained by fine-tuning the general domain BERT model through knowledge distillation.

[0027] Preferably, the natural language processing model is trained using AR / VR scene corpus, including the following steps:

[0028] During the training of the natural language processing model using AR / VR scene corpus, a dynamic incremental training system is constructed, and training is carried out in stages according to the increasing order of complexity.

[0029] The dynamic incremental training system divides the corpus into three levels of complexity: Level 1, Level 2, and Level 3, based on the comprehensive interaction complexity.

[0030] The method for constructing the comprehensive interaction complexity is as follows:

[0031] Record the number of operation types involved in each corpus and the total number of all operation types in the current AR / VR scene to construct the operation type complexity; determine the depth of semantic relations through semantic graph analysis and obtain the theoretical maximum value of semantic relation depth in the current AR / VR scene to construct the sentence relation complexity; determine the complexity of the sentence structure of the corpus and the average value of the sentence structure complexity of all corpus in the current AR / VR scene through grammatical number depth to construct the sentence structure complexity.

[0032] The weights corresponding to the complexity of operation type, statement relationship, and statement structure are determined by using the analytic hierarchy process (AHP), and then weighted and summed to obtain the comprehensive interaction complexity.

[0033] Preferably, the comprehensive interaction complexity includes the following steps:

[0034]

[0035] in, Indicates the overall interaction complexity. This indicates the number of operation types involved in a single corpus. This indicates the total number of all operation types in the current AR / VR scene. Indicates the depth of semantic relationships. This represents the theoretical maximum value of the semantic relationship depth in the current AR / VR scene. Indicates the complexity of the sentence structure in the corpus. This represents the average sentence structure complexity of all corpora in the current AR / VR scenario. , and These represent the corresponding weights;

[0036] The K-means clustering algorithm is used to automatically classify the overall interaction complexity into first-level, second-level, and third-level complexities.

[0037] Preferably, the depth of the semantic relationship includes the following steps:

[0038] By performing semantic analysis on the corpus, interactive objects and semantic relationships are extracted, and a semantic network graph is constructed.

[0039] Dijkstra's algorithm is used to calculate the shortest path length between all node pairs in the semantic network graph, and the average of all shortest path lengths is taken as the base path length.

[0040] A semantic relation type complexity scoring table is established, assigning complexity scores to different semantic relation types, and calculating the average complexity of all semantic relations in the corpus to obtain the depth of semantic relations. Specifically:

[0041]

[0042] in, Indicates the depth of semantic relationships. Indicates the base path length. This represents the average complexity of all semantic relations in the corpus. and These represent the corresponding weights.

[0043] Preferably, the complexity of the corpus sentence structure includes the following steps:

[0044] The syntax of the corpus sentences is parsed to generate a syntax tree. The maximum depth of the syntax tree is calculated and standardized to obtain the syntax complexity of the sentences.

[0045] The total number of modifiers in a sentence is counted. The scene relevance of the modifiers is evaluated. The relevance between the modifiers and the core vocabulary of the scene is calculated by word vector similarity. The relevance is used as a dynamic coefficient of the modifiers. The number of weighted modifiers is calculated and standardized to obtain the sentence modification complexity. The standardization adopts a dynamic threshold, which is updated on a rolling basis according to the maximum value of the number of weighted modifiers in nearly 300 scene corpora.

[0046] The nested structure in the statement is identified, and the nested structure is divided into simple nesting, medium nesting and deep nesting according to the scenario complexity level. Each nested structure corresponds to a different scenario complexity coefficient. The maximum nesting level is counted, and after multiplying it by the corresponding scenario complexity coefficient, it is standardized to obtain the statement nesting complexity.

[0047] By determining the corresponding weights using the analytic hierarchy process (AHP), the complexity of the sentence structure is obtained by weighting and adding the sentence syntax complexity, sentence modification complexity, and sentence nesting complexity.

[0048] Preferably, step S4 includes the following steps:

[0049] A multi-dimensional feature matching algorithm is adopted to perform matching calculations from three dimensions: interaction intent similarity, parameter matching degree, and scene relevance.

[0050] The interaction intent similarity is calculated by using a cosine similarity algorithm to determine the degree of similarity between the extracted interaction intent and a predefined intent template in the database.

[0051] The parameter matching degree is calculated by measuring the deviation rate between the extracted parameters and the attribute parameters of the interactive object;

[0052] The scene relevance is calculated based on the hierarchical relationship between the current scene label and the scene label to which the interactive object belongs;

[0053] If all three dimensions match successfully, the corresponding interaction object and operation type are directly determined; if any dimension fails to match, the fuzzy matching optimization mechanism is activated.

[0054] Furthermore, when generating control instructions, a structured instruction generation framework is adopted, and the instructions include object identifier fields, operation type fields, and parameter value fields.

[0055] A second aspect of the present invention provides a voice control interaction system based on AR and VR, comprising the following modules: a voice acquisition and conversion module: acquiring the user's voice commands and converting the acquired voice signals into electrical signals;

[0056] Signal preprocessing module: preprocesses the converted electrical signal, including noise reduction, speech enhancement and endpoint detection;

[0057] Intelligent semantic parsing module: It uses a deep learning-based natural language processing model to perform text conversion and semantic analysis on the preprocessed speech signal, extracting the user's interaction intent and related parameters. The natural language processing model is trained through AR / VR scene corpus, constructs a dynamic incremental training system, calculates the comprehensive interaction complexity and classifies it, and trains in stages according to the increasing order of complexity level.

[0058] Scene matching decision module: It adopts a multi-dimensional feature matching algorithm to perform matching calculations with the pre-stored AR / VR scene database from three dimensions: interaction intent similarity, parameter matching degree and scene relevance, to determine the corresponding interaction object and operation type, and generate control instructions based on the interaction object and operation type;

[0059] Interactive execution module: Based on the generated control commands, it performs corresponding operations on interactive objects in the virtual scene.

[0060] The beneficial effects of this invention are:

[0061] This invention improves the accuracy of speech recognition by using a concentric microphone array combined with beamforming technology and a hybrid noise reduction framework, enabling stable acquisition of clear speech signals even in noisy environments and solving the problem of speech acquisition in complex scenarios.

[0062] This invention constructs a modular collaborative architecture natural language processing model, utilizes AR / VR-specific word vectors and a dynamic incremental training system. Simultaneously, the dynamic incremental training system supports dynamic updates of comprehensive interaction complexity, and the scene database and matching algorithm can be adapted to different AR / VR application scenarios, thereby improving the accuracy of complex command intent recognition and the completeness of parameter extraction, and enhancing professional domain adaptability.

[0063] This invention uses a multi-dimensional feature matching algorithm combined with a fuzzy matching optimization mechanism to greatly improve the matching rate in three dimensions: similarity of interaction intent, matching degree of parameters, and relevance of scene, effectively avoiding cross-scene mismatch. Attached Figure Description

[0064] Figure 1 This is a schematic diagram of the method flow of the present invention;

[0065] Figure 2 This is a schematic diagram of the module flow of the present invention. Detailed Implementation

[0066] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0067] Please see Figure 1 As shown, this invention is a language control interaction method based on AR and VR, comprising the following steps:

[0068] S1: Collects the user's voice commands and converts the collected voice signals into electrical signals;

[0069] S2: Preprocess the converted electrical signal, including noise reduction, speech enhancement and endpoint detection;

[0070] S3: A deep learning-based natural language processing model is used to perform text conversion and semantic analysis on the preprocessed speech signal, extracting the user's interaction intent and related parameters. The natural language processing model is trained through AR / VR scene corpus, a dynamic incremental training system is constructed, the comprehensive interaction complexity is calculated and graded, and training is carried out in stages according to the increasing order of complexity level.

[0071] S4: Employ a multi-dimensional feature matching algorithm to perform matching calculations with the pre-stored AR / VR scene database from three dimensions: interaction intent similarity, parameter matching degree, and scene relevance, to determine the corresponding interaction object and operation type, and generate control commands based on the interaction object and operation type;

[0072] S5: Based on the generated control instructions, perform corresponding operations on the interactive objects in the virtual scene.

[0073] Specifically, after a user issues a voice command, the device acquires the voice signal through a concentric ring microphone array, converting the sound waves into electrical signals. The array uses beamforming technology to focus on the direction of the voice to improve the clarity of the acquisition. In the preprocessing stage, the electrical signal first undergoes adaptive RLS filtering to eliminate periodic noise, then an improved wavelet thresholding algorithm is used to process impulse noise, and noise enhancement and endpoint detection are performed simultaneously to locate the effective voice segment. The preprocessed signal is input into a natural language processing model, where the Transformer architecture completes the speech-to-text conversion. BERT combined with bidirectional LSTM recognizes the interaction intent, and a CRF sequence labeling model extracts parameters. The model is optimized in stages based on the comprehensive interaction complexity classification results of the corpus. Subsequently, a multi-dimensional algorithm is used to match the scene database to generate structured instructions containing object identifiers, operation types, and parameters, ultimately driving the virtual scene to aggregate interactive objects to perform corresponding operations.

[0074] In one embodiment of the present invention, step S1, which involves collecting the user's voice commands, includes the following steps:

[0075] Voice commands are captured using a microphone array.

[0076] The microphone array contains at least four high-sensitivity microphones with a signal-to-noise ratio of ≥65dB. The microphone array adopts a concentric ring distribution layout with a distance of 2 to 3 centimeters between the centers of adjacent microphones. The ring diameter is adapted to the industry standard wearing size of AR / VR devices. Beamforming technology based on spatial spectrum estimation is used to dynamically focus the user's voice direction.

[0077] Specifically, the microphone array synchronously acquires spatial acoustic signals in real time, converts the electrical signals received by each microphone into digital signals and inputs them into the signal processing unit. The multi-signal classification algorithm is used to estimate the spatial spectrum of the multi-channel signals, calculate the signal power spectral density at different azimuth angles, locate the direction of the speech signal, and dynamically adjust the beamforming weight matrix according to the estimated sound source azimuth angle. Phase compensation and amplitude weighting enhancement are performed on the speech signal in the target direction, while a suppression beam is formed for noise signals in non-target directions. The azimuth change of the speech signal is monitored in real time, and the spatial spectrum estimation results are updated every 10ms. The weight matrix is ​​dynamically optimized to ensure that the beam is always focused on the user's speech direction.

[0078] In one embodiment of the present invention, in step S2, the noise reduction process adopts a hybrid noise reduction framework. First, the periodic noise is adaptively eliminated using an adaptive RLS filtering algorithm, with a filtering convergence speed ≤30ms. Then, the non-periodic impulse noise is processed using an improved wavelet threshold noise reduction algorithm, employing a soft-hard threshold trade-off function based on Bayes estimation. The threshold calculation is specifically as follows:

[0079]

[0080] in, Indicates the threshold. Indicates the standard deviation of noise. Indicates the signal length.

[0081] Specifically, the preprocessed electrical signal is input into the adaptive RLS filtering module to initialize the filter weight vector and covariance matrix. Based on the speech signal sampling points, the error signal is calculated in real time and the filter coefficients are updated. The periodic noise characteristics are dynamically tracked through the recursive least squares algorithm, and the filtering convergence is completed within ≤30ms, adaptively eliminating the periodic noise components. Wavelet transform is performed on the RLS-filtered signal to decompose it into wavelet coefficients of different scales. The noise standard deviation is calculated through power spectrum analysis of the speech silence segment. The threshold is calculated based on the threshold calculation formula. The wavelet coefficients are thresholded using a soft and hard threshold trade-off function based on Bayes estimation to retain the effective speech coefficients and suppress the impulse noise coefficients. The denoised speech signal is reconstructed through inverse wavelet transform.

[0082] In one embodiment of the present invention, step S3 of the natural language processing model includes the following steps:

[0083] The natural language processing model adopts a modular collaborative architecture, including a speech-to-text sub-model, a consciousness recognition sub-model, and a parameter extraction sub-model.

[0084] The speech-to-text sub-model adopts a Transformer architecture with attention mechanism, which includes 6 layers of encoder and 6 layers of decoder, and the word vector dimension is set to 512 dimensions.

[0085] The consciousness recognition sub-model is constructed using a BERT pre-trained model combined with a bidirectional LSTM network. The LSTM network has a hidden layer dimension of 256, and AR / VR domain-specific word vectors are initialized through transfer learning.

[0086] The parameter extraction sub-model adopts an attention sequence labeling model supported by a CRF layer;

[0087] The three sub-models perform cross-membrane joint learning through a feature sharing layer, which dynamically fuses low-level acoustic features and high-level semantic features, and uses a gated gradient propagation mechanism to optimize the joint training process.

[0088] Among them, AR / VR domain-specific word vectors are obtained by fine-tuning the general domain BERT model through knowledge distillation.

[0089] Specifically, a modular collaborative architecture is constructed, loading three sub-models: speech-to-text, intent recognition, and parameter extraction. The speech-to-text sub-model initializes a Transformer architecture with an attention mechanism, configuring a 6-layer encoder and a 6-layer decoder, with word vector dimensions set to 512 and randomly initialized. Using a knowledge distillation method, a general-domain BERT model is used as the teacher model, and domain-specific word vectors are generated by fine-tuning using AR / VR scene corpora. These vectors are then used to initialize the word embedding layer of the intent recognition sub-model. This sub-model loads a pre-trained BERT model and connects to a bidirectional LSTM network with 256 hidden layers. The parameter extraction sub-model uses an attention sequence labeling network, superimposing a CRF layer on the output layer and initializing the attention weight matrix and transition probability matrix. A feature sharing layer is enabled to dynamically fuse the low-level acoustic features output by the speech-to-text sub-model with the high-level semantic features generated by the intent recognition sub-model. A gated gradient propagation mechanism is adopted to assign gradient update weights to the three sub-models. The joint loss function (speech-to-text loss + intent recognition loss + parameter extraction loss) is optimized through backpropagation and iteratively trained until the model converges. The preprocessed speech signal is input into the model, the speech-to-text sub-model outputs a text sequence, the intent recognition sub-model outputs interactive intent labels based on the text sequence, and the parameter extraction sub-model obtains the key parameter set through CRF layer decoding to complete semantic parsing.

[0090] In one embodiment of the present invention, the natural language processing model is trained using AR / VR scene corpus, including the following steps:

[0091] During the training of the natural language processing model using AR / VR scene corpus, a dynamic incremental training system is constructed, and training is carried out in stages according to the increasing order of complexity.

[0092] The dynamic incremental training system divides the corpus into three levels of complexity: Level 1, Level 2, and Level 3, based on the comprehensive interaction complexity.

[0093] The method for constructing the comprehensive interaction complexity is as follows:

[0094] Record the number of operation types involved in each corpus and the total number of all operation types in the current AR / VR scene to construct the operation type complexity; determine the depth of semantic relations through semantic graph analysis and obtain the theoretical maximum value of semantic relation depth in the current AR / VR scene to construct the sentence relation complexity; determine the complexity of the sentence structure of the corpus and the average value of the sentence structure complexity of all corpus in the current AR / VR scene through grammatical number depth to construct the sentence structure complexity.

[0095] The weights corresponding to the complexity of operation type, statement relationship, and statement structure are determined by using the analytic hierarchy process (AHP), and then weighted and summed to obtain the comprehensive interaction complexity.

[0096] Specifically, the complexity of operation types is obtained by counting the number of operation types designed for each corpus and dividing it by the total number of operation types in the current scenario. Interaction objects and semantic relationships are extracted using semantic graph analysis tools to construct a semantic network graph. The basic path length is calculated using Dijkstra's algorithm, and the semantic relationship depth is obtained by combining it with a semantic complexity scoring table. This depth is then divided by the theoretical maximum depth of the scenario to obtain the sentence relationship complexity. Syntactic parsing is performed on the corpus to generate a syntax tree. The grammatical, modification, and nesting complexity of the standardized sentences is calculated, and a weighted average is used to obtain the complexity of the corpus sentence structure. This weighted average is then divided by the average sentence structure complexity of the scenario to obtain the sentence structure complexity. Finally, weights are determined using hierarchical analysis and applied accordingly. The overall interaction complexity is calculated by summing the results. The phased training process includes: Phase 1: Training the model using a Level 1 complexity corpus, focusing on optimizing basic interaction intent recognition and simple parameter extraction, iterating for 50-100 rounds; Phase 2: Introducing a Level 2 complexity corpus, enabling a feature sharing layer to enhance mid-level semantic feature learning, adjusting the learning rate to 0.5 times the initial value, and training for 100-200 rounds; Phase 3: Importing a Level 3 complexity corpus, enabling a gated gradient propagation mechanism to focus on optimizing complex semantic relationship parsing, dynamically adjusting the number of training rounds (≥200 rounds) based on the validation set accuracy, and stopping training when the model's average parsing accuracy across all levels of corpus is ≥95%.

[0097] In one embodiment of the present invention, the comprehensive interaction complexity includes the following steps:

[0098]

[0099] in, Indicates the overall interaction complexity. This indicates the number of operation types involved in a single corpus. This indicates the total number of all operation types in the current AR / VR scene. Indicates the depth of semantic relationships. This represents the theoretical maximum value of the semantic relationship depth in the current AR / VR scene. Indicates the complexity of the sentence structure in the corpus. This represents the average sentence structure complexity of all corpora in the current AR / VR scenario. , and These represent the corresponding weights;

[0100] The K-means clustering algorithm is used to automatically classify the overall interaction complexity into first-level, second-level, and third-level complexities.

[0101] Specifically, comprehensive interaction complexity is a comprehensive indicator for measuring the complexity of the corpus. This indicates the number of operation types involved in a single corpus. Operation types include object movement, rotation, scaling, and multi-object combination operations, etc. The more operation types there are, the more complex the user interaction behaviors involved in the corpus are, and the higher the requirements for the model's understanding and processing capabilities. This represents the total number of all operation types in the current AR / VR scene. This is a fixed value based on the scene settings. For example, in a specific game scene, if a total of 10 basic operation types are defined, then when calculating all corpora in that scene, the total number of operation types is... The value is 10. The ratio of the two values ​​can be used to measure the proportion of the richness of the operation type of this corpus in the whole scenario. The depth of semantic relationships is represented by semantic graph analysis. For example, in the simple instruction "pick up an apple," the semantic relationship between the apple and the action of picking it up is relatively simple and direct, leading to... The value is low, while the instruction "place the red apple next to the blue box and open the box" involves two objects, the apple and the box, as well as operations such as placement and opening. There are complex spatial and semantic relationships between them, leading to... The value is relatively high. The semantic relationship depth is calculated based on a comprehensive evaluation of the shortest path length between nodes in the semantic network and the semantic relationship complexity. This represents the theoretical maximum value of semantic relationship depth in the current AR / VR scenario. The ratio of the two values ​​reflects the relative level of semantic relationship complexity of the corpus under extreme scenario conditions. The complexity of sentence structure in a corpus is measured using a multi-dimensional approach, considering factors such as syntax tree depth, the number of modifiers, and the number of nested sentence layers. For example, the simple declarative sentence "open the door" has a simple grammatical structure, leading to... The value is low, while the complex sentence structure "After the flashing red button in the upper right corner of the screen is clicked, the blue square that appears in the center of the field of vision slowly moves to the left edge of the green triangle" contains multiple modifiers, conditional adverbs, and complex action descriptions, resulting in a complex grammatical structure. The value is relatively high. This represents the average sentence structure complexity of all corpora in the current AR / VR scene. It is obtained by statistically calculating the sentence structure complexity of a large number of corpora. The ratio of the two is used to reflect the degree of deviation of the sentence structure complexity of the corpus from the average level. , and These represent the corresponding weights, with values ​​of 0.4, 0.3, and 0.3 respectively.

[0102] In one embodiment of the present invention, the depth of the semantic relationship includes the following steps:

[0103] By performing semantic analysis on the corpus, interactive objects and semantic relationships are extracted, and a semantic network graph is constructed.

[0104] Dijkstra's algorithm is used to calculate the shortest path length between all node pairs in the semantic network graph, and the average of all shortest path lengths is taken as the base path length.

[0105] A semantic relation type complexity scoring table is established, assigning complexity scores to different semantic relation types, and calculating the average complexity of all semantic relations in the corpus to obtain the depth of semantic relations. Specifically:

[0106]

[0107] in, Indicates the depth of semantic relationships. Indicates the base path length. This represents the average complexity of all semantic relations in the corpus. and These represent the corresponding weights.

[0108] Specifically, a natural language semantic parsing tool is used to perform structured analysis on the corpus text, extracting interactive objects (such as "red apple" and "blue box") and semantic relationships (such as "placed on", "open", and "located next to"). Based on the extraction results, a semantic network graph is constructed, using nodes to label interactive objects and operation commands, and labeled edges to label the semantic relationship types between nodes. A semantic relationship type complexity scoring table is established based on the interactive characteristics of AR / VR scenes. Basic action relationships (such as move and display) are assigned 1 point, spatial relationships (such as located next to and superimposed on) are assigned 2 points, logical relationships (such as conditional triggering and sequential order) are assigned 3 points, and multi-object collaborative relationships are assigned 4 points. The relationship types of all edges in the semantic network graph are traversed, and corresponding scores are assigned according to the scoring table. The theoretical maximum value is calculated by obtaining the set of all interactive objects in the current AR / VR scene, counting the number of objects, simulating the longest semantic relationship link containing all objects based on scene interaction rules, calculating the sum of the shortest path lengths between nodes in this link, and taking the maximum value of the semantic relationship type complexity in the link. The theoretical maximum value calculation formula is as follows: ,in, This represents the total shortest path length between the nodes in the link. Indicates the number of objects. This represents the maximum value of the semantic relationship type complexity in the link; where, and These represent the corresponding weights, with values ​​of 0.6 and 0.4 respectively.

[0109] In one embodiment of the present invention, the complexity of the corpus sentence structure includes the following steps:

[0110] The syntax of the corpus sentences is parsed to generate a syntax tree. The maximum depth of the syntax tree is calculated and standardized to obtain the syntax complexity of the sentences.

[0111] The total number of modifiers in a sentence is counted. The scene relevance of the modifiers is evaluated. The relevance between the modifiers and the core vocabulary of the scene is calculated by word vector similarity. The relevance is used as a dynamic coefficient of the modifiers. The number of weighted modifiers is calculated and standardized to obtain the sentence modification complexity. The standardization adopts a dynamic threshold, which is updated on a rolling basis according to the maximum value of the number of weighted modifiers in nearly 300 scene corpora.

[0112] The nested structure in the statement is identified, and the nested structure is divided into simple nesting, medium nesting and deep nesting according to the scenario complexity level. Each nested structure corresponds to a different scenario complexity coefficient. The maximum nesting level is counted, and after multiplying it by the corresponding scenario complexity coefficient, it is standardized to obtain the statement nesting complexity.

[0113] By determining the corresponding weights using the analytic hierarchy process (AHP), the complexity of the sentence structure is obtained by weighting and adding the sentence syntax complexity, sentence modification complexity, and sentence nesting complexity.

[0114] Specifically, a syntax parsing tool is used to perform deep syntax analysis on the corpus sentences, generating a corresponding syntax tree structure. The syntax tree nodes are traversed, and the path length from the root node to the deepest leaf node is calculated as the maximum depth. This path is then standardized to obtain the sentence's syntactic complexity. The standardization formula is as follows: ,in, Indicates the syntactic complexity of a statement. Indicates the maximum depth. This represents the maximum historical depth of the syntax tree in this type of scenario; a core vocabulary library for AR / VR scenarios is constructed, storing core vocabulary according to scenario type (e.g., game scenario, education scenario) (e.g., in a game scenario, core vocabulary includes: attack, level, launch, etc.). A pre-trained word vector model (Word2Vec or BERT word vectors) is used to calculate the semantic similarity between the extracted modifier text and the corresponding words in the core vocabulary library of the scenario, obtaining the relevance score r. If a modifier is associated with multiple scenario core words, the highest similarity score is taken as the final relevance score of the modifier. Based on the modifier type and relevance score, a weighted formula is used to calculate the total number of weighted modifiers, specifically: Among them, the types of modifiers include attributive, adverbial, and complement, with corresponding basic weights of 0.3, 0.4, and 0.3, respectively. , and These respectively indicate the number of attributive, adverbial, and complement phrases in a sentence. The relevance is represented by 0.5, 0.6, and 0.4, which represent the corresponding relevance adjustment coefficients. Among them, the adjustment coefficient for adverbs is the highest, reflecting their highest complexity in relation to the scene. Simultaneously, the total weighted modifier count values ​​of nearly 300 expected modifiers in the same scene are recorded in real time, forming a sliding window dataset. The maximum value is extracted from the sliding window dataset as the current dynamic threshold, and the threshold is updated by removing the oldest corpus entry for each new entry. A standardized formula is used, specifically: ,in, Indicates the complexity of statement modifiers. Indicates the total weighted number of modified components. This represents the maximum value extracted from the sliding window dataset. When identifying nested structures, a nesting type scenario complexity grading system is used, classifying nested structures into basic nesting (e.g., simple clauses), medium nesting (e.g., condition + action nesting), and deep nesting (e.g., multiple conditions + multiple object nesting), corresponding to scenario complexity coefficients of 1.0, 1.5, and 2.0 respectively, resulting in: ,in, Indicates the weighted nesting level. Indicates the maximum nesting level. This represents the complexity coefficient of the corresponding scenario. The standardization specifically refers to: ,in, Indicates the complexity of nested statements. This represents the maximum weighted nesting level of all statements in the corpus for this type of scenario; the weights are determined by the analytic hierarchy process, with the weights corresponding to statement syntax complexity, statement modification complexity, and statement nesting complexity being 0.3, 0.35, and 0.35, respectively.

[0115] In one embodiment of the present invention, step S4 includes the following steps:

[0116] A multi-dimensional feature matching algorithm is adopted to perform matching calculations from three dimensions: interaction intent similarity, parameter matching degree, and scene relevance.

[0117] The interaction intent similarity is calculated by using a cosine similarity algorithm to determine the degree of similarity between the extracted interaction intent and a predefined intent template in the database.

[0118] The parameter matching degree is calculated by measuring the deviation rate between the extracted parameters and the attribute parameters of the interactive object;

[0119] The scene relevance is calculated based on the hierarchical relationship between the current scene label and the scene label to which the interactive object belongs;

[0120] If all three dimensions match successfully, the corresponding interaction object and operation type are directly determined; if any dimension fails to match, the fuzzy matching optimization mechanism is activated.

[0121] Furthermore, a structured instruction generation framework is employed when generating control commands. Each command includes an object identifier field, an operation type field, and a parameter value field. Specifically, user interaction intent vectors are extracted from the semantic parsing results output in step S3 and compared with predefined intent template vectors in the AR / VR scene database. These predefined intent template vectors in the AR / VR scene database are structured encodings of typical user interaction needs. By abstracting and modeling common AR / VR scene operations, each intent is decomposed into core semantic features. Word2Vec or BERT word vector technology is used to convert these semantic features into fixed-dimensional numerical vectors. The cosine similarity algorithm is then used to calculate the cosine of the angle between the vectors. ,in, Indicates the similarity of interaction intents. This represents the extracted interaction intent vector. This indicates that the database pre-defines intent template vectors, and a successful match is achieved when the interaction intent similarity is ≥0.85. The parameter value set extracted in step S3 is obtained and compared with the attribute parameter range of candidate interaction objects in the scene database to calculate the deviation rate, i.e., the parameter matching degree. Specifically: ,in, Indicates the parameter matching degree. This indicates that parameter values ​​are being extracted. This represents the standard value of the interactive object's attributes. It is a preset range or value of attribute parameters for virtual objects in the scene database, used to quantitatively evaluate parameter matching. For example, the "standard value of rotation angle" for a virtual cube can be defined as a 0° baseline value, allowing a ±30° functional tolerance range, providing a reference benchmark for deviation rate calculation. A successful match is achieved when the parameter matching degree is <0.15. The hierarchical structure of the current AR / VR scene tag (e.g., industrial assembly scene - robotic arm sub-scene) and the scene tags of candidate interactive objects is analyzed (a multi-level classification system for AR / VR virtual scenes and objects, organized in a tree structure; the top level is the scene category, such as industrial scene, game scene, and education scene; the second level is the sub-scene type, such as assembly scene and repair scene under industrial scene; the third level is the object category, such as mechanical parts under assembly scene; and the fourth level is the specific object identifier, such as gear A under machine parts. Each interactive object is bound to a unique hierarchical tag path in the database). The correlation is calculated using a hierarchical distance weighted method, specifically: ,in, Indicates the relevance of the scene. The scene tag hierarchy distance is a quantitative indicator that measures the degree of correlation between the current scene and the scene to which the interactive object belongs in the hierarchical structure. It is calculated based on the tag paths of both, comparing each level starting from the top-level tag and counting the hierarchy difference up to the first different tag node. The smaller the hierarchy distance, the higher the scene correlation. A successful match is achieved when the hierarchy distance is ≤2 levels and the correlation is ≥0.7. If the interaction intent similarity, parameter matching degree, and scene correlation all meet the corresponding preset thresholds, the unique identifier and operation type code of the corresponding interactive object are retrieved from the scene database. If any dimension fails to meet the threshold, a fuzzy matching optimization mechanism is activated: combining the semantic confidence score generated in step S3, historical interactions are retrieved. The data undergoes similarity retrieval, and matching deviations are corrected through contextual semantic completion. The three-dimensional matching values ​​are recalculated until the threshold is met or a manual intervention prompt is triggered. Simultaneously, a standardized instruction framework is used to generate control instructions, including: an object identifier field: encoded in UUID format, uniquely corresponding to the interactive object in the virtual scene; an operation type field: encoded in 8-bit numbers, corresponding to the predefined operation type in the database (e.g., 00100203 represents "robotic arm rotation operation"); and a parameter value field: the specific parameters after scene attribute range verification, in the format "parameter name: value: unit". After the instructions are generated, they are sent to the interaction execution module via an encrypted transmission protocol to drive the virtual scene operation execution.

[0122] Please see Figure 2 As shown, this invention is a language control interaction system based on AR and VR, comprising the following modules:

[0123] Voice acquisition and conversion module: Acquires user voice commands and converts the acquired voice signals into electrical signals;

[0124] Signal preprocessing module: preprocesses the converted electrical signal, including noise reduction, speech enhancement and endpoint detection;

[0125] Intelligent semantic parsing module: It uses a deep learning-based natural language processing model to perform text conversion and semantic analysis on the preprocessed speech signal, extracting the user's interaction intent and related parameters. The natural language processing model is trained through AR / VR scene corpus, constructs a dynamic incremental training system, calculates the comprehensive interaction complexity and classifies it, and trains in stages according to the increasing order of complexity level.

[0126] Scene matching decision module: It adopts a multi-dimensional feature matching algorithm to perform matching calculations with the pre-stored AR / VR scene database from three dimensions: interaction intent similarity, parameter matching degree and scene relevance, to determine the corresponding interaction object and operation type, and generate control instructions based on the interaction object and operation type;

[0127] Interactive execution module: Based on the generated control commands, it performs corresponding operations on interactive objects in the virtual scene.

[0128] The above embodiments are only used to illustrate the technical methods of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical methods of the present invention without departing from the spirit and scope of the technical methods of the present invention.

Claims

1. A language control interaction method based on AR and VR, characterized in that, Includes the following steps: S1: Collects the user's voice commands and converts the collected voice signals into electrical signals; S2: Preprocess the converted electrical signal, including noise reduction, speech enhancement and endpoint detection; S3: A deep learning-based natural language processing model is used to perform text conversion and semantic analysis on the preprocessed speech signal, extracting the user's interaction intent and related parameters. The natural language processing model is trained through AR / VR scene corpus, a dynamic incremental training system is constructed, the comprehensive interaction complexity is calculated and graded, and training is carried out in stages according to the increasing order of complexity level. S4: Employ a multi-dimensional feature matching algorithm to perform matching calculations with the pre-stored AR / VR scene database from three dimensions: interaction intent similarity, parameter matching degree, and scene relevance, to determine the corresponding interaction object and operation type, and generate control commands based on the interaction object and operation type; S5: Based on the generated control instructions, perform corresponding operations on the interactive objects in the virtual scene.

2. The language control interaction method based on AR and VR according to claim 1, characterized in that, In step S1, collecting the user's voice commands includes the following steps: Voice commands are captured using a microphone array. The microphone array contains at least four high-sensitivity microphones with a signal-to-noise ratio of ≥65dB. The microphone array adopts a concentric ring distribution layout with a distance of 2 to 3 centimeters between the centers of adjacent microphones. The ring diameter is adapted to the industry standard wearing size of AR / VR devices. Beamforming technology based on spatial spectrum estimation is used to dynamically focus the user's voice direction.

3. The language control interaction method based on AR and VR according to claim 1, characterized in that, In step S2, the noise reduction process employs a hybrid noise reduction framework. First, an adaptive RLS filtering algorithm is used to adaptively eliminate periodic noise, with a convergence speed ≤30ms. Then, an improved wavelet thresholding noise reduction algorithm is used to process non-periodic impulse noise, employing a soft-hard threshold tradeoff function based on Bayes estimation. The threshold calculation is as follows: in, Indicates the threshold. Indicates the standard deviation of noise. Indicates the signal length.

4. The language control interaction method based on AR and VR according to claim 1, characterized in that, In step S3, the natural language processing model includes the following steps: The natural language processing model adopts a modular collaborative architecture, including a speech-to-text sub-model, a consciousness recognition sub-model, and a parameter extraction sub-model. The speech-to-text sub-model adopts a Transformer architecture with attention mechanism, which includes 6 layers of encoder and 6 layers of decoder, and the word vector dimension is set to 512 dimensions. The consciousness recognition sub-model is constructed using a BERT pre-trained model combined with a bidirectional LSTM network. The LSTM network has a hidden layer dimension of 256, and AR / VR domain-specific word vectors are initialized through transfer learning. The parameter extraction sub-model adopts an attention sequence labeling model supported by a CRF layer; The three sub-models perform cross-membrane joint learning through a feature sharing layer, which dynamically fuses low-level acoustic features and high-level semantic features, and uses a gated gradient propagation mechanism to optimize the joint training process. Among them, AR / VR domain-specific word vectors are obtained by fine-tuning the general domain BERT model through knowledge distillation.

5. The language control interaction method based on AR and VR according to claim 1, characterized in that, The natural language processing model is trained using AR / VR scene corpora, including the following steps: During the training of the natural language processing model using AR / VR scene corpus, a dynamic incremental training system is constructed, and training is carried out in stages according to the increasing order of complexity. The dynamic incremental training system divides the corpus into three levels of complexity: Level 1, Level 2, and Level 3, based on the comprehensive interaction complexity. The method for constructing the comprehensive interaction complexity is as follows: Record the number of operation types involved in each corpus and the total number of all operation types in the current AR / VR scene to construct the operation type complexity; determine the depth of semantic relations through semantic graph analysis and obtain the theoretical maximum value of semantic relation depth in the current AR / VR scene to construct the sentence relation complexity; determine the complexity of the sentence structure of the corpus and the average value of the sentence structure complexity of all corpus in the current AR / VR scene through grammatical number depth to construct the sentence structure complexity. The weights corresponding to the complexity of operation type, statement relationship, and statement structure are determined by using the analytic hierarchy process (AHP), and then weighted and summed to obtain the comprehensive interaction complexity.

6. The language control interaction method based on AR and VR according to claim 5, characterized in that, The overall interaction complexity includes the following steps: in, Indicates the overall interaction complexity. This indicates the number of operation types involved in a single corpus. This indicates the total number of all operation types in the current AR / VR scene. Indicates the depth of semantic relationships. This represents the theoretical maximum value of the semantic relationship depth in the current AR / VR scene. Indicates the complexity of the sentence structure in the corpus. This represents the average sentence structure complexity of all corpora in the current AR / VR scenario. , and These represent the corresponding weights; The K-means clustering algorithm is used to automatically classify the overall interaction complexity into first-level, second-level, and third-level complexities.

7. The language control interaction method based on AR and VR according to claim 6, characterized in that, The depth of the semantic relationship includes the following steps: By performing semantic analysis on the corpus, interactive objects and semantic relationships are extracted, and a semantic network graph is constructed. Dijkstra's algorithm is used to calculate the shortest path length between all node pairs in the semantic network graph, and the average of all shortest path lengths is taken as the base path length. A semantic relation type complexity scoring table is established, assigning complexity scores to different semantic relation types, and calculating the average complexity of all semantic relations in the corpus to obtain the depth of semantic relations. Specifically: in, Indicates the depth of semantic relationships. Indicates the base path length. This represents the average complexity of all semantic relations in the corpus. and These represent the corresponding weights.

8. The language control interaction method based on AR and VR according to claim 6, characterized in that, The complexity of the corpus's sentence structure includes the following steps: The syntax of the corpus sentences is parsed to generate a syntax tree. The maximum depth of the syntax tree is calculated and standardized to obtain the syntax complexity of the sentences. The total number of modifiers in a sentence is counted. The scene relevance of the modifiers is evaluated. The relevance between the modifiers and the core vocabulary of the scene is calculated by word vector similarity. The relevance is used as a dynamic coefficient of the modifiers. The number of weighted modifiers is calculated and standardized to obtain the sentence modification complexity. The standardization adopts a dynamic threshold, which is updated on a rolling basis according to the maximum value of the number of weighted modifiers in nearly 300 scene corpora. The nested structure in the statement is identified, and the nested structure is divided into simple nesting, medium nesting and deep nesting according to the nesting type scenario complexity level. Each nested structure corresponds to a different scenario complexity coefficient. The maximum nesting level is counted, and after multiplying it by the corresponding scenario complexity coefficient, it is standardized to obtain the statement nesting complexity. By determining the corresponding weights using the analytic hierarchy process (AHP), the complexity of the sentence structure is obtained by weighting and adding the sentence syntax complexity, sentence modification complexity, and sentence nesting complexity.

9. The language control interaction method based on AR and VR according to claim 1, characterized in that, Step S4 includes the following steps: A multi-dimensional feature matching algorithm is adopted to perform matching calculations from three dimensions: interaction intent similarity, parameter matching degree, and scene relevance. The interaction intent similarity is calculated by using a cosine similarity algorithm to determine the degree of similarity between the extracted interaction intent and a predefined intent template in the database. The parameter matching degree is calculated by measuring the deviation rate between the extracted parameters and the attribute parameters of the interactive object; The scene relevance is calculated based on the hierarchical relationship between the current scene label and the scene label to which the interactive object belongs; If all three dimensions match successfully, the corresponding interaction object and operation type are directly determined; if any dimension fails to match, the fuzzy matching optimization mechanism is activated. Furthermore, when generating control instructions, a structured instruction generation framework is adopted, and the instructions include object identifier fields, operation type fields, and parameter value fields.

10. A language control interaction system based on AR and VR, characterized in that, Includes the following modules: Voice acquisition and conversion module: Acquires user voice commands and converts the acquired voice signals into electrical signals; Signal preprocessing module: preprocesses the converted electrical signal, including noise reduction, speech enhancement and endpoint detection; Intelligent semantic parsing module: It uses a deep learning-based natural language processing model to perform text conversion and semantic analysis on the preprocessed speech signal, extracting the user's interaction intent and related parameters. The natural language processing model is trained through AR / VR scene corpus, constructs a dynamic incremental training system, calculates the comprehensive interaction complexity and classifies it, and trains in stages according to the increasing order of complexity level. Scene matching decision module: It adopts a multi-dimensional feature matching algorithm to perform matching calculations with the pre-stored AR / VR scene database from three dimensions: interaction intent similarity, parameter matching degree and scene relevance, to determine the corresponding interaction object and operation type, and generate control instructions based on the interaction object and operation type; Interactive execution module: Based on the generated control commands, it performs corresponding operations on interactive objects in the virtual scene.

Images