Virtual reality scene simulation analysis method and system based on ai large model and speech recognition

CN123030072APending Publication Date: 2026-11-03BEIJING SHENSIDE TECHNOLOGY DEVELOPMENT CO LTD
View PDF -1 Cites 0 Cited by

Patent Information

Application Number
CN202611192886.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-08-07
Publication Date
2026-11-03

AI Technical Summary

Technical Problem

Existing VR voice interaction technologies have weak recognition and anti-interference capabilities in complex acoustic environments, semantic and spatial separation, uncontrollable inference latency, easy command execution conflicts, and poor adaptability to complex interactions, making it difficult to achieve high accuracy and immersive experience in scenarios such as cultural and creative exhibitions and art exchanges.

Method used

By collecting speech signals and three-dimensional spatial information, acoustic environment perception judgment is performed, and the temporal peak threshold is adaptively adjusted to enhance the speech. The enhanced speech and the multimodal context of the virtual scene are input into the AI ​​big model for joint semantic reasoning, generating scene control commands and performing rationality verification to ensure that the speech and spatial logic match.

Benefits of technology

It improves the purity of speech recognition and the accuracy of spoken intent recognition, balances reasoning accuracy and real-time interaction, avoids spatial logic conflicts, and enhances the rationality and stability of virtual scene manipulation.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123030072A_ABST
    Figure CN123030072A_ABST
Patent Text Reader

Abstract

The application discloses a virtual reality scene simulation analysis method and system based on an AI large model and voice recognition, belongs to the technical field of scene simulation analysis, and comprises the following steps: voice front-end processing, semantic understanding and space reasoning, instruction generation, hierarchical evaluation and rationality verification.The application collects voice signals and three-dimensional space information, performs voice enhancement again through acoustic environment perception to judge whether to perform time domain peak value linkage amplitude threshold adaptive adjustment, inputs the enhanced voice and a virtual scene multi-modal context into a large model for joint semantic reasoning, outputs corrected recognition text and intention analysis results, maps the space information to a multi-modal feature alignment space for fusion, generates scene control instructions and evaluates the complexity of the instructions in stages, performs rationality verification on operation objects and behaviors according to three-dimensional space geometric information before execution, and generates feedback signals after the verification.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of scene simulation and analysis technology, and in particular to a virtual reality scene simulation and analysis method and system based on AI large model and speech recognition. Background Technology

[0002] Virtual reality scene simulation technology based on AI (Artificial Intelligence) large models and speech recognition has become a research hotspot in the field of human-computer interaction. Combining the semantic understanding and generation capabilities of large language models, automatic speech recognition technology, and the immersive environment of virtual reality provides users with a highly natural and intelligent scene simulation interactive experience.

[0003] Currently, virtual reality (VR) scene simulations often rely on voice interaction for human-computer control. However, existing technologies have numerous shortcomings across the entire chain, making it difficult to balance interaction accuracy, response speed, multimodal collaboration effects, and practical applicability. Traditional VR voice interaction employs a separate pipeline architecture, with speech recognition, semantic parsing, and scene rendering modules operating independently. Speech recognition only converts audio to text and cannot incorporate virtual space information to correct semantics, leading to a continuous accumulation of recognition errors along the chain. The complex acoustic conditions of VR, such as enclosed sound fields, device noise, and reverberation from multi-person conversations, weaken the anti-interference capabilities of conventional speech recognition. It also exhibits low accuracy in recognizing accents, creative terminology, and omitted speech, making it unsuitable for free-flowing spoken interaction in scenarios like cultural and creative exhibitions and art exchanges. Currently, most systems rely on rule-based scripts or small natural language models to parse speech, only recognizing preset commands and unable to leverage large AI models to retain long dialogue context, break down complex commands, and deduce fuzzy intentions. When users issue continuous commands based on virtual space location and gaze information, the system struggles to integrate multi-dimensional information to determine needs, resulting in a continuous decline in the semantic stability of long dialogues and limiting free control of the virtual scene. The computational bottleneck of large AI models makes it difficult to balance inference accuracy and real-time performance. Cloud-based calls are susceptible to network issues, resulting in high latency that exceeds VR anti-motion sickness response standards, leading to a disconnect between visual actions and voice commands. Local lightweight deployment weakens semantic parsing capabilities, hindering widespread adoption in scenarios such as electronics exhibitions and digital creative displays. The separate processing of various modal data across channels lacks feature alignment mechanisms, and voice commands cannot be validated using spatial visual information, easily leading to erroneous executions that violate scene logic. The timing of various feedback mechanisms (sound, light, touch) is also misaligned, reducing the immersive experience. Most interactive systems on the market are designed for gaming and industrial scenarios, but they suffer from functional redundancy, high hardware costs, and significant modification difficulties, resulting in poor adaptability to business scenarios such as advertising production, exhibition services, and electronics retail. Cloud transmission also poses a risk of interactive data leakage, failing to meet the data compliance requirements of enterprise consulting and market research. Summary of the Invention

[0004] In view of this, embodiments of the present invention provide a virtual reality scene simulation and analysis method and system based on AI large model and speech recognition. It can collect speech signals and three-dimensional spatial information, determine whether to perform temporal peak linkage amplitude threshold adaptive adjustment through acoustic environment perception, and then perform speech enhancement; input the enhanced speech and virtual scene multimodal context into the large model for joint semantic reasoning, and output corrected recognition text and intent parsing results; map spatial information to multimodal feature alignment space for fusion, generate scene control commands and evaluate their complexity in a hierarchical manner; before execution, perform rationality verification on the operation object and behavior based on three-dimensional spatial geometric information, and execute and generate feedback signals after the verification is passed.

[0005] On the one hand, a virtual reality scene simulation and analysis method based on AI large-scale model and speech recognition is provided. This method includes: Step S1, acquiring the user's original speech signal and three-dimensional spatial geometric information, performing multi-channel acoustic environment perception processing on the original speech signal, determining whether to perform temporal peak linkage amplitude threshold adaptive adjustment; if so, performing speech enhancement processing on the original speech signal after adjustment; otherwise, directly performing speech enhancement processing on the original speech signal to obtain an enhanced speech signal; Step S2, acquiring the current virtual scene multimodal context information, inputting the enhanced speech signal and the virtual scene multimodal context information together into a pre-trained AI large-scale language model, performing joint semantic reasoning, and outputting corrected recognition text and user information fused with virtual scene spatial information. Intent parsing results; Step S3, map the three-dimensional spatial geometric information to the multimodal feature alignment space for cross-modal feature alignment and fusion to obtain a multimodal fusion feature vector. Based on the multimodal fusion feature vector and the corrected recognition text and user intent parsing results, generate scene control instructions that match the spatial logic of the virtual reality scene. Perform complexity classification evaluation on the scene control instructions to determine whether the scene control instructions are simple atomic instructions or composite instructions; Step S4, generate feedback signals based on the execution results of the scene control instructions. Before executing the scene control instructions, perform a rationality check on the operation object and operation behavior pointed to by the scene control instructions based on the three-dimensional spatial geometric information of the virtual scene to determine whether the operation behavior conforms to the spatial logic of the virtual scene.

[0006] On the other hand, a virtual reality scene simulation and analysis system based on AI large-scale models and speech recognition is provided. This system includes: a speech front-end processing module, a semantic understanding and spatial reasoning module, an instruction generation and hierarchical evaluation module, and a rationality verification module. The speech front-end processing module collects the user's original speech signal and three-dimensional spatial geometric information, performs multi-channel acoustic environment perception processing on the original speech signal, and determines whether to perform time-domain peak-to-amplitude threshold adaptive adjustment. If so, speech enhancement processing is performed on the original speech signal after adjustment; otherwise, speech enhancement processing is directly performed on the original speech signal to obtain an enhanced speech signal. The semantic understanding and spatial reasoning module obtains the multimodal context information of the current virtual scene and inputs the enhanced speech signal and the multimodal context information of the virtual scene into a pre-trained AI large-scale language model for joint semantic reasoning. The system outputs corrected and recognized text and user intent parsing results that fuse virtual scene spatial information; a command generation and hierarchical evaluation module is used to map 3D spatial geometric information to a multimodal feature alignment space for cross-modal feature alignment and fusion, obtaining a multimodal fusion feature vector. Based on the multimodal fusion feature vector, corrected and recognized text, and user intent parsing results, it generates scene control commands that match the spatial logic of the virtual reality scene, performs a complexity hierarchical evaluation of the scene control commands, and determines whether the scene control commands are simple atomic commands or composite commands; a rationality verification module is used to generate feedback signals based on the execution results of the scene control commands. Before executing the scene control commands, it performs rationality verification on the operation object and operation behavior pointed to by the scene control commands based on the 3D spatial geometric information of the virtual scene, and determines whether the operation behavior conforms to the spatial logic of the virtual scene.

[0007] Beneficial effects The beneficial effects of the technical solutions provided in the embodiments of the present invention include at least the following: By employing an integrated technical architecture encompassing end-to-end acoustic adaptive optimization, multimodal spatial semantic fusion, hierarchical distributed inference of instructions, and multi-level spatial logic verification, this invention effectively addresses industry pain points in traditional VR voice interaction recognition, such as weak anti-interference capabilities, semantic and spatial separation, uncontrollable inference latency, easy command execution conflicts, and poor adaptability to complex interactions. This invention relies on a temporal peak-linked amplitude threshold adaptive adjustment mechanism to dynamically update the human voice judgment threshold based on complex acoustic environments such as VR enclosed sound fields, device noise, and multi-person reverberation. This accurately filters out sudden noises and stable background noise, significantly improving the purity of voice enhancement and recognition in complex scenarios. By linking enhanced voice with multimodal contextual information such as VR 3D pose, gaze point, scene state, and historical interactions, and inputting it into a large AI model, this invention utilizes a cross-modal attention alignment mechanism to complete low-confidence text spatial semantic correction. This achieves deep integration of voice semantics and virtual scene spatial logic, completely eliminating the problem of separation between literal transcription and scene understanding in traditional technologies, and significantly improving the accuracy and disambiguation capability of spoken intent recognition. Meanwhile, this invention achieves precise hierarchical classification of control commands by quantifying multi-dimensional command features. Simple atomic commands are inferred locally using lightweight methods, while complex composite commands are decomposed and processed in a distributed cloud environment. Combined with dynamic frame rate smoothing, this effectively balances inference accuracy and real-time interaction, strictly controlling interaction latency and ensuring VR audio-visual synchronization and an immersive experience. Furthermore, this invention constructs a four-dimensional verification system encompassing spatial feasibility, physical support constraints, motion collision interference, and state compatibility. This system can comprehensively identify unauthorized control commands and output rectification prompts, avoiding spatial logic conflicts and invalid or abnormal operations, thereby improving the rationality, stability, and security of virtual scene control. Attached Figure Description

[0008] Figure 1 A flowchart of a virtual reality scene simulation and analysis method based on AI large model and speech recognition provided in this application embodiment; Figure 2 A flowchart illustrating the adaptive adjustment of the time-domain peak linkage amplitude threshold of the virtual reality scene simulation analysis method based on AI large model and speech recognition provided in the embodiments of this application; Figure 3 This is a schematic diagram of the structure of a virtual reality scene simulation and analysis system based on AI large model and speech recognition provided in an embodiment of this application. Detailed Implementation

[0009] To make the technical problems, technical solutions and advantages of the present invention clearer, a detailed description will be given below in conjunction with the accompanying drawings and specific embodiments.

[0010] like Figure 1 The diagram shown is a flowchart of a virtual reality scene simulation and analysis method based on AI large model and speech recognition provided in an embodiment of this application. The method includes the following steps: Step S1: Acquire the user's original speech signal and three-dimensional spatial geometric information. Perform multi-channel acoustic environment perception processing on the original speech signal and determine whether to perform time-domain peak linkage amplitude threshold adaptive adjustment. If yes, perform speech enhancement processing on the original speech signal after adjustment; otherwise, directly perform speech enhancement processing on the original speech signal to obtain an enhanced speech signal. The original speech signal is acquired through the microphone array built into the virtual reality head-mounted display device. The acoustic environment types include at least a closed sound field environment, a device noise environment, a multi-person reverberation environment, and combinations thereof. The three-dimensional spatial geometric information includes head pose data, eye tracking data, and the virtual scene.

[0011] It should be understood that, such as Figure 2 The flowchart shown below illustrates the specific process of the adaptive adjustment of temporal peak linkage amplitude threshold in the virtual reality scene simulation analysis method based on AI large model and speech recognition provided in this application embodiment. The process involves: performing frame segmentation on the original speech and extracting the amplitude peak of each frame; calculating the mean, standard deviation, and maximum value of the peak values ​​using a sliding window; subsequently, performing a lower threshold check by comparing the difference between the lower threshold and the previous effective lower threshold. If the difference exceeds the maximum step size, the lower threshold is updated step-by-step along the reference direction; otherwise, the reference value is directly used as the current threshold. The effective lower threshold is determined by the following steps: After the lower threshold is processed, the upper threshold is checked using the same logic. Based on whether the difference between the upper and lower thresholds exceeds the maximum step size, the upper threshold is updated by step or the baseline upper threshold is directly assigned. After obtaining the final effective upper and lower thresholds, the amplitude range of the speech signal is determined. When the signal amplitude is between the upper and lower limits, the effective human voice is retained. If the amplitude is lower than the lower limit, silence and stable noise are suppressed. If the amplitude is higher than the upper limit, sudden impact noise is filtered out. The three processing paths are finally combined to output the processed pure human voice. The whole process relies on threshold step limits to avoid threshold jump distortion and relies on interval processing to achieve noise differentiation and removal and human voice preservation.

[0012] It needs to be explained that the specific process for determining whether to perform adaptive adjustment of the time-domain peak amplitude threshold is as follows: The acquired raw speech signal is processed in the time domain based on the preset frame length and frame shift. The temporal amplitude envelope of each analysis frame is extracted, and the temporal amplitude peak value of the analysis frame is determined. Construct a sliding time window buffer to store the temporal amplitude peaks of consecutive historical analysis frames in time sequence, forming a temporal amplitude peak sequence, and calculate the statistical distribution parameters of the temporal amplitude peak sequence. The statistical distribution parameters include at least the peak mean and peak standard deviation. Based on statistical distribution parameters, an adaptive dual-threshold threshold interval is dynamically constructed, which includes an effective lower threshold and an effective upper threshold. The effective lower threshold is determined based on the weighted difference between the mean of the peak values ​​and the standard deviation of the peak values. ,in The first weighting coefficient is preset, and the effective lower threshold is used to distinguish human voice signals from stable background noise; the effective upper threshold is determined based on the tracked value of the maximum amplitude peak within the sliding time window buffer; that is... ,in The second weighting coefficient is preset and 0 < <1, the effective upper limit threshold is used to suppress sudden, large-amplitude non-human sound impact interference; Set the maximum step size for a single adjustment, and compare the baseline effective lower limit threshold and baseline effective upper limit threshold calculated in the current analysis frame with the corresponding thresholds that actually took effect in the previous time step, specifically: If the absolute value of the difference between the baseline effective lower limit threshold and the effective lower limit threshold that actually took effect at the previous moment is greater than the maximum step size, then it is determined that the current lower limit threshold has changed. The effective lower limit threshold that actually took effect at the current moment is updated to the effective lower limit threshold that actually took effect at the previous moment plus or minus the maximum step size, and the update direction is in the direction of the baseline effective lower limit threshold.

[0013] The comparison is made with the corresponding threshold that actually took effect at the previous time step, and also includes: If the absolute value of the difference between the baseline effective lower limit threshold and the effective lower limit threshold that actually took effect at the previous moment is less than or equal to the maximum step size, then the effective lower limit threshold that actually took effect at the current moment will be updated to the baseline effective lower limit threshold. If the absolute value of the difference between the baseline effective upper limit threshold and the effective upper limit threshold that actually took effect at the previous moment is greater than the maximum step size, it is determined that the current upper limit threshold has changed. The effective upper limit threshold that actually took effect at the current moment is updated to the effective value that actually took effect at the previous moment plus or minus the maximum step size, and the update direction is in the direction of the baseline effective upper limit threshold. If the absolute value of the difference between the baseline effective upper limit threshold and the effective upper limit threshold that actually took effect at the previous moment is less than or equal to the maximum step size, then the effective upper limit threshold that actually took effect at the current moment will be updated to the baseline effective upper limit threshold. Based on the final effective lower threshold and effective upper threshold, the amplitude threshold of the time domain signal of the current analysis frame is determined, and the signal components whose time domain amplitude is within the closed interval formed by the effective lower threshold and the effective upper threshold are determined as effective human voice components and retained. Low-amplitude components with time-domain amplitudes below the effective lower limit threshold are identified as silent segments or stable noise components and are suppressed or removed. High-amplitude components with time-domain amplitudes exceeding the effective upper threshold are identified as sudden impact noise components and clipped or filtered out, thereby dynamically extracting pure and effective human voice time-domain signals.

[0014] In this embodiment, addressing the technical problems of unstable signal-to-noise ratio, poor adaptability of fixed human voice thresholds, and frequent noise misjudgments and omissions caused by complex and non-stationary acoustic environments such as enclosed sound fields, equipment fan noise, and reverberation from multi-person conversations in VR virtual reality scenarios, this invention employs a time-domain peak-linked amplitude threshold adaptive adjustment mechanism to achieve dynamic and accurate updates of the effective amplitude threshold range for human voices. The specific implementation process is as follows: First, the raw speech signal collected by the VR headset microphone array is subjected to standardized time-domain frame segmentation preprocessing. A fixed frame length and frame shift are uniformly set to complete continuous signal segmentation, and each analysis frame is extracted frame by frame. The corresponding temporal amplitude envelope curve is obtained, and the maximum vibration amplitude in each frame of the temporal waveform is accurately located, i.e., the peak value of the temporal amplitude, which is used as the core criterion for judging the strength of human voice in a single frame. Furthermore, a fixed-duration sliding time window buffer is constructed to perform temporal buffering on the peak values ​​of multiple consecutive frames of temporal amplitude acquired in real time, forming a stable temporal amplitude peak sequence. The two core statistical distribution parameters of the peak sequence, the peak mean and the peak standard deviation, are solved through statistical operations. Based on these two parameters, the overall noise floor and human voice amplitude fluctuation characteristics of the current VR scene are quantified, providing data support for adaptive threshold calculation. In this embodiment, the adaptive dual-threshold threshold range includes an effective lower threshold for distinguishing human voices from stable background noise, and an effective upper threshold for suppressing sudden large-amplitude impact interference. The effective lower threshold is calculated using a mean-standard deviation weighted algorithm, with the formula Tlow=μ-α×σ, where μ is the mean peak value within the sliding window, σ is the peak standard deviation, and α is a preset first weighting coefficient. By weighting and offsetting the amplitude base of stable environmental noise, weak and effective human voices are accurately filtered out. The effective upper threshold is calculated using a real-time peak tracking weighted method, with the formula Thigh=Pmax×β, where Pmax is the maximum amplitude peak tracking value within the sliding time window buffer, and β is a preset second weighting coefficient with a value between 0 and 1. This effectively suppresses instantaneous large-amplitude non-human voice interference such as touch noise, current pulses, and sudden sound field disturbances.To avoid speech signal truncation distortion and intermittent voice loss caused by excessively large threshold jumps in a single iteration, this embodiment pre-configures a maximum step size constraint for single threshold adjustment. The baseline effective lower threshold and baseline effective upper threshold calculated in the current frame are compared and verified one by one with the corresponding thresholds actually effective in the previous time step, completing a smooth iterative update of the thresholds. For lower threshold iteration, when the absolute difference between the baseline effective lower threshold and the effective lower threshold in the previous time step is greater than the preset maximum step size, it is determined that the current lower threshold has an abnormal jump. The system does not directly apply the baseline threshold, but instead... Using the previous effective threshold as a benchmark, the maximum step size is increased or decreased in the direction approaching the benchmark threshold to complete a smooth update. When the absolute difference between the two is less than or equal to the maximum step size, the benchmark effective lower limit threshold is directly assigned as the current effective lower limit threshold, ensuring accurate threshold adaptation under small environmental fluctuations. For the upper limit threshold iteration, the same step size constraint logic is adopted. When the absolute difference between the benchmark effective upper limit threshold and the previous effective upper limit threshold exceeds the maximum step size, a step-smoothing correction is performed; otherwise, it is directly updated to the benchmark effective upper limit threshold, completely avoiding the voice frame error filtering problem caused by threshold mutation. After completing the smooth iterative correction of the upper and lower thresholds and determining the adaptive amplitude threshold range that will ultimately take effect for the current frame, the system performs precise amplitude threshold judgment and signal filtering processing on the temporal speech signal of the current analysis frame. Signal components whose temporal amplitude values ​​are strictly within the closed interval formed by the effective lower threshold and the effective upper threshold are judged as pure and effective human voice components and preserved completely for subsequent speech recognition and semantic parsing. Low-amplitude signal components with temporal amplitude values ​​below the effective lower threshold are uniformly judged as silent intervals or stable environmental background noise components, and are suppressed, attenuated, or frame-level clipped to eliminate the interference of continuous background noise on recognition accuracy. High-amplitude abrupt signal components with temporal amplitude values ​​above the effective upper threshold are judged as sudden impact noise, and targeted clipping, filtering, and interference removal processing are performed. Finally, without damaging the effective human voice details, the system adaptively adapts to the dynamic and complex acoustic environment of VR and dynamically extracts high-purity and high-robust effective human voice temporal signals.

[0015] Step S2: Obtain the multimodal context information of the current virtual scene, input the enhanced speech signal and the multimodal context information of the virtual scene into the pre-trained AI large language model, perform joint semantic reasoning, and output the corrected recognition text and user intent parsing results that integrate the spatial information of the virtual scene.

[0016] It should be further explained that the specific process for joint semantic reasoning is as follows: The enhanced speech signal is input into the acoustic encoder and language decoder of a pre-trained multilingual speech recognition model to perform speech-to-text decoding processing and output the initial recognized text sequence and its corresponding word-level confidence score and sentence-level confidence score. Based on the word-level confidence score, low-confidence words in the initial recognition text sequence are marked. Low-confidence words are those whose word-level confidence scores are lower than the preset confidence threshold. Obtain the virtual scene multimodal context information of the current virtual reality scene. The virtual scene multimodal context information includes the scene state information of the virtual reality scene, the spatial position information of virtual objects, the coordinates of the user's gaze point, and the historical dialogue context information. The multimodal context information of the virtual scene is structured and encoded according to a preset encoding template to form a scene context embedding vector in a unified format. The encoding template includes: scene semantic slots, which describe the global semantic category of the current virtual scene; object spatial relationship slots, which describe the relative spatial position between virtual objects and their relative pose relationship with the user; gaze focus slots, which represent the target virtual object corresponding to the user's gaze point and its three-dimensional coordinates; and interaction history slots, which store the state-action pair sequence of the most recent N rounds of dialogue history. Construct multimodal joint reasoning prompts that integrate initial recognition text with multimodal contextual information from the virtual scene.

[0017] Multimodal joint reasoning cues include: System role instructions are used to assign the role of the AI ​​large language model as an agent for semantic understanding and spatial reasoning in virtual reality scenes; The speech recognition candidate text field is used to carry the initial recognition text sequence; The spatial context description field is used to describe the specific content of each slot in the scene context embedding vector in natural language. The low confidence correction indicator field is used to indicate the specific location of the low confidence word and the correction constraints. The correction constraints are the semantic fluency constraints and spatial rationality constraints of the low confidence word in the virtual scene context.

[0018] After word embedding and position encoding of the multimodal joint reasoning prompts, a large language model input sequence is generated; The input sequence of the large language model is fed into the pre-trained AI large language model, which contains at least a multi-layer Transformer decoder structure. In each layer of the Transformer decoder, cross-modal semantic alignment is performed, specifically as follows: By using a self-attention mechanism to model the global contextual dependency of each element in the input sequence, the semantic relationship between speech recognition candidate text and scene context information is captured. By using a cross-attention mechanism, attention weights are calculated for each word in the candidate text field of speech recognition and each spatial entity in the spatial context description field, forming a speech-space cross-modal attention alignment matrix; Based on the speech-space cross-modal attention alignment matrix, spatial semantic correction is performed on each low-confidence word position.

[0019] The specific process for spatial semantic correction of each low-confidence word position is as follows: Based on the distribution of attention weights between low-confidence words and virtual objects and spatial locations, candidate words with the highest attention weights and semantic fluency that meet preset conditions are selected from the latent space of the AI ​​large language model to replace low-confidence words and generate corrected recognition text. The AI-powered large language model performs joint semantic reasoning based on the corrected recognized text and scene context embedding vectors, specifically: The action predicates and action parameters involved in the text are parsed, corrected and identified. The action predicates represent the type of operation that the user expects to perform, and the action parameters include at least the target object identifier and spatial operation parameters. The spatial logic consistency of the action parameters and the virtual object spatial location information and gaze focus slot information embedded in the scene context vector is checked to determine whether the target object pointed to by the action parameters exists in the current virtual scene and whether the operation type is executable in spatial geometry. Based on the results of spatial logical consistency verification and historical dialogue context information, semantic disambiguation and intent completion are performed on the corrected and identified text, and the user intent parsing results are output. The output layer of the AI ​​large language model is structured and decoded to generate structured output data containing corrected recognized text and user intent parsing results; The user intent parsing result should include at least the intent category identifier, the target object identifier, the list of operation parameters, and the operation execution priority.

[0020] In this embodiment, the high-purity enhanced speech signal, after adaptive threshold adjustment and noise reduction enhancement processing, is first input into a pre-trained multilingual speech recognition model. This model is an end-to-end multilingual speech recognition model optimized for VR immersive interactive scenarios. It consists of a lightweight convolutional acoustic encoder, a multi-head attention conversion module, and a multilingual semantic decoder. It supports mixed reading scenarios in Mandarin, multiple dialects, and common foreign languages, and is adaptable to complex interactive scenarios involving multiple languages ​​and accents, such as cultural and creative exhibitions, cross-border displays, and electronic product sales. Unlike traditional monolingual recognition models, this multilingual speech recognition model utilizes a massive amount of multilingual spoken language and industry-specific data. Vocabulary and accented speech samples are pre-trained and fine-tuned to effectively adapt to distorted speech features in VR's closed sound field, resulting in stronger acoustic feature extraction and text decoding capabilities. The enhanced speech signal is then passed through the acoustic encoder and language decoder of the multilingual speech recognition model for feature extraction and speech-to-text decoding. The acoustic encoder extracts refined acoustic features from the time and frequency domains, while the language decoder, combined with a multilingual dictionary and spoken grammar model, decodes the sequence, ultimately outputting a complete initial recognition text sequence. Simultaneously, word-level confidence scores for each character and sentence-level confidence scores for the entire sentence are output to quantify the credibility of each recognized element. Subsequently, the system performs word-by-word screening of the initial recognition text sequence based on a preset confidence threshold. Words with confidence scores below the preset threshold are marked as low-confidence words. These low-confidence words commonly exhibit issues such as ambiguity, homophony misidentification, accent misjudgment, and bias in identifying specialized vocabulary, making them the core optimization targets for subsequent semantic correction. Simultaneously, the system synchronously collects a complete set of multimodal context information in the current virtual reality scene in real time. This includes the overall scene state information of the virtual scene, the three-dimensional spatial position and posture information of each virtual exhibit and model, the three-dimensional coordinate information of the user's real-time gaze point, and the historical dialogue context and interaction behavior information of multiple rounds prior to the current interaction, providing comprehensive scene prior support for semantic disambiguation and spatial reasoning. To achieve the unified fusion of heterogeneous multimodal data, this invention structures and encodes the above-mentioned scattered multimodal context information according to a preset standardized encoding template, generating a scene context embedding vector with unified dimensions that can be recognized by large models. The encoding template contains four core semantic slots: scene semantic slots for defining the current virtual scene display type and scene operation mode; object spatial relationship slots for describing the relative position of each virtual object and its relative to the user's pose; gaze focus slots for accurately binding the user's gaze target; and interaction history slots for orderly storing historical interaction states and action sequences, thoroughly realizing the structured and regularized spatial information, visual information, and historical interaction information.Based on this, the system constructs multimodal joint reasoning prompts adapted to VR spatial voice interaction. These prompts are structured and contain four core fields: system role instructions to constrain the task identity of the large model, enabling the AI ​​large language model to accurately position itself as a virtual reality scene-specific semantic understanding and spatial reasoning intelligent agent; a speech recognition candidate text field to carry the initial decoding results of the model; a spatial context description field to describe the space and scene information of each slot in structured natural language; and a low-confidence correction indicator field to accurately locate the words to be corrected and limit the correction rules. The correction rules strictly constrain the correction results of low-confidence words to meet the dual conditions of semantic fluency and spatial logical rationality of the virtual scene. After constructing the prompt words, word embedding encoding and positional encoding are performed on the overall prompt word text to transform discrete natural language content into a continuous numerical input sequence for a large model, ensuring that the model accurately captures semantic order and related features. This input sequence is then fed into a pre-trained AI large language model with a multi-layer Transformer decoder structure for deep inference. Within each Transformer decoder layer, a dual-mechanism cross-modal alignment operation using self-attention and cross-attention is performed simultaneously. The self-attention mechanism models the global dependencies of the input sequence, uncovering implicit semantic associations within the text and scene context. The cross-attention mechanism is specifically used to establish a one-to-one correspondence between speech text words and virtual scene spatial entities, spatial coordinates, and interaction states, calculating and generating a speech-space cross-modal attention alignment matrix to accurately quantify the association weight of each word with scene spatial objects. Based on this attention alignment matrix, the system performs intelligent spatial semantic correction on all low-confidence words. Relying on the semantic retrieval capabilities of the large model's hidden space, it selects the optimal candidate words with the highest attention weights that fit the current scene semantics and spatial logic for replacement, eliminating ambiguities caused by acoustic recognition and generating high-precision corrected recognition text. After obtaining the corrected and recognized text, the AI ​​large language model combines structured scene context embedding vectors to conduct deep joint semantic reasoning. It accurately parses the action predicates representing the operation type in the text, as well as action parameters including the operation target object identifier and spatial operation parameters. It actively performs fully automatic spatial logic consistency verification with the current virtual scene object distribution, user gaze focus, and spatial constraints to determine whether the operation target actually exists and whether the operation behavior meets the three-dimensional spatial execution conditions. It also combines historical dialogue context to complete referential resolution, semantic completion, and ambiguity removal, and finally outputs accurate user intent parsing results. Finally, through the structured decoding mechanism of the model output layer, the reasoning results are organized into standardized structured output data, which fully includes intent category identifier, operation target object identifier, refined operation parameter list, and operation execution priority.

[0021] Step S3: Map the three-dimensional spatial geometric information to the multimodal feature alignment space for cross-modal feature alignment and fusion to obtain a multimodal fusion feature vector. Based on the multimodal fusion feature vector and the corrected recognition text and user intent parsing results, generate scene control instructions that match the spatial logic of the virtual reality scene. Perform complexity classification evaluation on the scene control instructions to determine whether the scene control instructions are simple atomic instructions or composite instructions.

[0022] It should be understood that the specific process for evaluating the complexity of scene control commands is as follows: The generated scene control instructions are structured and parsed to extract quantized feature parameters, including the number of action elements, the number of operation targets, the number of spatial constraints, and the temporal dependency depth. Specifically, the number of action elements represents the number of independent action predicates contained in the scene control instructions; the number of operation targets represents the number of different virtual operation objects involved in the scene control instructions; the number of spatial constraints represents the number of sub-conditions (spatial position constraints, directional constraints, or distance constraints) contained in the scene control instructions; and the temporal dependency depth represents the maximum nesting level of the sequential dependencies between multiple sub-actions in the scene control instructions. Based on the quantized feature parameters, a multi-dimensional complexity evaluation vector of the current scene control command is constructed, and the preset weight coefficients corresponding to each evaluation dimension are obtained. The preset weight coefficients include the weight of the number of action elements, the weight of the number of operation targets, the weight of the number of spatial constraints, and the weight of the temporal dependency depth. Based on each evaluation dimension and its corresponding preset weight coefficients, a weighted complexity quantification calculation is performed to obtain a comprehensive complexity score. The correlation between the action predicates in the scene control commands is analyzed, specifically: The specific calculation formula is as follows: ; Among them, the preset weight coefficients of each evaluation dimension are all positive real numbers, and the sum of the preset weight coefficients is normalized to 1. Calculate the semantic similarity between any two action predicates. The semantic similarity is determined based on the vector distance between the action predicates in the preset action semantic space. Based on semantic similarity, an action predicate association graph is constructed. The action predicate association graph uses each action predicate as a node, and the lines connecting nodes with semantic similarity greater than a preset similarity threshold as edges, with the semantic similarity value as the edge weight. Extract the number of connected components in the action predicate association graph. The number of connected components represents the degree of interrelation between action predicates in the scene control command. Obtain historical dialogue context information prior to the current interaction round, and extract the sequence of preceding interaction events associated with the current scene control command from the historical dialogue context information; Based on the referential and causal relationships between the preceding sequence of interactive events and the current scene control command, the context dependency depth of the current scene control command is calculated. The context dependency depth represents the degree to which the current scene control command depends on historical interactive information semantically, and is positively correlated with the number of consecutive interaction rounds in the preceding sequence of interactive events that have referential or causal relationships with the current command.

[0023] The complexity classification assessment of scene control commands also includes: Using the overall complexity score, the number of connected components, and the context dependency depth as input parameters, if the first set of conditions is met simultaneously, the current scene control instruction is determined to be a simple atomic instruction, specifically: The overall complexity score is less than or equal to the preset simple instruction complexity threshold; A number of connected components equal to 1 indicates that each action predicate in the scene control command points to the same operation target or belongs to the same action sequence and cannot be separated. The context dependency depth is less than or equal to the preset shallow context threshold, which indicates that the current scene control commands do not depend on the complex context reasoning of previous interactions. If any condition or any combination of conditions in the second condition group is met, the current scene control command is determined to be a composite command, specifically: The overall complexity score is greater than the simple instruction complexity threshold; A number of connected components greater than 1 indicates that the scene control instructions contain a set of action predicates pointing to different operation targets or independent action paths, and have the space for split execution. If the context dependency depth is greater than the shallow context threshold, it indicates that the current scene control command needs to rely on the context information of multiple rounds of historical dialogue to complete the intent or resolve the reference. When the current scene control command is determined to be a composite command, the scene control command is decomposed into several sub-commands according to the boundaries of each connected component and the temporal dependency depth in the action predicate association graph. Each sub-instruction obtained from the decomposition is labeled with the number of action elements, the number of operation targets, the number of spatial constraints, and the depth of temporal dependencies, and each sub-instruction is assigned an independent sub-instruction identifier and execution priority.

[0024] In this embodiment, the structured scene control instructions generated by the AI ​​large-scale model inference are first subjected to full-domain structured analysis. Four core quantitative feature parameters that can characterize the difficulty and logical complexity of instruction execution are extracted in depth. These parameters include the number of action elements, the number of operation targets, the number of spatial constraints, and the depth of temporal dependencies. The number of action elements is defined as the total number of independent action predicates contained in a single scene control instruction, used to quantify the basic action volume of the instruction. The number of operation targets is defined as the total number of different virtual models, virtual exhibits, scene components, and other operation objects involved in the instruction, used to characterize the scope of instruction operation. The number of spatial constraints is defined as the total number of various sub-constraints such as spatial position restrictions, movement direction restrictions, and distance threshold restrictions attached during instruction execution, used to measure the severity of instruction spatial execution. The depth of temporal dependencies is defined as the maximum number of levels of the sequential order and nested dependencies between multiple sub-actions within the instruction, used to characterize the temporal coupling complexity of actions within the instruction. After extracting all quantified feature parameters, the system constructs a multi-dimensional instruction complexity evaluation vector containing the above four types of parameters, and retrieves the preset weight coefficients corresponding to each dimension, namely, the weight of the number of action elements, the weight of the number of operation targets, the weight of the number of spatial constraints, and the weight of the temporal dependency depth. All weight coefficients are positive real numbers greater than zero and have been normalized to ensure that the sum of all weight coefficients is 1, thereby achieving a balanced quantification and reasonable allocation of the complexity contribution of each dimension. The multi-dimensional features are fused and calculated to obtain the comprehensive complexity score of the current scene's control instructions. This score can comprehensively and accurately quantify the overall execution difficulty of a single instruction. To further explore the correlation logic of actions within the instruction and avoid misclassification caused by a single numerical score, this embodiment further performs semantic correlation analysis on all action predicates within the instruction. By mapping each action predicate to a pre-constructed standardized action semantic space, the semantic similarity between any two action predicates is calculated based on the spatial distance between predicate vectors. The smaller the vector distance, the higher the semantic similarity and the tighter the action correlation. Based on the calculated semantic similarity values, the system constructs a complete action predicate correlation graph. Each independent action predicate is used as a topological node, and nodes with semantic similarity greater than the system's preset similarity threshold are used as connected edges. The corresponding semantic similarity values ​​are assigned as edge weights. Finally, by extracting the number of connected components in the correlation graph, the system accurately represents the degree of aggregation and correlation of each action within the entire instruction. The larger the number of connected components, the more independent and separable action modules the instruction contains, and the higher the overall complexity of the instruction.At the same time, the system retrieves all historical dialogue context information before the current interaction round, filters out the sequence of preceding interaction events that have semantic referential association and causal execution association with the current scene control command, and calculates the context dependency depth of the current command by combining the number of preceding interaction rounds, the nesting relationship of referential association, and the causal dependency level. This parameter is positively correlated with the number of associated historical interaction rounds, which can accurately reflect the degree of dependence of the current command on historical dialogue and historical operation behavior, and effectively characterize the reasoning difficulty of command semantic disambiguation and intent completion. Based on this, the system uses three core judgment dimensions—comprehensive complexity score, number of connected components of action predicate, and context dependency depth—to achieve precise classification and judgment of scene control commands. Specifically, when all three conditions are met simultaneously—comprehensive complexity score less than or equal to a preset simple command complexity threshold, number of connected components equal to 1, and context dependency depth less than or equal to a preset shallow context threshold—the current command is determined to be a simple atomic command. Such commands have a single action, a unique target, no complex spatial constraints, do not rely on historical interaction information, and their overall logic is indivisible; they can be executed quickly directly through the local lightweight inference engine. If a command meets any one or a combination of the following conditions—comprehensive complexity score exceeding the simple command complexity threshold, number of connected components greater than 1, or context dependency depth exceeding the shallow context threshold—it is determined to be a composite command. Such commands generally exhibit characteristics such as multi-action coupling, parallel operation targets, complex spatial constraints, and the need to rely on multiple rounds of historical dialogue to complete referential resolution and intent completion; they cannot be accurately completed through local simple inference. For the complex instructions obtained from the judgment, the system relies on the connected component boundaries and temporal dependency depth hierarchy of the action predicate association graph to intelligently decompose the complex instructions, resulting in several logically independent and temporally ordered refined sub-instructions. At the same time, each decomposed sub-instruction is re-matched and labeled with corresponding action elements, operation targets, spatial constraints, temporal dependency depth and other feature parameters. A unique sub-instruction identifier is assigned to each sub-instruction, and the execution priority is automatically sorted and assigned according to the temporal priority and spatial execution priority of the sub-instructions. This realizes the modular, ordered and hierarchical processing of complex instructions, and provides a precise hierarchical basis for subsequent local lightweight inference and cloud distributed collaborative inference computing power scheduling, task diversion and latency control. It effectively solves the technical problems of computing power waste, insufficient inference accuracy of highly complex instructions, chaotic instruction execution timing and poor immersive interactive experience caused by the unified processing of traditional VR voice interaction instructions.

[0025] Step S4: Based on the execution result of the scene control command, a feedback signal is generated. Before executing the scene control command, the rationality of the operation object and operation behavior pointed to by the scene control command is verified based on the three-dimensional spatial geometric information of the virtual scene to determine whether the operation behavior conforms to the spatial logic of the virtual scene. The feedback signal includes voice feedback signal, virtual scene screen update signal, and haptic feedback drive signal. In response to the generation of the scene control command, the target operation object identifier and the type of operation behavior to be executed are parsed from the scene control command.

[0026] It is important to understand that the specific process for conducting a reasonableness check is as follows: If the rationality check passes, execute the scene manipulation instructions and update the virtual reality scene; If the validity check fails, the scene control command will not be executed, and a prompt message will be displayed to the user. After the virtual reality scene is updated, the scene status information and the spatial location information of virtual objects are updated synchronously.

[0027] In this embodiment, based on the target object identifier, the current spatial pose data of the target object is queried and extracted from the 3D scene database of the current virtual scene. The current spatial pose data includes at least the 3D coordinate position, Euler angles of rotation, and bounding volume dimensions of the target object in the world coordinate system of the current virtual scene. Spatial geometric constraint data of the local environment where the target object is located in the virtual scene is extracted. This spatial geometric constraint data includes at least the ground plane equation, wall plane equation, spatial occupancy mesh of other static obstacles, and the normal vector of the support surface to which the target object is attached. Based on the operation behavior type, a preset operation behavior-spatial condition mapping library is queried to obtain the minimum spatial execution conditions corresponding to the operation behavior type. The minimum spatial execution conditions include at least the minimum operational space radius required for the operation, the operational direction vector requirement, and the minimum safe distance threshold between the target object and other objects during the operation. The current spatial pose data of the target object is compared with the minimum spatial execution conditions, and the following initial judgment logic is executed: If, at the current position of the target object, there is an overlap in three-dimensional space between the bounding volume of the target object and the spatial occupancy grid of the local environment (i.e., the shortest distance between the bounding volume of the target object and the spatial occupancy grid of any static obstacle is less than zero), then the current scene control command is determined to be spatially unreachable, and the initial spatial feasibility judgment result is output as "failed". If the nearest distance to the grid is greater than or equal to zero, and the available operational space radius of the target object's current position is greater than or equal to the minimum operational space radius, then the current scene's control commands are determined to be spatially reachable, and the initial spatial feasibility assessment result is output as "passed." When the initial spatial feasibility assessment result is "passed," further physical support and gravity constraint verification is performed based on the target object's current spatial pose data and support surface normal vector. Based on the target object's bounding volume size parameters and 3D coordinate position, the coordinates of the lowest point of the target object's bounding volume bottom surface in the vertical direction are calculated. Starting from the lowest point coordinates, a ray is emitted vertically downwards, and the intersection points and intersection distances between the ray and each support surface in the local environment are calculated. If the intersection distance... If the distance between the ray and the support surface is less than or equal to the preset maximum support distance threshold, and the angle between the normal vector of the support surface and the vertical upward direction is less than or equal to the preset maximum support tilt angle threshold, then the target object is determined to have effective physical support at the current spatial position, and the physical support verification passes. If the distance between the intersection points is greater than the maximum support distance threshold, or the angle between the normal vector of the support surface and the vertical upward direction is greater than the maximum support tilt angle threshold, or the ray has no intersection with the support surface, then the target object is determined to be suspended or lack effective support at the current spatial position, and the physical support verification fails. When the physical support and gravity constraint verifications pass, the trajectory of the target object from its current spatial pose to its expected spatial pose after the execution of the operation behavior is simulated according to the operation behavior type.Multiple trajectory points are sampled at preset time steps along the simulated motion trajectory. Spatial overlap detection is performed on the bounding body of the target operation object at each trajectory point and the spatial occupancy grid of other virtual objects and static obstacles in the local environment. If spatial overlap is detected between the bounding body of the target operation object and the spatial occupancy grid of other virtual objects or static obstacles at any trajectory point of the motion trajectory, it is determined that a collision interference will occur during the execution of the operation, and the collision interference prediction result is output as "failed". If no spatial overlap is detected at any trajectory point of the motion trajectory, it is determined that there is no collision risk during the execution of the operation, and the collision interference prediction result is output as "passed". The current state attributes of the target operation object are obtained. The current state attributes include at least the current locked / unlocked state, current carrying state, and current connection relationship state of the target operation object. Based on the operation type and the current state attributes of the target operation object, a preset state transition rule base is queried to determine whether the target operation object is allowed to migrate from the current state attribute to the target state attribute after the operation is executed. If there is a legal state transition path from the current state attribute to the target state attribute in the state transition rule base, the operation state compatibility check is determined to be passed. If there is no legal state transition path in the state transition rule base, the operation state compatibility check is passed. If there is a logical conflict between the state transition path or the current state attribute and the operation behavior type, the operation behavior state compatibility check is deemed to have failed. The initial spatial feasibility assessment result, physical support and gravity constraint verification result, collision interference prediction result, and operation behavior state compatibility check result are used as four independent verification dimensions for comprehensive rationality judgment. If all four verification dimensions pass, the operation object and operation behavior pointed to by the scene control command are deemed to fully conform to the three-dimensional spatial geometry and spatial logic of the virtual scene, and the scene control command rationality check is passed, allowing the execution of the scene control command and updating the virtual reality scene. If any one of the four verification dimensions fails, the scene control command is deemed to have a spatial logic conflict with the virtual scene, and the scene control command rationality check fails. When the rationality check fails, a corresponding prompt message is generated based on the type of the failed verification dimension and the current spatial pose data and current state attribute of the target operation object. The prompt message includes at least the failure reason type identifier, a detailed description of the failed dimension, and a suggested alternative operation plan. The prompt message is output to the user interface, and the execution of the scene control command is refused. ;

[0028] like Figure 3The diagram shows the structure of a virtual reality scene simulation and analysis system based on AI large-scale model and speech recognition provided in this application embodiment. It includes: a speech front-end processing module, a semantic understanding and spatial reasoning module, an instruction generation and hierarchical evaluation module, and a rationality verification module. The speech front-end processing module collects the user's original speech signal and three-dimensional spatial geometric information, performs multi-channel acoustic environment perception processing on the original speech signal, and determines whether to perform temporal peak linkage amplitude threshold adaptive adjustment. If so, it performs speech enhancement processing on the original speech signal after adjustment; otherwise, it directly performs speech enhancement processing on the original speech signal to obtain an enhanced speech signal. The semantic understanding and spatial reasoning module obtains the current virtual scene multimodal context information and inputs the enhanced speech signal and the virtual scene multimodal context information into a pre-trained AI large-scale language model for joint processing. The system comprises four modules: a semantic reasoning module, which outputs corrected recognition text and user intent parsing results fused with virtual scene spatial information; a command generation and hierarchical evaluation module, which maps 3D spatial geometric information to a multimodal feature alignment space for cross-modal feature alignment and fusion to obtain a multimodal fusion feature vector; a scene control command generation and hierarchical evaluation module, which generates scene control commands that match the spatial logic of the virtual reality scene based on the execution results of the scene control commands; a rationality verification module, which generates feedback signals based on the execution results of the scene control commands; and a rationality verification module, which verifies the operation object and operation behavior pointed to by the scene control commands based on the 3D spatial geometric information of the virtual scene before executing the scene control commands to determine whether the operation behavior conforms to the spatial logic of the virtual scene.

[0029] In this embodiment, the virtual reality scene simulation and analysis system based on AI large-scale models and speech recognition is composed of four core functional modules: a speech front-end processing module, a semantic understanding and spatial reasoning module, an instruction generation and hierarchical evaluation module, and a rationality verification module. These modules are interconnected, linked at each level, and collaborate in a closed loop to achieve intelligent processing throughout the entire process, including high-precision speech interaction, spatial semantic fusion reasoning, intelligent hierarchical instruction scheduling, and scene logic compliance verification in complex VR sound field environments. This effectively solves the problems of fragmented modules, weak speech anti-interference capabilities, disconnect between semantic and spatial information, lack of hierarchical instruction execution, and simplistic operation logic in traditional VR speech interaction systems. The technical defects of the conflict; among them, the voice front-end processing module, as the system's front-end perception input unit, is responsible for the overall data acquisition and voice signal optimization preprocessing. Specifically, it is used to collect the user's original voice signal and the user's three-dimensional spatial geometric information in the virtual scene in real time during virtual reality interaction. Simultaneously, it performs multi-channel acoustic environment perception processing on the collected original voice signal, accurately identifies complex acoustic environment characteristics such as enclosed sound field interference, equipment fan noise, and multi-person reverberation in the current scene, and intelligently determines whether to activate the time-domain peak linkage amplitude threshold adaptive adjustment mechanism. When it is determined that adaptive adjustment needs to be executed, this module dynamically updates the human... After noise adaptive adaptation is completed within the effective amplitude threshold range, the original speech signal undergoes refined speech enhancement processing to effectively filter out various environmental noises and sudden interference noises. When it is determined that threshold adjustment is not required, the original speech signal is directly optimized for speech enhancement, ultimately outputting an enhanced speech signal with high signal-to-noise ratio, complete human voice features, and adaptability to the complex acoustic environment of VR, providing high-quality speech data support for backend semantic reasoning. The semantic understanding and spatial reasoning module, as the core intelligent parsing unit of the system, receives the enhanced speech signal output from the front end and acquires complete multimodal context information in the current virtual reality scene in real time, covering scene operating status, virtual... The system incorporates key information such as the simulated object's spatial location, the user's gaze point, and historical interaction context. It then inputs enhanced speech signals and standardized, structured virtual scene multimodal context information into a pre-trained AI large language model to conduct deep joint semantic reasoning. Relying on a cross-modal attention alignment mechanism, it completes low-confidence word correction, semantic disambiguation, pronoun completion, and spatial logic verification. Finally, it outputs corrected and recognized text that fully integrates virtual reality 3D spatial information and fits the real interaction logic of the scene, as well as accurate and structured user intent parsing results. This solves the problems of traditional speech recognition, which only transcribes words, is detached from scene spatial logic, and has large deviations in spoken intent recognition.The instruction generation and hierarchical evaluation module, serving as the system's instruction generation and intelligent scheduling unit, is responsible for multimodal feature fusion and intelligent hierarchical processing of instructions. This module maps the collected 3D spatial geometric information to a unified multimodal feature alignment space, completing cross-modal feature alignment and deep fusion of speech semantic features and spatial geometric features. This generates a high-dimensional unified multimodal fusion feature vector. Combined with the corrected recognition text output by the aforementioned modules and the user intent parsing results, it intelligently generates standardized scene control instructions that perfectly match the spatial logic of the virtual reality scene and have no spatial logic conflicts. Simultaneously, it performs multi-dimensional quantitative complexity hierarchical evaluation on the generated scene control instructions. By analyzing quantitative parameters such as the number of instruction action elements, the number of operation targets, the number of spatial constraints, and the depth of temporal dependencies, and combining semantic association graphs and historical contextual dependency features, it accurately determines whether the current scene control instruction is a simple atomic instruction or a complex composite instruction. This provides a hierarchical basis for the differentiated scheduling and execution of subsequent local lightweight inference and cloud-based collaborative distributed inference, achieving reasonable allocation of computing resources and precise control of interaction latency. The security verification module, as the final execution security verification and feedback output unit of the system, plays a core role in pre-execution verification of instructions and generation of interactive feedback. Before any scene control instruction is formally executed, this module performs a comprehensive rationality verification of the target operation object and the operation to be executed based on the three-dimensional spatial geometric information of the virtual scene. It judges whether the operation behavior conforms to the inherent spatial logic and physical rules of the virtual scene from multiple dimensions such as spatial feasibility, physical support constraints, motion collision interference, and state compatibility. It intercepts all invalid instructions with spatial conflicts, physical violations, and state incompatibilities and outputs corresponding prompts. Only compliant and valid instructions are allowed to drive the virtual scene update. At the same time, based on the final execution result of the scene control instruction, it can synchronously generate multi-dimensional feedback signals, including voice feedback signals, virtual scene screen update signals, and haptic feedback drive signals, to realize closed-loop feedback of human-computer interaction. This effectively ensures the rationality, security, stability, and smoothness of virtual reality scene control, and improves the overall intelligence level and commercial adaptability of VR multimodal voice interaction.

[0030] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the scope and intent of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application is also intended to include such modifications and variations.

Claims

1. A virtual reality scene simulation and analysis method based on AI large-scale models and speech recognition, characterized in that, Includes the following steps: Step S1: Collect the user's original speech signal and three-dimensional spatial geometric information, perform multi-channel acoustic environment perception processing on the original speech signal, and determine whether to perform time-domain peak linkage amplitude threshold adaptive adjustment. If yes, perform speech enhancement processing on the original speech signal after adjustment; otherwise, directly perform speech enhancement processing on the original speech signal to obtain an enhanced speech signal. Step S2: Obtain the current virtual scene multimodal context information, input the enhanced speech signal and the virtual scene multimodal context information into the pre-trained AI large language model, perform joint semantic reasoning, and output the corrected recognition text and user intent parsing results that integrate virtual scene spatial information. Step S3: Map the three-dimensional spatial geometric information to the multimodal feature alignment space for cross-modal feature alignment and fusion to obtain a multimodal fusion feature vector. Based on the multimodal fusion feature vector and the corrected recognition text and user intent parsing results, generate scene control instructions that match the spatial logic of the virtual reality scene. Perform complexity classification evaluation on the scene control instructions to determine whether the scene control instructions are simple atomic instructions or composite instructions. Step S4: Based on the execution result of the scene control command, generate a feedback signal. Before executing the scene control command, perform a rationality check on the operation object and operation behavior pointed to by the scene control command based on the three-dimensional spatial geometric information of the virtual scene, and determine whether the operation behavior conforms to the spatial logic of the virtual scene.

2. The virtual reality scene simulation and analysis method based on AI large model and speech recognition as described in claim 1, characterized in that: The specific process for determining whether to perform adaptive adjustment of the time-domain peak linkage amplitude threshold is as follows: The acquired raw speech signal is processed by time-domain framing based on the preset frame length and frame shift. The time-domain amplitude envelope of each analysis frame is extracted, and the time-domain amplitude peak value of the analysis frame is determined. A sliding time window buffer is constructed to store the temporal amplitude peaks of consecutive historical analysis frames in chronological order, forming a temporal amplitude peak sequence. The statistical distribution parameters of the temporal amplitude peak sequence are calculated, and the statistical distribution parameters include at least the peak mean and peak standard deviation. Based on statistical distribution parameters, an adaptive dual-threshold threshold interval is dynamically constructed. The adaptive dual-threshold threshold interval includes an effective lower threshold and an effective upper threshold. The effective lower threshold is determined based on the weighted difference between the mean of the peak value and the standard deviation of the peak value, and the effective upper threshold is determined based on the tracking value of the maximum amplitude peak value within the sliding time window buffer. Set the maximum step size for a single adjustment, and compare the baseline effective lower limit threshold and baseline effective upper limit threshold calculated in the current analysis frame with the corresponding thresholds that actually took effect in the previous time step, specifically: If the absolute value of the difference between the baseline effective lower limit threshold and the effective lower limit threshold that actually took effect at the previous moment is greater than the maximum step size, then it is determined that the current lower limit threshold has changed. The effective lower limit threshold that actually took effect at the current moment is updated to the effective lower limit threshold that actually took effect at the previous moment plus or minus the maximum step size, and the update direction is in the direction of the baseline effective lower limit threshold.

3. The virtual reality scene simulation and analysis method based on AI large model and speech recognition as described in claim 2, characterized in that: The comparison with the corresponding threshold that actually took effect at the previous moment also includes: If the absolute value of the difference between the baseline effective lower limit threshold and the effective lower limit threshold that actually took effect at the previous moment is less than or equal to the maximum step size, then the effective lower limit threshold that actually took effect at the current moment will be updated to the baseline effective lower limit threshold. If the absolute value of the difference between the baseline effective upper limit threshold and the effective upper limit threshold that actually took effect at the previous moment is greater than the maximum step size, it is determined that the current upper limit threshold has changed. The effective upper limit threshold that actually took effect at the current moment is updated to the effective value that actually took effect at the previous moment plus or minus the maximum step size, and the update direction is in the direction of the baseline effective upper limit threshold. If the absolute value of the difference between the baseline effective upper limit threshold and the effective upper limit threshold that actually took effect at the previous moment is less than or equal to the maximum step size, then the effective upper limit threshold that actually took effect at the current moment will be updated to the baseline effective upper limit threshold. Based on the final effective lower threshold and effective upper threshold, the amplitude threshold of the time domain signal of the current analysis frame is determined, and the signal components whose time domain amplitude is within the closed interval formed by the effective lower threshold and the effective upper threshold are determined as effective human voice components and retained. Low-amplitude components with time-domain amplitudes below the effective lower limit threshold are identified as silent segments or stable noise components and are suppressed or removed. High-amplitude components with time-domain amplitudes exceeding the effective upper threshold are identified as sudden impact noise components and clipped or filtered out, thereby dynamically extracting pure and effective human voice time-domain signals.

4. The virtual reality scene simulation and analysis method based on AI large model and speech recognition as described in claim 1, characterized in that: The specific process for performing joint semantic reasoning is as follows: The enhanced speech signal is input into the acoustic encoder and language decoder of a pre-trained multilingual speech recognition model to perform speech-to-text decoding processing and output the initial recognized text sequence and its corresponding word-level confidence score and sentence-level confidence score. Based on the word-level confidence score, low-confidence words in the initial recognition text sequence are marked. The low-confidence words refer to words whose word-level confidence scores are lower than a preset confidence threshold. Obtain the virtual scene multimodal context information of the current virtual reality scene, which includes the scene state information of the virtual reality scene, the spatial location information of virtual objects, the coordinates of the user's gaze point, and the historical dialogue context information; The multimodal context information of the virtual scene is structured and encoded according to a preset encoding template to form a scene context embedding vector in a unified format. The encoding template includes: scene semantic slots, used to describe the global semantic category of the current virtual scene; object spatial relationship slots, used to describe the relative spatial position between virtual objects and their relative pose relationship with the user; gaze focus slots, used to represent the target virtual object corresponding to the user's gaze point and its three-dimensional coordinates; and interaction history slots, used to store the state-action pair sequence of the most recent N rounds of dialogue history. Construct multimodal joint reasoning prompts that integrate initial recognition text with multimodal contextual information from the virtual scene.

5. The virtual reality scene simulation and analysis method based on AI large model and speech recognition as described in claim 4, characterized in that: The multimodal joint reasoning prompts include: System role instructions are used to assign the role of the AI ​​large language model as an agent for semantic understanding and spatial reasoning in virtual reality scenes; The speech recognition candidate text field is used to carry the initial recognition text sequence; The spatial context description field is used to describe the specific content of each slot in the scene context embedding vector in natural language. The low confidence correction indication field is used to indicate the specific location of the low confidence word position and the correction constraints. The correction constraints are the semantic fluency constraints and spatial rationality constraints of the low confidence word position in the virtual scene context. After word embedding and position encoding of the multimodal joint reasoning prompts, a large language model input sequence is generated; The input sequence of the large language model is input into a pre-trained AI large language model, which contains at least a multi-layer Transformer decoder structure; In each layer of the Transformer decoder, cross-modal semantic alignment is performed, specifically as follows: By using a self-attention mechanism to model the global contextual dependency of each element in the input sequence, the semantic relationship between speech recognition candidate text and scene context information is captured. By using a cross-attention mechanism, attention weights are calculated for each word in the candidate text field of speech recognition and each spatial entity in the spatial context description field, forming a speech-space cross-modal attention alignment matrix; Based on the speech-space cross-modal attention alignment matrix, spatial semantic correction is performed on each low-confidence word position.

6. The virtual reality scene simulation and analysis method based on AI large model and speech recognition as described in claim 5, characterized in that: The specific process for spatial semantic correction of each low-confidence word position is as follows: Based on the attention weight distribution between low-confidence words and each virtual object and spatial location, candidate words with the highest attention weight and semantic fluency that meet preset conditions are selected from the latent space of the AI ​​large language model to replace the low-confidence words, thereby generating corrected recognition text. The AI ​​large language model performs joint semantic reasoning based on the corrected recognized text and the scene context embedding vector, specifically as follows: The action predicate and action parameters involved in the text are parsed, corrected and identified. The action predicate represents the type of operation that the user expects to perform, and the action parameters include at least the target object identifier and spatial operation parameters. The spatial logic consistency of the action parameters and the virtual object spatial location information and gaze focus slot information embedded in the scene context vector is checked to determine whether the target object pointed to by the action parameters exists in the current virtual scene and whether the operation type is executable in spatial geometry. Based on the results of spatial logical consistency verification and historical dialogue context information, semantic disambiguation and intent completion are performed on the corrected and identified text, and the user intent parsing results are output. The output layer of the AI ​​large language model is structured and decoded to generate structured output data containing corrected recognized text and user intent parsing results; The user intent parsing result includes at least the intent category identifier, the target object identifier, the operation parameter list, and the operation execution priority.

7. The virtual reality scene simulation and analysis method based on AI large model and speech recognition as described in claim 1, characterized in that: The specific process for evaluating the complexity of scene control commands is as follows: The generated scene control instructions are structured and parsed to extract quantized feature parameters from the scene control instructions. The quantized feature parameters include the number of action elements, the number of operation targets, the number of spatial constraints, and the temporal dependency depth. Based on the quantized feature parameters, a multi-dimensional complexity evaluation vector of the current scene control command is constructed, and the preset weight coefficients corresponding to each evaluation dimension are obtained. The preset weight coefficients include the weight of the number of action elements, the weight of the number of operation targets, the weight of the number of spatial constraints, and the weight of the temporal dependency depth. Based on each evaluation dimension and its corresponding preset weight coefficients, a weighted complexity quantification calculation is performed to obtain a comprehensive complexity score. The correlation between the action predicates in the scene control commands is analyzed, specifically: Calculate the semantic similarity between any two action predicates, wherein the semantic similarity is determined based on the vector distance between the action predicates in a preset action semantic space; Based on semantic similarity, an action predicate association graph is constructed. The action predicate association graph uses each action predicate as a node, the connection between nodes with semantic similarity greater than a preset similarity threshold as edges, and the value of semantic similarity as the edge weight. Extract the number of connected components in the action predicate association graph, whereby the number of connected components represents the degree of interrelation between the action predicates in the scene control command; Obtain the historical dialogue context information prior to the current interaction round, and extract the sequence of preceding interaction events associated with the current scene control command from the historical dialogue context information; Based on the referential and causal relationships between the preceding interactive event sequence and the current scene control command, the context dependency depth of the current scene control command is calculated. The context dependency depth represents the degree to which the current scene control command depends on historical interactive information semantically, and is positively correlated with the number of consecutive interactive rounds in the preceding interactive event sequence that have referential or causal relationships with the current command.

8. The virtual reality scene simulation and analysis method based on AI large model and speech recognition as described in claim 7, characterized in that: The complexity-based evaluation of scene control commands also includes: Using the overall complexity score, the number of connected components, and the context dependency depth as input parameters, if the first set of conditions is met simultaneously, the current scene control instruction is determined to be a simple atomic instruction, specifically: The overall complexity score is less than or equal to the preset simple instruction complexity threshold; The number of connected components is equal to 1, indicating that each action predicate in the scene control command points to the same operation target or belongs to the same action sequence and cannot be separated. The context dependency depth is less than or equal to a preset shallow context threshold, indicating that the current scene control commands do not depend on the complex context reasoning of previous interactions. If any condition or any combination of conditions in the second condition group is met, the current scene control command is determined to be a composite command, specifically: The overall complexity score is greater than the simple instruction complexity threshold; The number of connected components is greater than 1, indicating that the scene control command contains a set of action predicates pointing to different operation targets or independent action paths, and has the space for split execution. The context dependency depth is greater than the shallow context threshold, indicating that the current scene control command needs to rely on the context information of multiple rounds of historical dialogue for intent completion or reference resolution. When the current scene control command is determined to be a composite command, the scene control command is decomposed into several sub-commands according to the boundaries of each connected component in the action predicate association graph and the temporal dependency depth. Each sub-instruction obtained from the decomposition is labeled with the number of action elements, the number of operation targets, the number of spatial constraints, and the depth of temporal dependencies, and each sub-instruction is assigned an independent sub-instruction identifier and execution priority.

9. The virtual reality scene simulation and analysis method based on AI large model and speech recognition as described in claim 1, characterized in that: The specific process for performing the rationality verification is as follows: If the rationality check passes, the scene manipulation command is executed and the virtual reality scene is updated; If the validity check fails, the scene control command will not be executed, and a prompt message will be output to the user. After the virtual reality scene is updated, the scene status information and the spatial location information of virtual objects are updated synchronously.

10. A system applying the virtual reality scene simulation and analysis method based on AI large model and speech recognition as described in any one of claims 1-9, characterized in that, include: The system includes a voice front-end processing module, a semantic understanding and spatial reasoning module, an instruction generation and hierarchical evaluation module, and a rationality verification module. The voice front-end processing module is used to collect the user's original voice signal and three-dimensional spatial geometric information, perform multi-channel acoustic environment perception processing on the original voice signal, determine whether to perform time-domain peak linkage amplitude threshold adaptive adjustment, if yes, then perform voice enhancement processing on the original voice signal after adjustment, if no, then directly perform voice enhancement processing on the original voice signal to obtain an enhanced voice signal. The semantic understanding and spatial reasoning module is used to obtain the multimodal context information of the current virtual scene, input the enhanced speech signal and the multimodal context information of the virtual scene into the pre-trained AI large language model, perform joint semantic reasoning, and output the corrected recognition text and user intent parsing results that integrate the spatial information of the virtual scene. The instruction generation and hierarchical evaluation module is used to map three-dimensional spatial geometric information to a multimodal feature alignment space for cross-modal feature alignment and fusion, obtain a multimodal fusion feature vector, generate scene control instructions that match the spatial logic of the virtual reality scene based on the multimodal fusion feature vector and the corrected recognition text and user intent parsing results, perform complexity hierarchical evaluation on the scene control instructions, and determine whether the scene control instructions are simple atomic instructions or composite instructions. The rationality verification module is used to generate feedback signals based on the execution results of scene control commands. Before executing scene control commands, it performs rationality verification on the operation objects and operation behaviors pointed to by the scene control commands based on the three-dimensional spatial geometric information of the virtual scene, and determines whether the operation behaviors conform to the spatial logic of the virtual scene.