Voice Control Method, System and Storage Medium of Virtual Platform

Abstract

The present application provides a voice control method, system and storage medium for a virtual platform. The method includes: performing noise reduction and start / end point recognition on the acquired initial voice command to obtain a processed voice command; mapping the text converted from the processed voice command to a predefined atomic operation instruction set to obtain an initialization instruction; generating an atomic operation instruction or an atomic operation instruction sequence according to the complexity of the initialization instruction; the atomic operation instruction and the atomic operation instruction sequence are respectively instructions that can be directly executed; combining the content currently virtualized by the virtual platform and the natural law to verify the rationality of the atomic operation instruction or the atomic operation instruction sequence; and executing the verified atomic operation instruction or atomic operation instruction sequence. This method can solve the problem that the commonly used voice control methods at present cannot balance the response speed and complex instruction processing, thereby improving the efficiency of voice control of the virtual platform.

Description

Technical Field

[0001] This application relates to the technical field of virtual platform control. Specifically, it relates to a voice control method, system, and storage medium for a virtual platform. Background Art

[0002] With the development of the era, technical means such as WebGL 3D rendering and physical engine simulation have achieved digital reconstruction of experimental processes in disciplines such as chemistry, biology, and physics. In the evolution process of virtual experiment interaction methods, voice control technology has gradually become a key breakthrough point for improving operation efficiency.

[0003] However, the currently commonly used voice control methods have exposed serious adaptability problems in virtual platforms. Taking a typical chemical titration experiment as an example, users need to perform multiple click operations, with a high accidental touch rate and a relatively long average operation time. At the same time, in a multilingual environment, the misoperation rate caused by operation semantic differences is increasing steadily.

[0004] That is to say, the current method for voice control of virtual platforms is not efficient enough. Summary of the Invention

[0005] The purpose of the embodiments of this application is to provide a voice control method, system, and storage medium for a virtual platform. Based on the dual cognitive mechanism of "conditioned reflex + deep thinking", this method can solve the problem that the currently commonly used voice control methods cannot balance response speed and complex instruction processing during the control of virtual platforms, thereby improving the efficiency of voice control of virtual platforms.

[0006] In the first aspect, the embodiments of this application provide a voice control method for a virtual platform, including: performing noise reduction and start / end point recognition on the obtained initial voice instruction to obtain a processed voice instruction; mapping the text converted from the processed voice instruction to a predefined atomic operation instruction set to obtain an initialization instruction; generating an atomic operation instruction or an atomic operation instruction sequence according to the complexity of the initialization instruction; where the atomic operation instruction and the atomic operation instruction sequence are respectively instructions that can be directly executed; combining the content currently virtualized by the virtual platform and natural laws to verify the rationality of the atomic operation instruction or the atomic operation instruction sequence; and executing the verified atomic operation instruction or atomic operation instruction sequence.

[0007] The voice control method of the above virtual platform performs noise reduction and start / end point recognition on the acquired initial voice command, maps the converted text to a predefined atomic operation instruction set to obtain an initialized voice command, and performs corresponding processing based on the complexity of the initialized voice command to generate an atomic operation instruction or an atomic operation instruction sequence that can be directly executed. Before executing the generated atomic operation instruction or atomic operation instruction sequence, its rationality is verified, which improves the processing speed and accuracy of the voice, and thus improves the efficiency of voice control of the virtual platform.

[0008] In combination with the first aspect, optionally, the noise reduction and start / end point recognition of the acquired initial voice command to obtain a processed voice command includes: representing the initial voice command as a time-domain signal; and using spectral subtraction to remove the noise component from the initial voice command; wherein, the implementation formula of the spectral subtraction is:

[0009] ;

[0010] In the formula, f represents the frequency, X clean (f) represents the pure voice signal after removing the noise component from the initial voice command, X(f) represents the frequency-domain signal after Fourier transform of the time-domain signal, N(f) represents a preset room environment noise template; α is a subtraction factor, β is a spectral lower limit protection coefficient, represents the phase information of the initial voice command. [[ID=K15]]

[0011] The voice control method of the above virtual platform performs noise reduction processing on the initial voice command by using spectral subtraction, removing the noise component from the noisy voice spectrum, ensuring the pure voice characteristics, thus obtaining a clearer and more easily recognizable voice signal, and also facilitating the subsequent accurate recognition of the start / end point. Finally, it further improves the processing speed and accuracy of the voice, and correspondingly further improves the efficiency of voice control of the virtual platform.

[0012] In combination with the first aspect, optionally, the noise reduction and start / end point recognition of the acquired initial voice command to obtain a processed voice command further includes: converting the spectrum after inverse Fourier transform of the frequency-domain signal into a time-domain signal; judging whether the number of consecutive frames satisfying and in the time-domain signal exceeds a first frame number threshold; if it is determined to be satisfied, taking the first frame in the consecutive frames as the start point of the voice command; judging whether the number of consecutive frames satisfying and in the time-domain signal exceeds a second frame number threshold; if it is determined to be satisfied, taking the last frame in the consecutive frames as the end point of the voice command; wherein, En With Z n Are calculated respectively by the following formulas:

[0013] ;

[0014] ;

[0015] In the formula, E n represents the short-time energy of the nth frame in the time-domain signal, L represents the frame length, x(k) represents the amplitude of the kth sampling point, and Z n represents the zero-crossing rate of the nth frame in the time-domain signal, I represents a judgment function used to judge whether it holds, μE and μZ respectively represent the mean of the short-time energy and the mean of the zero-crossing rate of the noise component; σE and σZ represent the standard deviation of the short-time energy and the standard deviation of the zero-crossing rate of the noise component.

[0016] The above voice control method of the virtual platform calculates the short-time energy and zero-crossing rate of the initial voice command converted into a time-domain signal, and determines the start and end points by judging whether the short-time energy and zero-crossing rate simultaneously satisfy the relationships corresponding to the mean of the short-time energy and zero-crossing rate of the noise component, the standard deviation of the short-time energy and zero-crossing rate of the noise component. It more effectively distinguishes the voice signal and background noise, reduces false triggers, improves the accuracy of voice recognition, and ultimately further improves the processing speed and accuracy of voice, and correspondingly further improves the efficiency of voice control of the virtual platform.

[0017] Combined with the first aspect, optionally, mapping the text converted from the processed voice command to a predefined atomic operation instruction set to obtain an initialization instruction includes: performing frame segmentation on the processed voice command and inputting it into a pre-constructed semantic feature extraction module to generate a phoneme sequence; inputting the phoneme sequence into a pre-constructed semantic feature alignment module to generate a semantic vector; and performing a normalization mapping on the semantic vector to obtain the initialization instruction; where the formula for the normalization mapping is:

[0018] ;

[0019] In the formula, z 标 represents the initialization instruction, h CLS is the semantic vector, z is the predefined atomic operation instruction set, and z i is an element in the atomic operation instruction set, and E(z i ) represents the pre-trained instruction embedding vector.

[0020] The voice control method of the above virtual platform realizes the conversion of the processed voice instruction into a semantic vector by performing frame processing and semantic feature alignment on the processed voice instruction. Then, the semantic vector is subjected to a normalization mapping to generate a structured and initialized instruction that can be directly used for subsequent processing, ensuring the accuracy and executability of the instruction, thereby further improving the processing speed and accuracy of the voice, and correspondingly further improving the efficiency of voice control of the virtual platform.

[0021] In combination with the first aspect, optionally, generating an atomic operation instruction or an atomic operation instruction sequence according to the complexity of the initialization instruction includes: calculating the instruction complexity of the initialization instruction by a reflexive neural network; wherein, the reflexive neural network includes a fast response channel and a complex instruction processing channel; the calculation formula of the instruction complexity is as follows:

[0022] ;

[0023] In the formula, lexical_complexity represents the lexical complexity of the initialization instruction, syntactic_depth represents the syntactic depth of the initialization instruction, and Score represents the instruction complexity; determining whether the instruction complexity exceeds a complexity threshold; if it is determined to exceed, then the complex instruction processing channel receives the initialization instruction and the cached historical operation information, and performs an embedding layer transformation on the initialization instruction to output an embedding vector; calculating the relative position encoding of the initialization instruction by using a position encoding formula; wherein, the position encoding formula is:

[0024] ;

[0025] In the formula, k represents the dimension index, d represents the embedding dimension, and i and j respectively represent the sequence positions of two elements in the initialization instruction; fusing the embedding vector and the relative position encoding to obtain a fused embedding vector; processing the fused embedding vector by using a multi-head attention mechanism to obtain a context-aware vector; and combining the context-aware vector with the historical operation information to generate the atomic operation instruction or the atomic operation instruction sequence.

[0026] The voice control method of the above virtual platform starts from the input semantic vector sequence and historical operation information, and through the processing of relative position encoding and multi-head attention mechanism, enables the complex instruction processing channel to effectively capture the position information in the sequence and integrate it into the understanding and processing of complex instructions. This helps to more accurately parse instructions with complex syntactic structures and long-distance dependencies, and generate accurate operation instructions in combination with historical context. That is, it improves the processing speed and accuracy of speech, thereby improving the efficiency of voice control of the virtual platform.

[0027] In combination with the first aspect, optionally, the generating of atomic operation instructions or atomic operation instruction sequences according to the complexity of the initialization instruction further includes: if it is determined that the limit is not exceeded, then the fast response channel performs a convolution operation based on the Mel-frequency cepstral coefficients of the initialization instruction to capture key acoustic patterns; performs global pooling dimensionality reduction on the initialized instruction after the convolution operation, and uses an activation function to output the probability distribution of elements in the atomic operation instruction set; and generates the atomic operation instruction or atomic operation instruction sequence according to the probability distribution.

[0028] The voice control method of the above virtual platform realizes the fast processing of simple instructions, the efficient recognition and fast response of simple instructions through the convolution operation of Mel-frequency cepstral coefficients on the initialization instruction, global pooling dimensionality reduction, and the output of the probability distribution by the activation function, and the output of atomic operation instructions or atomic operation instruction sequences in combination with the probability distribution. Finally, it also improves the processing speed and accuracy of speech, thereby improving the efficiency of voice control of the virtual platform.

[0029] In combination with the first aspect, optionally, the verifying of the rationality of the atomic operation instruction or atomic operation instruction sequence by combining the currently virtualized content of the virtual platform and natural laws includes: constructing a four-dimensional state tensor for describing the virtual platform picture; where the tensor formula is:

[0030] ;

[0031] In the formula, x, y, z represent the scene coordinate system of the virtual platform picture, t represents the time stamp, v represents the state of the operation object corresponding to the atomic operation instruction or atomic operation instruction sequence, c p represents the operation stage divided for the operation process; r represents the constraint rule;

[0032] Extract the image features of the virtual platform screen to obtain an image feature vector; perform linear projection on the atomic operation instruction or atomic operation instruction sequence to obtain a projected speech feature vector; combine the image feature vector and the projected speech feature vector to perform cross-modal attention calculation to obtain an attention weight distribution; where the formula for cross-modal attention calculation is:

[0033] ;

[0034] In the formula, Q i is the linear projection of the speech feature vector, K j is the linear projection of the image feature vector, and d is the dimension of the speech feature vector and the image feature vector; combine the attention weight distribution and the four-dimensional state tensor to verify the rationality of the atomic operation instruction or atomic operation instruction sequence.

[0035] The above voice control method for the virtual platform can determine the specific position or object in the virtual experimental platform referred to by the current voice instruction (atomic operation instruction or atomic operation instruction sequence) through the attention weight distribution, so as to verify the rationality of the atomic operation instruction or atomic operation instruction sequence, ensuring the rationality, feasibility, and safety of the instruction in the experimental environment, etc. This further improves the processing speed and accuracy of the voice, thereby improving the efficiency of voice control of the virtual platform.

[0036] Combined with the first aspect, optionally, the combining the attention weight distribution and the four-dimensional state tensor to verify the rationality of the atomic operation instruction or atomic operation instruction sequence includes: combining the attention weight distribution and the four-dimensional state tensor to verify the atomic operation instruction or atomic operation instruction sequence according to the natural law; and comparing the object positioning heat map corresponding to the attention weight distribution with the four-dimensional state tensor to verify whether the operation object corresponding to the atomic operation instruction or atomic operation instruction sequence is consistent with the actual object in the virtual platform screen.

[0037] The above voice control method for the virtual platform compares the relevant area in the object positioning heat map with the coordinate system in the four-dimensional state tensor to confirm whether the operation object in the instruction is consistent with the actual object in the virtual platform, so as to verify the instruction. This further ensures the rationality, feasibility, and safety of the instruction in the experimental environment, etc. This further improves the processing speed and accuracy of the voice, thereby improving the efficiency of voice control of the virtual platform.

[0038] In a second aspect, an embodiment of the present application further provides a voice control system for a virtual platform, including: a voice input layer for denoising and start / end point recognition of the acquired initial voice command to obtain a processed voice command; a multi-language processing layer for mapping the text converted from the processed voice command to a predefined atomic operation instruction set to obtain an initialization instruction; a reflexive neural network for generating an atomic operation instruction or an atomic operation instruction sequence according to the complexity of the initialization instruction; wherein the atomic operation instruction and the atomic operation instruction sequence are respectively instructions that can be directly executed; a spatio-temporal perception matrix for verifying the rationality of the atomic operation instruction or the atomic operation instruction sequence by combining the content currently virtualized by the virtual platform and natural laws; and an execution layer for executing the verified atomic operation instruction or atomic operation instruction sequence.

[0039] The above voice control system device of the virtual platform has the same beneficial effects as the voice control method of the virtual platform provided in the above first aspect, or any optional implementation manner of the first aspect, and will not be elaborated herein.

[0040] In a third aspect, an embodiment of the present application further provides a storage medium, the storage medium includes a computer-readable storage medium, and a computer program is stored on the computer-readable storage medium, and when the computer program is run by a processor, it executes the method described above.

[0041] The above storage medium has the same beneficial effects as the voice control method of the virtual platform provided in the above first aspect, or any optional implementation manner of the first aspect, and will not be elaborated herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings required to be used in the embodiments of the present application. It should be understood that the following drawings only show some embodiments of the present application, and therefore should not be regarded as limiting the scope. For those of ordinary skill in the art, other related drawings can be obtained based on these drawings without creative efforts.

[0043] Figure 1 It is a flowchart of the voice control method of the virtual platform provided by the embodiment of the present application;

[0044] Figure 2 It is a specific flowchart of step S 110 in the voice control method of the virtual platform provided by the embodiment of the present application;

[0045] Figure 3 It is a specific flowchart of step S 120 in the voice control method of the virtual platform provided by the embodiment of the present application;

[0046] Figure 4 It is the specific flowchart of step S130 in the voice control method of the virtual platform provided by the embodiment of the present application;

[0047] Figure 5 It is the specific flowchart of step S140 in the voice control method of the virtual platform provided by the embodiment of the present application;

[0048] Figure 6 It is the specific flowchart of step S145 in the voice control method of the virtual platform provided by the embodiment of the present application;

[0049] Figure 7 It is the schematic diagram of the voice control system of the virtual platform provided by the embodiment of the present application. Detailed implementation manners

[0050] Next, the embodiments of the technical solutions of the present application will be described in detail with reference to the accompanying drawings. The following embodiments are only used to illustrate the technical solutions of the present application more clearly, so they are only examples and cannot be used to limit the protection scope of the present application.

[0051] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those of ordinary skill in the technical field to which this application belongs; the terms used herein are only for the purpose of describing specific embodiments and are not intended to limit this application.

[0052] In the description of the embodiments of the present application, technical terms such as "first" and "second" are only used to distinguish different objects and cannot be understood as indicating or implying relative importance or implicitly indicating the quantity, specific order or primary and secondary relationship of the indicated technical features. In the description of the embodiments of the present application, "multiple" means more than two unless otherwise specifically defined.

[0053] Please refer to Figure 1 , Figure 1 which is the flowchart of the voice control method of the virtual platform provided by the embodiment of the present application. The voice control method of the virtual platform provided by the embodiment of the present application may include:

[0054] Step S110: Perform noise reduction and start / end point recognition on the obtained initial voice command to obtain a processed voice command.

[0055] In the above step S110, voice data can be obtained first through an audio input device such as a microphone. Specifically, a multi-microphone array (4 channels, 120° pickup angle), a sampling rate of 48 kHz, and a quantization precision of 24 bits can be adopted, and a 8th-order Butterworth low-pass filter (cutoff frequency 20 kHz) can be used to eliminate high-frequency noise. After noise reduction and other preliminary processing of the initial voice command, since during the process of voice control, the time period during which audio can be collected is usually not limited to the time period when the user issues the control voice, but in addition to this time period, there is also a lot of complex background noise. Therefore, the starting point and ending point of the initial voice command can be identified to exclude the interference caused by the audio outside the time period when the user issues the control voice.

[0056] Regarding the identification of the start and end points, specifically, the double-threshold detection algorithm can be used to calculate the short-time energy and zero-crossing rate to accurately judge the start and end points of the voice. Even in a complex noise environment such as the sound of an air conditioner or the sound of keyboard tapping, the user's voice command can be accurately identified.

[0057] Step S120: Map the text converted from the processed voice command to a predefined atomic operation instruction set to obtain an initialization instruction.

[0058] In the above step S120, specifically, cross-language semantic alignment technology can be used to generate semantic vectors, and the text can be mapped to a predefined atomic operation instruction set to obtain an initialization instruction. Among them, the atomic operation instruction and the atomic operation instruction sequence are respectively instructions that can be directly executed. Among them, the initialization instruction obtained at this stage cannot be directly executed yet. Taking the evolution of virtual experiments as an example, the instruction obtained at this stage cannot be directly executed yet because it has not been combined with the experimental state, the experimental resource state has not been considered, the parameter integrity is insufficient, the parameter rationality is doubtful, and the timing relationship is not clear, etc.

[0059] Step S130: Generate an atomic operation instruction or an atomic operation instruction sequence according to the complexity of the initialization instruction.

[0060] In the above step S130, the atomic operation instruction and the atomic operation instruction sequence are both instructions that can be directly executed. According to the complexity of the initialization instruction, different processing channels can be set. For example, if the complexity of the initialization instruction is divided into three levels, then three channels can be correspondingly set to process the initialization instructions with different levels of complexity respectively. Another example: if the complexity of the initialization instruction is divided into two levels, then two channels can be correspondingly set to process the initialization instructions with different levels of complexity respectively. Of course, those skilled in the art can divide the complexity of the initialization instruction into corresponding numbers of levels according to actual application requirements and correspondingly set the corresponding numbers of channels. Different algorithms can be used in these channels to handle the initialization instructions with different complexities. The embodiments of the present application do not make specific limitations on this.

[0061] Exemplarily, the initialization instructions are divided into two levels: simple and complex. For simple and high-frequency instructions, such as "start titration" or "stop", they can be quickly processed through the corresponding channel. The network structure of this channel includes an input layer, a convolutional layer, a global pooling layer, and an output layer, which can quickly generate the probability of atomic operation instructions. The predefined "start titration" atomic operation instruction can be directly triggered. This operation instruction corresponds to a series of simple and fixed system actions, such as initializing the position of the burette, setting the initial volume, starting the titration timer, etc. These instructions are relatively simple and concise, that is, atomic operation instructions.

[0062] For complex instructions, such as "after titrating to a pH value of 7, record the current volume and calculate the concentration", another channel can be correspondingly enabled to understand the complete logic of this complex instruction by combining the previous experimental operations (for example, the initial preparation of acid-base solutions and the start of titration may have been carried out before) and the current experimental state (such as the pH value of the current solution may be higher or lower than 7, the volume of the titrated solution, etc.). The complex instruction is decomposed into a series of specific operation steps. First, continuously monitor the change in the pH value of the solution; second, pause the titration and record the volume of the solution in the burette when the pH value reaches 7; finally, calculate the concentration of the sodium hydroxide solution using chemical formulas based on the known initial concentration and titration volume of the acid-base solution. Generate a sequence containing multiple atomic operation instructions, such as "monitor pH value", "compare pH value with target value", "pause titration", "record volume", "apply concentration calculation formula", etc., and determine the execution order of these operations. These instructions are relatively complex, that is, an atomic operation instruction sequence composed of multiple atomic operation instructions.

[0063] Step S140: Check the rationality of the atomic operation instruction or the atomic operation instruction sequence in combination with the content currently virtualized by the virtual platform and the natural laws.

[0064] In the above step S140, by constructing a four-dimensional state tensor and combining the experimental scenario, time, object state, and experimental stage, the feasibility of the instruction in the current experimental environment can be verified, and it is ensured that the voice instruction is accurately matched with the objects and operations in the experimental scenario.

[0065] Continuing with the evolution of virtual experiments as an example, specifically, first, check whether the resources involved in the experiment (such as solutions, instruments) are available and whether the state of the experimental object meets the requirements of the instruction. For example, if the instruction is "heat the solution to 80 °C", it will be verified whether the heating equipment on the virtual experimental bench is ready and whether the solution container is placed correctly, etc. Then, ensure that the atomic operation instruction or atomic operation instruction sequence conforms to experimental laws such as physics and chemistry. For example, in a chemical experiment, if the atomic operation instruction or atomic operation instruction sequence is "mix 100 ml of water with 500 ml of hydrochloric acid", it will be checked according to chemical common sense and experimental settings whether the volume of the mixed solution is reasonable and whether there will be a violent reaction, etc. Verify whether the instruction conforms to the current stage of the experiment. For example, in a chemical titration experiment, if the current stage is the "preparation stage" and the instruction is "record the data of the titration end point", it will be determined that the instruction does not match the current stage and needs to be further confirmed or corrected.

[0066] Step S150: Execute the verified atomic operation instruction or atomic operation instruction sequence.

[0067] In the above step S150, the virtual platform can combine three-dimensional rendering technologies (such as WebGL), physics engines, and interaction designs to simulate the visual and operation experiences of real experiments. Users interact with the virtual platform, and the experimental results are calculated and updated in real time according to the users' operations, providing visual feedback.

[0068] Exemplarily, in a virtual experiment of chemical titration, users can drag a virtual dropper to drip acid or base solution into a virtual conical flask. The system will calculate the change in the pH value of the solution in real time according to the dosage and speed of the users' operations, and display the color change or pH curve on the screen.

[0069] In the above implementation process, by performing noise reduction and start / end point recognition on the obtained initial voice instruction, mapping the converted text to a predefined atomic operation instruction set, obtaining an initialized voice instruction, and performing corresponding processing based on the complexity of the initialized voice instruction to generate an atomic operation instruction or atomic operation instruction sequence that can be directly executed, and verifying its rationality before executing the generated atomic operation instruction or atomic operation instruction sequence, the processing speed and accuracy of the voice are improved, thereby improving the efficiency of voice control of the virtual platform.

[0070] Please refer to Figure 2 , Figure 2It is a specific flowchart of step S110 in the voice control method of the virtual platform provided by the embodiments of the present application. In some optional embodiments, step S110 may include:

[0071] Step S111: Represent the initial voice command as a time-domain signal.

[0072] In the above step S111, the initial voice command can be represented as a time-domain waveform signal x(t), where t is a time variable and the unit can be milliseconds.

[0073] Step S112: Use spectral subtraction to remove the noise component in the initial voice command.

[0074] In the above step S112, the implementation formula of spectral subtraction is:

[0075] ;

[0076] In the formula, f represents frequency, X clean (f) represents the clean voice signal after removing the noise component from the initial voice command, X(f) represents the frequency-domain signal after Fourier transform of the time-domain signal, and N(f) represents a preset room environment noise template. α is a subtraction factor, β is a spectral lower limit protection coefficient, represents the phase information of the initial voice command. Spectral subtraction can ensure the accuracy of subsequent recognition processing of the initial voice command.

[0077] In the above implementation process, by using spectral subtraction to perform noise reduction processing on the initial voice command, the noise component is removed from the noisy voice spectrum, ensuring the pure voice features, thus obtaining a clearer and more easily recognizable voice signal, and also facilitating the subsequent accurate recognition of the start and end points. Finally, the processing speed and accuracy of the voice are further improved, and accordingly, the efficiency of voice control of the virtual platform is further improved.

[0078] Please continue to refer to Figure 2 , in some optional embodiments, step S110 may further include:

[0079] Step S113: Convert the spectrum after inverse Fourier transform of the frequency-domain signal into a time-domain signal.

[0080] Step S114: Determine whether in the time-domain signal, the continuous number of frames satisfying and exceeds a first frame number threshold.

[0081] In the above step S114, E n and Z n are calculated by the following formulas respectively:

[0082] ;

[0083] ;

[0084] In the formula, E n represents the short-time energy of the nth frame in the time-domain signal, L represents the frame length, x(k) represents the amplitude of the kth sampling point, Z n represents the zero-crossing rate of the nth frame in the time-domain signal, I represents a judgment function used to judge whether it holds, μE and μZ respectively represent the mean of the short-time energy of the noise component and the mean of the zero-crossing rate; represents the standard deviation of the short-time energy of the noise component and the standard deviation of the zero-crossing rate.

[0085] If the determination is satisfied, execute step S115: Take the first frame in the continuous number of frames as the starting point of the voice command.

[0086] In the above step S115, in order to avoid misjudgment due to short-term fluctuations, it is possible to detect whether multiple consecutive frames all meet the above conditions. Exemplarily, if 4 consecutive frames (exceeding the first frame number threshold "3") all meet and , it is considered that the voice signal starts, that is, the starting point. Of course, those skilled in the art can also determine the first frame number threshold according to actual needs.

[0087] Step S116: Judge whether in the time-domain signal, the number of consecutive frames that meet and exceeds the second frame number threshold.

[0088] In the above step S116, similarly, in order to avoid misjudgment due to short-term fluctuations, it is possible to detect whether multiple consecutive frames all meet the above conditions. Exemplarily, if 4 consecutive frames (exceeding the first frame number threshold "3") all meet and , it is considered that the voice signal ends, that is, the termination point. Of course, those skilled in the art can also determine the second frame number threshold according to actual needs.

[0089] If the determination is satisfied, execute step S117: Take the last frame in the continuous number of frames as the termination point of the voice command.

[0090] In the above step S117, E n and Z n are respectively calculated by the following formulas:

[0091] ;

[0092] ;

[0093] In the formula, En represents the short-time energy of the nth frame in the time domain signal, L represents the frame length, x(k) represents the amplitude of the kth sampling point, Z n Indicates the zero-crossing rate of the nth frame in the time domain signal, and I represents the judgment function, which is used to judge Is it true? μE and μZ represent the mean of the short-time energy and the mean of the zero-crossing rate of the noise component respectively; σE and σZ represent the standard deviation of the short-time energy and the standard deviation of the zero-crossing rate of the noise component respectively.

[0094] In this implementation, the short-term energy and zero-crossing rate of the initial voice command converted into a time-domain signal are calculated, and the start and end points are determined by determining whether the short-term energy and zero-crossing rate simultaneously satisfy the corresponding relationship between the mean value of the short-term energy of the noise component and the zero-crossing rate, and the standard deviation of the short-term energy of the noise component and the standard deviation of the zero-crossing rate. This more effectively distinguishes the voice signal from background noise, reduces false triggers, and improves the accuracy of voice recognition. Ultimately, this further increases the speed and accuracy of voice processing, and accordingly further improves the efficiency of voice control of the virtual platform.

[0095] Please refer to Figure 3 , Figure 3 This is a specific flow chart of step S120 in the voice control method of the virtual platform provided in the embodiment of the present application. In some optional implementations, step S120 may include:

[0096] Step S121: The processed voice instruction is divided into frames and input into a pre-built semantic feature extraction module to generate a phoneme sequence.

[0097] In the above step S121, the voice instructions processed in the above steps can be processed based on the constructed WeNet multilingual acoustic model. The network structure of the model can be specifically:

[0098] Encoder: 8-layer Conformer (convolution-enhanced Transformer), each layer with 4 self-attention heads (head dimension 256), kernel size 15, and gated linear unit (GLU) activations;

[0099] Decoder: 2-layer Transformer, using CTC / Attention joint training.

[0100] For example, the processed voice command is divided into 128-dimensional Mel filter bank features (frame length 25ms, frame shift 10ms), which are input into the above model, and the model outputs the phoneme sequence , where . And can support shared encoders in 12 languages.

[0101] Step S122: Input the phoneme sequence into the pre-built semantic feature alignment module to generate a semantic vector.

[0102] In the above step S122, the XLM-R model can be specifically used as the semantic feature alignment module for cross-lingual semantic alignment to generate a semantic vector:

[0103] ;

[0104] where h CLS represents the semantic vector corresponding to the position of the CLS token, XLM-R base represents the XLM-R base model, s represents the input text sequence (such as the Chinese "Centrifuge for 5 minutes"), CLS represents the token, and the vector representation of the CLS token is usually used as the semantic representation of the entire sequence.

[0105] Step S123: Perform a normalization mapping on the semantic vector to obtain an initialization instruction.

[0106] In the above step S123, the formula for the normalization mapping is:

[0107] ;

[0108] where z 标 represents the initialization instruction, h CLS is the semantic vector, z is the predefined atomic operation instruction set, z i is an element in the atomic operation instruction set, and E(z i ) represents the pre-trained instruction embedding vector.

[0109] Exemplarily:

[0110] Input statement: "Centrifuge for 5 minutes";

[0111] Output {action: centrifuge, duration: 300}.

[0112] In the above implementation process, through frame processing and semantic feature alignment of the processed voice instructions, the conversion of the processed voice instructions into semantic vectors is achieved. Then, the normalization mapping is performed on the semantic vector to generate a structured and initialized instruction that can be directly used for subsequent processing, ensuring the accuracy and executability of the instruction, thereby further improving the processing speed and accuracy of the voice, and correspondingly further improving the efficiency of voice control of the virtual platform.

[0113] Please refer to Figure 4 , Figure 4It is the specific flowchart of step S130 in the voice control method of the virtual platform provided by the embodiments of the present application. In some optional embodiments, step S130 may include:

[0114] Step S131: Calculate the instruction complexity of the initialization instruction by a recurrent neural network.

[0115] In the above step S131, the recurrent neural network may include a fast response channel and a complex instruction processing channel. The fast response channel may specifically be a reflection channel, and the complex instruction processing channel may specifically be a cognitive channel. The network structure of the reflection channel may specifically be as shown in Table 1.

[0116] Table 1

[0117] Hierarchy Operation Output Dimension Function Input Layer MFCC Feature Input 128×256 Extract Speech Time-Frequency Characteristics Convolutional Layer 1 3×3 Convolution, stride = 2, ReLU Activation 64×128×128 Capture Local Acoustic Patterns Convolutional Layer 2 5×5 Convolution, stride = 1, ReLU Activation 32×124×124 Enhance Feature Abstraction Ability Global Pooling Layer Average Pooling 32×1×1 Reduce Dimension and Retain Key Information Output Layer Sigmoid Activation 286-Dimensional Probability Distribution Generate Probability of Atomic Operation Instructions

[0118] The calculation formula of the instruction complexity is as follows:

[0119] ;

[0120] In the formula, lexical_complexity represents the lexical complexity of the initialization instruction, syntactic_depth represents the syntactic depth of the initialization instruction, and Score represents the instruction complexity.

[0121] Step S132: Determine whether the instruction complexity exceeds the complexity threshold.

[0122] In the above step S132, exemplarily, the complexity threshold may be 6.5, then it can be determined whether Score > 0.65 holds. Of course, those skilled in the art can determine other values of the complexity threshold according to the actual application situation.

[0123] If it is determined that it exceeds, then execute step S133: The complex instruction processing channel receives the initialization instruction and the cached historical operation information, and performs an embedding layer transformation on the initialization instruction to output an embedding vector.

[0124] In the above step S133, if it is determined that it exceeds, it means that the complexity of the initialization instruction is relatively high, and then it can be processed by the complex instruction processing channel. Specifically, it can be implemented as follows: First, receive the initialization instruction and the cached historical operation information. The core component of the cognitive channel may be a memory unit, which can cache the hidden states of the most recent 20 instructions. Convert each element (such as a word or a sub-word unit, etc.) in the initialization instruction into a vector representation of a fixed dimension. Exemplarily, assume the input is a tokenized instruction sequence , each token w i is converted into a vector e i through the embedding layer, and an embedding matrix is formed.

[0125] Step S134: Calculate the relative position encoding of the initialization instruction using the position encoding formula.

[0126] In the above step S134, the position encoding formula is:

[0127] ;

[0128] In the formula, k represents the dimension index, d represents the embedding dimension, and i and j respectively represent the sequence positions of two elements in the initialization instruction. The embedding dimension can specifically be 512. Specifically, i and j in the position encoding formula respectively represent the position indices of two elements in the sequence: i represents the position of the current element in the sequence; j represents the position of the other element in the sequence.

[0129] The goal of the relative position encoding is to calculate the relative distance (i - j) between these two positions, and then map this relative distance to a high-dimensional space through a sine or cosine function to generate the corresponding position encoding. Through the relative position encoding, the relative position relationship between elements can be reflected.

[0130] Step S135: Fuse the embedding vector with the relative position encoding to obtain the fused embedding vector.

[0131] In the above step S135, the fusion method can specifically be addition. For example: the embedding vector is e i , and the relative position encoding is R ij , and the fused vector is e i ' = e i + R ij .

[0132] Step S136: Process the fused embedding vector using the multi-head attention mechanism to obtain the context-aware vector.

[0133] In the above step S136, the multi-head attention mechanism calculates in parallel through multiple attention heads to capture the associations between different elements in the sequence. Each attention head calculates query (Query), key (Key), and value (Value) vectors, and obtains the output through weighted summation of the attention scores. The multi-head attention mechanism outputs a new set of vector representations, which are the context-aware vectors. These vectors fuse the semantic information, relative position information, and association information of the input elements. These vector representations not only contain the semantic information and relative position information of the input elements, but also capture the association information between elements.

[0134] Among them, the formula of the multi-head attention mechanism can be:

[0135] ;

[0136] In the formula, each attention head is defined as: , where Q is the query vector, representing the features of the position or element that needs to be focused on currently. K is the key vector, representing the features of other positions or elements, which is used to match with the query vector. V is the value vector, representing the actual information content to be focused on. is the output weight matrix, which is used to linearly transform the concatenated vector into the final output space. Concat represents the concatenation operation, a function that concatenates the outputs of multiple attention heads into a vector. W i Q is the query weight matrix of the i-th attention head, which is used to linearly transform the input query vector into the i-th subspace. W i K is the key weight matrix of the i-th attention head, which is used to linearly transform the input key vector into the i-th subspace. W i V is the value weight matrix of the i-th attention head, which is used to linearly transform the input value vector into the i-th subspace. Attention is a function of the process of calculating attention scores and weighted summing value vectors.

[0137] Step S137: Combine the context-aware vector with the historical operation information to generate atomic operation instructions or a sequence of atomic operation instructions.

[0138] In the above step S137, by combining the weight information in the context-aware vector with each historical operation instruction in the historical operation information, atomic operation instructions or a sequence of atomic operation instructions that can be directly executed can be generated.

[0139] In the above implementation process, starting from the input semantic vector sequence and historical operation information, through the processing of relative position encoding and the multi-head attention mechanism, the complex instruction processing channel can effectively capture the position information in the sequence and integrate it into the understanding and processing of complex instructions. This helps to more accurately parse instructions with complex syntactic structures and long-distance dependencies, and generate accurate operation instructions in combination with historical context. That is, it improves the processing speed and accuracy of speech, thereby improving the efficiency of voice control of the virtual platform.

[0140] Please continue to refer to [[ID=_{36]]Figure 4 , in some optional implementation manners, according to the complexity of the initialization instruction, generating atomic operation instructions or a sequence of atomic operation instructions may further include:

[0141] If it is determined that it has not exceeded, then execute step S138: The fast response channel performs a convolution operation based on the Mel-frequency cepstral coefficients of the initialization instruction to capture key acoustic patterns.

[0142] In the above step S138, first, MFCC (Mel Frequency Cepstral Coefficients) features can be extracted from the input initialization instruction. Since MFCC simulates the sensitivity of the human auditory system to different frequencies, MFCC can effectively represent the acoustic characteristics of the speech signal. Specifically, it can be implemented as follows: first, convert the speech signal of the initialization instruction into a spectrum, then extract the energy through a Mel filter bank, and finally perform logarithmic transformation and discrete cosine transformation to obtain the MFCC coefficients.

[0143] Then, referring to Table 1, the convolutional layer 1 can be used to process the MFCC features. Specifically, it can be implemented as follows: the convolutional kernels in the convolutional layer 1 slide on the MFCC feature sequence to perform convolutional operations. Among them, hyperparameters such as the size and stride of the convolutional kernel usually affect the dimension and details of the output features. For example, using a convolutional kernel with a size of 3×3 and a stride of 2 can reduce the dimension of the features while extracting local acoustic patterns.

[0144] Step S139: Perform global pooling dimensionality reduction on the initialized instruction after the convolutional operation, and use an activation function to output the probability distribution of the elements in the atomic operation instruction set.

[0145] In the above step S139, first, the feature map after the convolutional operation usually has a high dimension, and a global pooling layer can be used to reduce the dimension of the feature map. Specifically, it can be implemented as follows: the global pooling layer performs operations such as averaging or taking the maximum value of all elements in each feature map to obtain a fixed-length vector, so as to reduce the dimension of the features, reduce the computational complexity, and retain key information.

[0146] Then, the dimensionality-reduced feature vector is input into an activation function, such as Sigmoid. The activation function maps the features to a specific numerical range. The activation function maps the output to the range between 0 and 1, which can represent the probability distribution and is used to convert the dimensionality-reduced features into the probability distribution of each element in the predefined atomic operation instruction set.

[0147] Step S1310: Generate an atomic operation instruction or an atomic operation instruction sequence according to the probability distribution.

[0148] In the above step S1310, first, since the output after being processed by the activation function is a probability distribution, which represents the possibility of each element in the predefined atomic operation instruction set corresponding to the input speech instruction being executed. For example, assume that the atomic operation instruction set includes operations such as "start titration", "record data", "add solution", etc., and the output probability distribution will give the probability of each operation being executed. Then, according to the probability distribution, select the atomic operation instruction with the highest probability as the atomic operation instruction or the atomic operation instruction sequence and output it.

[0149] In the above implementation process, through the convolution operation of Mel-frequency cepstral coefficients on the initialization instruction, global pooling for dimensionality reduction, and the output of the probability distribution by the activation function, and combining the probability distribution to output atomic operation instructions or atomic operation instruction sequences, the rapid processing of simple instructions is achieved, and the efficient recognition and rapid response of simple instructions are realized. Finally, the processing speed and accuracy of speech are also improved, thereby improving the efficiency of voice control of the virtual platform.

[0150] Please refer to Figure 5 , Figure 5 which is the specific flowchart of step S140 in the voice control method of the virtual platform provided by the embodiment of the present application. In some optional implementation manners, step S140 may include:

[0151] Step S141: Construct a four-dimensional state tensor for describing the virtual platform screen.

[0152] In the above step S141, the tensor formula is:

[0153] ;

[0154] In the formula, x, y, and z represent the scene coordinate system of the virtual platform screen, which can represent the position information of each object in the virtual environment represented by the virtual platform, such as the position targets of experimental equipment, chemical reagent bottles, virtual operating tables, etc. This coordinate system can specifically be the WebGL scene coordinate system. t represents the time stamp, which can be used to record the time information of the experimental operation and can help track the progress of the experiment and the timing relationship of the operations. v represents the state of the operation object corresponding to the atomic operation instruction or atomic operation instruction sequence, for example: the volume, temperature, concentration of the solution, the on / off state of the equipment, etc. c p represents the operation stage divided for the operation process, for example: such as "preparation stage", "titration stage", "heating stage", etc. r represents the constraint rule, r ∈ {0,1} 63 Activation state, such as max_temp = 100 °C.

[0155] Step S142: Extract the image features of the virtual platform screen to obtain an image feature vector.

[0156] In the above step S142, exemplarily, an experimental scene image generated by the WebGL renderer, for example: a 224×224×3 RGB image, is input into the ResNet-50 model to extract image features, and finally an image feature vector is generated.

[0157] Step S143: Perform linear projection on the atomic operation instruction or atomic operation instruction sequence to obtain a projected speech feature vector.

[0158] In the above step S143, the linear projection can be to convert the original vector to a new vector space through matrix multiplication. In the embodiments of the present application, it can be to map the semantic vector of the voice instruction to a space that matches the image feature vector. The obtained voice feature vector is usually in the same space as the image feature vector and can be meaningfully compared, fused, etc. with the image feature.

[0159] Step S144: Combine the image feature vector and the projected voice feature vector to perform cross-modal attention calculation to obtain the attention weight distribution.

[0160] In the above step S144, the formula for cross-modal attention calculation is:

[0161] ;

[0162] In the formula, Q i is the linear projection of the voice feature vector, K j [[ID= XVII]]is the linear projection of the image feature vector, and d is the dimension of the voice feature vector and the image feature vector.

[0163] Attention calculation can be regarded as a process of query (Query), key (Key), and value (Value). Specifically, the voice feature vector can be used as the query, the image feature vector as the key and value, calculate the similarity score between the query and the key, and use these scores to perform weighted summation on the value, so as to determine the degree of relevance between the operations mentioned in the voice instruction and each region in the image.

[0164] Step S145: Combine the attention weight distribution and the four-dimensional state tensor to verify the rationality of the atomic operation instruction or the atomic operation instruction sequence.

[0165] In the above step S145, on the one hand, the region in the image related to the instruction can be found through the attention weight distribution, and combined with the scene coordinate system information in the four-dimensional state tensor, it is confirmed whether the spatial position of the operation object is reasonable. For example, if the instruction is "add solution", check whether the attention weight is concentrated at the position of the virtual beaker.

[0166] On the other hand, the timestamp information in the four-dimensional state tensor can be used to verify whether the instruction conforms to the timing logic of the experiment. For example, it is reasonable to receive the "record data" instruction after the "heating stage".

[0167] On the third hand, according to the experimental object state information, it is verified whether the current state of the operation object is suitable for executing the instruction. For example, check whether the solution has reached a stable state that can be recorded.

[0168] In a fourth aspect, in combination with the information in the experimental stage, it is confirmed whether the instruction matches the current stage. For example, the "centrifugation" instruction is reasonable in the "mixing stage", but may not be reasonable in the "observation stage".

[0169] In the above implementation process, the specific position or object in the virtual experimental platform referred to by the current voice instruction (atomic operation instruction or atomic operation instruction sequence) can be determined through the attention weight distribution, so that the rationality of the atomic operation instruction or atomic operation instruction sequence can be verified, ensuring the rationality, feasibility, and safety of the instruction in the experimental environment, etc. This further improves the processing speed and accuracy of the voice, and thus improves the efficiency of voice control of the virtual platform.

[0170] Please refer to Figure 6 , Figure 6 is the specific flowchart of step S145 in the voice control method of the virtual platform provided by the embodiment of the present application. In some optional implementation manners, step S145 may include:

[0171] Step S1451: In combination with the attention weight distribution and the four-dimensional state tensor, verify the atomic operation instruction or atomic operation instruction sequence according to the natural laws.

[0172] Step S1452: Compare the object localization heat map corresponding to the attention weight distribution with the four-dimensional state tensor to verify whether the operation object corresponding to the atomic operation instruction or atomic operation instruction sequence is consistent with the actual object in the virtual platform screen.

[0173] In the above step S1452, the relevant region in the object localization heat map can be compared with the coordinate system in the four-dimensional state tensor to confirm whether the operation object in the instruction is consistent with the actual object in the virtual platform. For example, check whether the "add solution" instruction corresponds to the position of the beaker in the heat map.

[0174] In the above implementation process, by comparing the relevant region in the object localization heat map with the coordinate system in the four-dimensional state tensor, it is confirmed whether the operation object in the instruction is consistent with the actual object in the virtual platform, so as to verify the instruction. This further ensures the rationality, feasibility, and safety of the instruction in the experimental environment, etc. This further improves the processing speed and accuracy of the voice, and thus improves the efficiency of voice control of the virtual platform.

[0175] Please refer to Figure 7 , Figure 7 is a schematic diagram of the voice control system of the virtual platform provided by the embodiment of the present application. Based on the same concept, the embodiment of the present application provides a voice control system 700 for a virtual platform, including:

[0176] The voice input layer 710 is used to perform noise reduction and start and end point recognition on the acquired initial voice command to obtain a processed voice command;

[0177] The multi-language processing layer 720 is used to map the text converted from the processed voice instruction to a predefined atomic operation instruction set to obtain an initialization instruction;

[0178] The reflective neural network 730 is configured to generate an atomic operation instruction or an atomic operation instruction sequence according to the complexity of the initialization instruction; wherein the atomic operation instruction and the atomic operation instruction sequence are instructions that can be directly executed;

[0179] The spatiotemporal perception matrix 740 is used to verify the rationality of the atomic operation instruction or atomic operation instruction sequence in combination with the current virtualized content of the virtual platform and the laws of nature;

[0180] The execution layer 750 is used to execute the verified atomic operation instruction or atomic operation instruction sequence.

[0181] It should be understood that this system corresponds to the aforementioned embodiment of the voice control method for a virtual platform and is capable of executing each of the steps involved in the aforementioned method embodiment. The specific functions of this system can be found in the description above; to avoid repetition, a detailed description is omitted here. The system can include at least one software functional module that can be stored in a memory in the form of software or firmware or embedded in the device's operating system (OS).

[0182] The present application also provides a storage medium including a computer-readable storage medium having a computer program stored thereon, which executes the above method when executed by a processor.

[0183] Among them, the computer-readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM for short), electrically erasable programmable read-only memory (EEPROM for short), erasable programmable read-only memory (EPROM for short), programmable read-only memory (PROM for short), read-only memory (ROM for short), magnetic memory, flash memory, magnetic disk or optical disk.

[0184] In several embodiments provided by the embodiments of the present application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are only illustrative. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architectures, functions, and operations of devices, methods, and computer program products according to multiple embodiments of the embodiments of the present application. In this regard, each block in the flowchart or block diagram may represent a module, a program segment, or a part of the code. A module, a program segment, or a part of the code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order from that marked in the accompanying drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, as well as the combination of blocks in the block diagram and / or flowchart, can be implemented by a dedicated hardware-based system that performs the specified functions or actions, or can be implemented by a combination of dedicated hardware and computer instructions.

[0185] In addition, in each embodiment of the embodiments of the present application, the various functional modules can be integrated together to form an independent part, or each module can exist alone, or two or more modules can be integrated to form an independent part.

[0186] The above description is only an optional implementation manner of the embodiments of the present application, but the protection scope of the embodiments of the present application is not limited thereto. Any person skilled in the art within the technical scope disclosed by the embodiments of the present application can easily think of changes or substitutions, which should all be covered within the protection scope of the embodiments of the present application.

Claims

1. A voice control method for a virtual platform, characterized in that: include: Perform noise reduction and start and end point recognition on the acquired initial voice command to obtain a processed voice command; Mapping the text converted from the processed voice instruction to a predefined atomic operation instruction set to obtain an initialization instruction; Generate an atomic operation instruction or an atomic operation instruction sequence according to the complexity of the initialization instruction; wherein the atomic operation instruction and the atomic operation instruction sequence are instructions that can be directly executed; Verifying the rationality of the atomic operation instruction or atomic operation instruction sequence based on the current virtualized content of the virtual platform and natural laws; and Executing the verified atomic operation instruction or atomic operation instruction sequence; Generating an atomic operation instruction or an atomic operation instruction sequence according to the complexity of the initialization instruction includes: The instruction complexity of the initialization instruction is calculated by a reflective neural network; wherein the reflective neural network includes a fast response channel and a complex instruction processing channel; the calculation formula of the instruction complexity is as follows: ； Wherein, lexical_complexity represents the lexical complexity of the initialization instruction, syntactic_depth represents the syntactic depth of the initialization instruction, and Score represents the instruction complexity; Determining whether the complexity of the instruction exceeds a complexity threshold; If it is determined that the number of times the initialization instruction is exceeded, the complex instruction processing channel receives the initialization instruction and the cached historical operation information, and performs embedding layer conversion on the initialization instruction to output an embedding vector; The relative position code of the initialization instruction is calculated using a position code formula; wherein the position code formula is: ； Where k represents the dimension index, d represents the embedding dimension, i and j represent the sequence bits of the two elements in the initialization instruction respectively; fusing the embedding vector with the relative position code to obtain a fused embedding vector; Processing the fused embedding vector using a multi-head attention mechanism to obtain a context-aware vector; and The context-aware vector is combined with the historical operation information to generate the atomic operation instruction or atomic operation instruction sequence.

2. The method according to claim 1, characterized in that The noise reduction and start and end point recognition of the acquired initial voice command to obtain a processed voice command includes: Representing the initial voice command as a time domain signal; and The noise component in the initial voice command is removed by using spectral subtraction; wherein the implementation formula of the spectral subtraction is: ； Where f represents frequency, X clean (f) represents the pure voice signal after the noise component of the initial voice command is removed, X(f) represents the frequency domain signal after the time domain signal is Fourier transformed, N(f) represents the preset room environment noise template; α is the over-reduction factor, β is the spectrum lower limit protection coefficient, e jφx（f） Indicates phase information of the initial voice command.

3. The method according to claim 2, characterized in that The step of performing noise reduction and start and end point recognition on the acquired initial voice instruction to obtain a processed voice instruction further includes: Converting the frequency spectrum of the frequency domain signal after inverse Fourier transform into a time domain signal; Determine whether the time domain signal satisfies and Whether the number of consecutive frames exceeds a first frame number threshold; If the condition is satisfied, the first frame in the continuous number of frames is used as the starting point of the voice command; Determine whether the time domain signal satisfies and Whether the number of consecutive frames exceeds a second frame number threshold; If the condition is satisfied, the last frame in the continuous number of frames is used as the end point of the voice command; Among them, E n With Z n Calculated by the following formulas: ； ； Where, E n represents the short-time energy of the nth frame in the time domain signal, L represents the frame length, x(k) represents the amplitude of the kth sampling point, Z n represents the zero-crossing rate of the nth frame in the time domain signal, and I represents the judgment function used to judge Is it true? μE and μZ represent the mean of the short-time energy and the mean of the zero-crossing rate of the noise component respectively; σE and σZ represent the standard deviation of the short-time energy and the standard deviation of the zero-crossing rate of the noise component.

4. The method according to claim 1, wherein Mapping the text converted from the processed voice instruction to a predefined atomic operation instruction set to obtain an initialization instruction includes: performing frame processing on the processed voice command and inputting the frame into a pre-built semantic feature extraction module to generate a phoneme sequence; Inputting the phoneme sequence into a pre-built semantic feature alignment module to generate a semantic vector; and Performing normalized mapping on the semantic vector to obtain the initialization instruction; wherein the normalized mapping formula is: ； Where z 标 Indicates the initialization instruction, h CLS is the semantic vector, z is the predefined atomic operation instruction set, z i is an element in the atomic operation instruction set, E(z i ) represents the pre-trained instruction embedding vector.

5. The method according to claim 1, wherein Generating an atomic operation instruction or an atomic operation instruction sequence according to the complexity of the initialization instruction further includes: If it is determined that the threshold is not exceeded, the fast response channel performs a convolution operation based on the Mel-frequency cepstral coefficients of the initialization instruction to capture key acoustic modes; Performing global pooling dimensionality reduction on the initialization instructions that have undergone the convolution operation, and outputting the probability distribution of elements in the atomic operation instruction set using an activation function; and The atomic operation instruction or atomic operation instruction sequence is generated according to the probability distribution.

6. The method according to claim 1, characterized in that The checking of the rationality of the atomic operation instruction or atomic operation instruction sequence in combination with the current virtualized content of the virtual platform and the laws of nature includes: Construct a four-dimensional state tensor for describing the virtual platform screen; wherein the tensor formula is: ； Wherein, x, y, z represent the scene coordinate system of the virtual platform screen, t represents the timestamp, v represents the state of the operation object corresponding to the atomic operation instruction or atomic operation instruction sequence, c p It represents the operation phases of the operation process; r represents the constraint rules; Extracting image features of the virtual platform screen to obtain an image feature vector; Performing linear projection on the atomic operation instruction or the atomic operation instruction sequence to obtain a projected speech feature vector; Combining the image feature vector and the projected speech feature vector, a cross-modal attention calculation is performed to obtain an attention weight distribution; wherein the formula for cross-modal attention calculation is: ； Where Q i is the linear projection of the speech feature vector, K j is the linear projection of the image feature vector, and d is the dimension of the speech feature vector and the image feature vector; In combination with the attention weight distribution and the four-dimensional state tensor, the rationality of the atomic operation instruction or the atomic operation instruction sequence is verified.

7. The method according to claim 6, characterized in that The checking of the rationality of the atomic operation instruction or the atomic operation instruction sequence in combination with the attention weight distribution and the four-dimensional state tensor includes: combining the attention weight distribution and the four-dimensional state tensor, and verifying the atomic operation instruction or atomic operation instruction sequence according to the natural law; and By comparing the object positioning heat map corresponding to the attention weight distribution with the four-dimensional state tensor, it is verified whether the operation object corresponding to the atomic operation instruction or the atomic operation instruction sequence is consistent with the actual object in the virtual platform screen.

8. A voice control system for a virtual platform, characterized in that: include: The voice input layer is used to perform noise reduction and start and end point recognition on the initial voice commands obtained to obtain processed voice commands; A multi-language processing layer, configured to map the text converted from the processed voice instruction to a predefined atomic operation instruction set to obtain an initialization instruction; A reflective neural network is used to generate an atomic operation instruction or an atomic operation instruction sequence according to the complexity of the initialization instruction; wherein the atomic operation instruction and the atomic operation instruction sequence are instructions that can be directly executed; A spatiotemporal perception matrix, used to verify the rationality of the atomic operation instruction or atomic operation instruction sequence in combination with the current virtualized content of the virtual platform and the laws of nature; An execution layer, configured to execute the verified atomic operation instruction or atomic operation instruction sequence; In the process of generating an atomic operation instruction or an atomic operation instruction sequence according to the complexity of the initialization instruction, the reflective neural network is specifically used to calculate the instruction complexity of the initialization instruction by the reflective neural network; wherein the reflective neural network includes a fast response channel and a complex instruction processing channel; the calculation formula of the instruction complexity is as follows: ； Wherein, lexical_complexity represents the lexical complexity of the initialization instruction, syntactic_depth represents the syntactic depth of the initialization instruction, and Score represents the instruction complexity; it is determined whether the instruction complexity exceeds the complexity threshold; if it is determined that it exceeds, the complex instruction processing channel receives the initialization instruction and the cached historical operation information, and performs embedding layer conversion on the initialization instruction to output an embedding vector; the relative position encoding of the initialization instruction is calculated using the position encoding formula; wherein, the position encoding formula is: ； Wherein, k represents the dimension index, d represents the embedding dimension, i and j represent the sequence bits of two elements in the initialization instruction respectively; the embedding vector is fused with the relative position encoding to obtain a fused embedding vector; the fused embedding vector is processed using a multi-head attention mechanism to obtain a context-aware vector; and the context-aware vector is combined with the historical operation information to generate the atomic operation instruction or atomic operation instruction sequence.

9. A storage medium, characterized in that: The storage medium comprises a computer-readable storage medium; a computer program is stored on the computer-readable storage medium, and the computer program executes the method according to any one of claims 1 to 7 when executed by a processor.

Images