Language model-based instruction execution method and device, computer device and program

Abstract

This application provides a method and apparatus for executing commands based on a language model, a computer device, and a program. When request information of a target object is received, a language model is first invoked to perform command recognition on the request information based on guide presentation information to obtain candidate execution commands. A feasibility check is then performed on the candidate execution commands. If the candidate execution command fails the feasibility check, the language model is invoked to perform command recognition on the request information based on the candidate execution commands and guide presentation information to obtain new candidate execution commands. A feasibility check is then performed on the new candidate execution command again until the feasibility check passes. The candidate execution command that passes the feasibility check is then used as a target execution command. The terminal is then controlled to execute an operation task corresponding to the request information based on the target execution command. The embodiments of this application can improve interaction efficiency and user experience. The embodiments of this application can be applied to various scenarios involving interaction with AI.

Description

[Technical Field]

[0001] This application claims priority to a Chinese patent application filed with the China Patent Office on June 30, 2023, bearing application number 2023108010693 and entitled "Method, device and storage medium for executing instructions based on language model," the entire contents of which are incorporated herein by reference.

[0002] The present application relates to the technical field of artificial intelligence (AI), and in particular to natural language processing technology. [Background technology]

[0003] With the continuous development of AI technology, many terminal devices support user interaction. For example, users can interact with terminal devices through voice or text. When the terminal device recognizes the user's request, it can perform different operations based on the recognized request, such as opening an application, checking the weather, playing music, etc.

[0004] Terminal devices generally recognize user request commands based on neural network models. However, neural network models cannot always accurately recognize user request commands. When the neural network model cannot accurately recognize the user request command, the user needs to interact with the terminal device again by using another description of the request command until the neural network model can accurately recognize the user request command. In this process, the user needs to interact with the terminal device multiple times to ensure that the terminal device can accurately perform the user's desired operation. Therefore, problems such as low interaction efficiency and poor user experience exist. Summary of the Invention [Problem to be solved by the invention]

[0005] The embodiments of the present application aim to provide a method and apparatus for executing instructions based on a language model, a computer device, and a program, which can improve interaction efficiency and thereby improve user experience. [Means for solving the problem]

[0006] According to one aspect, an embodiment of the present application provides a method for executing instructions based on a language model, which is executed by a computer device and includes the following steps: Receive target request information; Execute at least one round of target operations until a candidate execution instruction that passes a feasibility check is obtained, and for a first round of target operations, invoke the language model to perform command recognition on the request information based on predetermined guide prompt information to obtain candidate execution instructions for the target operation of the first round, and perform the feasibility check on the candidate execution instructions for the target operation of the first round to determine whether the candidate execution instructions for the target operation of the first round pass the feasibility check; for an i-th round of target operations, invoke the language model to perform command recognition on the request information based on the candidate execution instructions for the target operation of the previous i-1 rounds and the guide prompt information to obtain candidate execution instructions for the target operation of the i-th round, and perform the feasibility check on the candidate execution instructions for the target operation of the i-th round to determine whether the candidate execution instructions for the target operation of the i-th round pass the feasibility check, where i is an integer greater than 1; and A step of determining the candidate execution command that has passed the feasibility check as a target execution command, and controlling the terminal to execute the operation task corresponding to the request information based on the target execution command.

[0007] According to another aspect, an embodiment of the present application further provides an instruction execution device based on a language model, which includes: an information receiving unit for receiving request information of the target object; an instruction processing unit for executing at least one round of target operations until a candidate execution instruction that passes a feasibility check is obtained, wherein for a first round of target operations, the language model is invoked to perform instruction recognition on the request information based on predetermined guide presentation information to obtain a candidate execution instruction for the target operation of the first round, and the feasibility check is performed on the candidate execution instruction for the target operation of the first round to determine whether the candidate execution instruction for the target operation of the first round passes the feasibility check; for an i-th round of target operations, the language model is invoked to perform instruction recognition on the request information based on the candidate execution instruction for the target operation of a previous i-1 round and the guide presentation information to obtain a candidate execution instruction for the target operation of the i-th round, and the feasibility check is performed on the candidate execution instruction for the target operation of the i-th round to determine whether the candidate execution instruction for the target operation of the i-th round passes the feasibility check, wherein i is an integer greater than 1; and The candidate execution command that passes the feasibility check is set as a target execution command, and an instruction execution unit is included for controlling the terminal to execute the operation task corresponding to the request information based on the target execution command.

[0008] According to another aspect, embodiments of the present application further provide a computer device, comprising: at least one processor; and at least one memory coupled to the processor; The storage device stores at least one program, At least one of said programs, when executed by at least one of said processors, implements a language model based instruction execution method as described above.

[0009] According to another aspect, an embodiment of the present application further provides a computer-readable storage medium having stored therein a processor-executable computer program, the processor-executable computer program realizing the language model-based instruction execution method as described above when executed by a processor.

[0010] According to another aspect, an embodiment of the present application further provides a computer program product, which includes a computer program or computer instructions, the computer program or the computer instructions being stored in a computer-readable storage medium, wherein a processor of a language model-based instruction execution device reads the computer program or the computer instructions from the computer-readable storage medium, and the processor executes the computer program or the computer instructions, thereby causing the instruction execution device to perform the language model-based instruction execution method as described above. [Effects of the Invention]

[0011] The embodiments of the present application have at least the following advantageous effects: when receiving request information of a target object, at least one round of target operations is executed until a candidate execution command that passes the feasibility check is obtained; in the first round of target operations, a language model is first invoked to perform command recognition on the request information based on predetermined guide presentation information to obtain candidate execution commands for the target operation of the first round; the candidate execution commands are obtained by the language model performing command recognition on the request information based on the guide presentation information, which improves the reliability of the candidate execution commands, thereby helping to obtain candidate execution commands that are more closely matched with the request information; and then a feasibility check is performed on the candidate execution commands for the target operation of the first round to obtain candidate execution commands. Determine whether the candidate execution command passes the feasibility check. If the candidate execution command does not pass the feasibility check, it indicates that the candidate execution command obtained by the language model recognition at the current time (in this round) is not accurate. At this time, continue to perform the target operation of the next round. In the target operation of the next round, call the language model to perform command recognition again on the request information based on the candidate execution command in the previous target operation and the guide presentation information to obtain the candidate execution command for the target operation of this round. Then, perform a feasibility check on the candidate execution command again to determine whether the candidate execution command passes the feasibility check. In this way, several rounds of target operations are repeatedly performed until a candidate execution command that passes the feasibility check is obtained.Furthermore, after obtaining the candidate execution command that passes the feasibility check, the candidate execution command is taken as the target execution command, and the terminal is controlled to execute the operation task corresponding to the request information based on the target execution command. The candidate execution command that passes the feasibility check is obtained by calling the language model multiple times to perform command recognition on the request information based on the guide presentation information and the previous candidate execution command, so the reliability of the candidate execution command can be guaranteed. In addition, this process is completed automatically, and does not require the target object to perform any additional operations, nor does it require the target object to perform multiple repeated interactions. This can improve the interaction efficiency between the target object and the terminal and achieve the purpose of simplifying the operation (use) of the target object, thereby improving the usage experience (user experience) of the target object. [Brief explanation of the drawings]

[0012] [Figure 1] FIG. 1 illustrates an implementation environment provided in an embodiment of the present application. [Figure 2] FIG. 1 illustrates another implementation environment provided in an embodiment of the present application. [Figure 3] 1 is a flowchart of a language model-based instruction execution method provided in an embodiment of the present application; [Figure 4] FIG. 10 is a diagram showing the configuration of guide presentation information provided in an embodiment of the present application. [Figure 5] FIG. 10 is a diagram illustrating a process of replacing an instruction function function with an empty function provided in an embodiment of the present application. [Figure 6] FIG. 2 is a diagram illustrating an interface provided in one embodiment of the present application for interacting with a terminal through an artificial intelligence interaction module; [Figure 7] FIG. 10 is a diagram showing the contents of guide presentation information provided in an embodiment of the present application. [Figure 8] FIG. 10 is a diagram illustrating an interface provided in another embodiment of the present application for interacting with a terminal through an artificial intelligence interaction module; [Figure 9]1 is a flowchart of an overall language model-based instruction execution method provided in one example of the present application. [Figure 10] 1 is a flowchart of steps of a language model-based instruction execution method provided in one specific example of the present application. [Figure 11] FIG. 1 illustrates a language model-based instruction execution device provided in an embodiment of the present application. [Figure 12] FIG. 10 is a diagram illustrating another language model-based instruction execution device provided in an embodiment of the present application. DETAILED DESCRIPTION OF THE INVENTION

[0013] The present application will be further described below in conjunction with the specification, drawings and specific examples. The described examples do not limit the present application. In addition, all other examples that a person skilled in the art can obtain without any creative effort fall within the scope of protection of the present application.

[0014] When a user uses a terminal device to perform an operation task, the user generally needs to manually operate the terminal device to complete the operation task, such as manually dragging a window, manually opening an application, manually entering a query request into a browser to obtain search content, etc. Since all of these operations need to be performed manually by the user, there are problems such as a low level of interaction intelligence and low interaction efficiency; in addition, when the operation path of the operation task is relatively long, the user needs to perform relatively complicated operation procedures to complete the operation task, which results in a poor user experience.

[0015] To solve these problems, related technologies have proposed a method for realizing interaction between a user and a terminal device through voice or text, whereby the terminal device recognizes the request command issued by the user during the interaction process and can perform different operation tasks, such as opening an application, checking the weather, or playing music, based on the recognized request command. The related technologies generally employ a neural network model to recognize the user's request command. However, because the output result of the neural network model is probabilistic, the neural network model cannot accurately recognize the user's request command every time. When the neural network model cannot accurately recognize the user's request command, the user needs to use another way of describing the request command and interact with the terminal device again. For example, if the neural network model cannot accurately recognize the user's voice command "play music," the user needs to interact with the terminal device again using the voice command "play music" until the neural network model can accurately recognize the user's request command. Therefore, it can be seen that in this process, the user needs to interact with the terminal device multiple times in order to make the terminal device accurately perform the operation task desired by the user, which results in problems such as low interaction efficiency and poor user experience.

[0016] In order to improve the interaction efficiency and user experience between a user and a terminal device, an embodiment of the present application proposes a command execution method based on a language model. In this method, when receiving request information of a target object, at least one round of target operations is executed until a candidate execution command that passes a feasibility check is obtained. In the first round of target operations, a language model is first invoked to perform command recognition on the request information based on predetermined guide presentation information to obtain candidate execution commands for the target operation of the first round. The candidate execution commands are obtained by the language model performing command recognition on the request information based on the guide presentation information, which can improve the reliability of the candidate execution commands, thereby helping to obtain candidate execution commands that are more closely matched with the request information. Next, a feasibility check is performed on the candidate execution commands for the target operation of the first round to determine whether the candidate execution commands are feasible. It is determined whether the check has passed, and if the candidate execution command does not pass the feasibility check, it indicates that the candidate execution command currently (in this round) obtained by recognition by the language model is not accurate, at this time, the target operation of the next round is continued, and in the target operation of the next round, the language model is called to perform command recognition again on the request information based on the candidate execution command in the previous target operation and the guide presentation information to obtain the candidate execution command in the target operation of this round, and then a feasibility check is performed again on the candidate execution command to determine whether the candidate execution command has passed the feasibility check, and in this way, several rounds of target operations are repeatedly performed until a candidate execution command that has passed the feasibility check is obtained.Furthermore, after obtaining a candidate execution command that passes the feasibility check, the candidate execution command is taken as a target execution command, and the terminal is controlled to execute the operation task corresponding to the request information based on the target execution command. The candidate execution command that passes the feasibility check is obtained by calling the language model multiple times to perform command recognition on the request information based on the guide presentation information and the previous candidate execution command, so the reliability of the candidate execution command can be guaranteed. In addition, this process is completed automatically, and does not require the target object to perform any additional operations, nor does it require the target object to perform multiple repeated interactions. This can improve the interaction efficiency between the target object and the terminal and achieve the purpose of simplifying the operation of the target object, thereby improving the user experience of the target object.

[0017] Referring to Figure 1, Figure 1 is a diagram illustrating an implementation environment provided in an embodiment of the present application. The implementation environment may include a first user terminal 110 and a first server 120, and the first user terminal 110 and the first server 120 may be directly or indirectly connected by wire or wireless communication. Wherein, the first user terminal 110 and the first server 120 may be nodes in a blockchain, which is not limited in this embodiment.

[0018] The first server 120 may be an independent physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDNs (Content Delivery Networks), big data, and artificial intelligence platforms. A trained language model may be deployed on the first server 120, and the language model may be a large language model (LLM) that can understand and learn human language to interact, or a natural language understanding model (NLU) that can perform semantic recognition on speech information or text information input by a user.

[0019] The first user terminal 110 may include, but is not limited to, a smartphone, a tablet computer, a smart voice interaction device, a smart home appliance, an in-vehicle terminal, an aircraft, etc. Optionally, the first user terminal 110 may be provided with an artificial intelligence interaction module 111, which can realize interaction between a user and the first user terminal 110, so that the first user terminal 110 can automatically perform an operation task desired by the user based on request information input by the user, for example, the artificial intelligence interaction module 111 can call a language model in the first server 120 to recognize the request information input by the user and obtain an execution command corresponding to the request information, so that the first user terminal 110 can perform an operation task corresponding to the request information based on the execution command.

[0020] Referring to Figure 2, Figure 2 is a diagram illustrating another implementation environment provided in an embodiment of the present application, which may include a second user terminal 210, where the second user terminal 210 may be a node in a blockchain, which is not limited in this embodiment.

[0021] The second user terminal 210 may include, but is not limited to, a smartphone, a tablet computer, a smart voice interaction device, a smart home appliance, an in-vehicle terminal, an aircraft, etc. Optionally, the second user terminal 210 may be provided with an artificial intelligence interaction module 211, and a trained language model may be deployed in the second user terminal 210, which may be an LLM model capable of interacting by understanding and learning human language, or an NLU model capable of performing semantic recognition on voice information or text information input by a user. The artificial intelligence interaction module 211 can realize interaction between a user and the second user terminal 210, so that the second user terminal 210 can automatically perform an operation task desired by the user based on request information input by the user. For example, the artificial intelligence interaction module 211 can call the language model in the second user terminal 210 to recognize the request information input by the user and obtain an execution command corresponding to the request information, so that the second user terminal 210 can perform an operation task corresponding to the request information based on the execution command.

[0022] As shown in Figure 1 or Figure 2, in one application scenario, after the first user terminal 110 or the second user terminal 210 receives request information input by a user, the first user terminal 110 can use the artificial intelligence interaction module 111 to call the language model in the first server 120 and perform command recognition on the request information based on the specified guide presentation information to obtain candidate execution commands, and the second user terminal 210 can use the artificial intelligence interaction module 211 to call the language model in the second user terminal 210 and perform command recognition on the request information based on the specified guide presentation information to obtain candidate execution commands.

[0023] Wherein, when a user inputs voice information, the first user terminal 110 or the second user terminal 210 first invokes a pre-trained voice recognition model to perform voice recognition on the voice information to obtain text information, and then performs at least one of adding punctuation, correcting spelling, and correcting word order on the text information to obtain request information. In addition, when the first user terminal 110 or the second user terminal 210 invokes a pre-trained voice recognition model to perform voice recognition on the voice information, it first performs at least one of noise removal and voice segmentation on the voice information to obtain pre-processed voice information, then invokes a pre-trained voice feature extraction model to perform voice feature extraction on the pre-processed voice information to obtain voice feature information, and then invokes the pre-trained voice recognition model to perform voice recognition on the voice feature information to obtain text information.

[0024] When the first user terminal 110 or the second user terminal 210 calls a language model and performs command recognition on the request information based on specified guide presentation information to obtain candidate execution commands, it first generates model call request information based on the request information and the specified guide presentation information, and then sends the model call request information to the language model call interface in the first server 120 or the language model call interface in the second user terminal 210, thereby calling the language model based on the model call request information and performing command recognition on the request information based on the guide presentation information to obtain candidate execution commands.

[0025] After the first user terminal 110 or the second user terminal 210 obtains the candidate execution command recognized by the language model, the first user terminal 110 or the second user terminal 210 first performs a feasibility check on the candidate execution command to obtain a feasibility check result. When the first user terminal 110 or the second user terminal 210 performs a feasibility check on the candidate execution command, it first replaces the instruction function function in the candidate execution command with a null function whose function result is always true to obtain a check-awaiting execution command, and then invokes the instruction execution engine to simulate the execution of the check-awaiting execution command to obtain the simulated execution result. If the simulated execution result is "the check-awaiting execution command is successfully executed", the feasibility check result indicates that the candidate execution command has passed the feasibility check; if the simulated execution result is "the check-awaiting execution command is not successfully executed", the feasibility check result indicates that the candidate execution command has not passed the feasibility check.

[0026] After the first user terminal 110 or the second user terminal 210 performs a feasibility check on the candidate execution command and obtains a feasibility check result, if the feasibility check result is "the candidate execution command fails to pass the feasibility check", it calls the language model in the first server 120 or the second user terminal 210 to perform command recognition on the request information again based on the candidate execution command and the guide presentation information to obtain a new candidate execution command, and then performs a feasibility check on the new candidate execution command again, and thus repeats the above process until it obtains a candidate execution command that passes the feasibility check.

[0027] Subsequently, the first user terminal 110 or the second user terminal 210 sets the candidate execution command that passes the feasibility check as a target execution command, and the AI ​​interaction module 111 in the first user terminal 110 or the AI ​​interaction module 211 in the second user terminal 210 controls the first user terminal 110 or the second user terminal 210 to execute the operation task corresponding to the request information based on the target execution command. When the first user terminal 110 or the second user terminal 210 executes the operation task corresponding to the request information based on the target execution command, it first obtains a corresponding command execution code based on the target execution command, and then executes the command execution code in a virtual environment, thereby simulating the user's control to make the user terminal execute the operation task corresponding to the request information. Among them, in the process in which the first user terminal 110 or the second user terminal 210 executes an instruction execution code in a virtual environment to simulate the user controlling the user terminal to perform the operation task corresponding to the request information, the first user terminal 110 or the second user terminal 210 first executes an instruction execution process in the virtual environment, and then calls the instruction execution process to execute the instruction execution code, thereby simulating the user controlling the user terminal to perform the operation task corresponding to the request information.

[0028] If the first user terminal 110 or the second user terminal 210 includes a display module for displaying an operation cursor, when simulating a user controlling the user terminal to perform an operation task corresponding to the request information, the first user terminal 110 or the second user terminal 210 can first invoke an application programming interface to obtain the screen resolution of the display module, obtain the initial coordinates of the operation cursor based on the screen resolution, then determine the target coordinates when performing the operation task corresponding to the request information, and then simulate the user controlling the operation cursor to move from the initial coordinates to the target coordinates, and then simulate the user controlling the user terminal at the target coordinates to perform the operation task corresponding to the request information.

[0029] In addition, in the process in which the artificial intelligence interaction module 111 or the artificial intelligence interaction module 211 controls the user terminal to execute the operation task corresponding to the request information based on the target execution command, an execution record table can be further generated, which includes the target execution command and the command execution code, and the execution record table further includes an execution state table entry and an execution result table entry. When the first user terminal 110 or the second user terminal 210 completes the execution of the operation task corresponding to the request information, the first user terminal 110 or the second user terminal 210 can update the content of the execution state table entry in the execution record table to completed, and write the execution result into the execution result table entry.

[0030] In addition, in each specific implementation of the present application, when performing related processing based on data related to the characteristics of a target object (e.g., a user), such as attribute information or attribute information sets of the target object (characteristic-related data), the permission or consent of the target object must be obtained in advance, and the collection, use, processing, etc. of such data must all comply with relevant laws, regulations, and standards. In addition, in the embodiments of the present application, when obtaining attribute information of a target object, the individual permission or consent of the target object must be obtained by a method such as a pop-up window or jumping to a confirmation page, and only after the individual permission or consent of the target object has been clearly obtained can the relevant data of the target object be obtained so that the embodiments of the present application can be normally implemented.

[0031] 3 is a flowchart of a language model-based instruction execution method provided in an embodiment of the present application. The language model-based instruction execution method may be executed by a computer device, for example, by a terminal, or may be executed in cooperation with a terminal and a server. In the embodiment of the present application, the method is described as being executed by a terminal. As shown in FIG. 3, the language model-based instruction execution method may include, but is not limited to, steps 310 to 330.

[0032] Step 310: Receive request information from a target object.

[0033] In one embodiment, the request information of the target object may be a direct control command of the target object to the terminal, or a question given by the target object to the terminal to cause the terminal to provide a corresponding answer, and is not limited thereto. For example, when the request information is a direct control command for the target object to the terminal, it may be a command content that directly controls the terminal, such as "play music," "open a browser," or "set an alarm for 10 AM." Alternatively, when the request information is a question content that the target object gives to the terminal to make the terminal provide a corresponding answer, it may be a question content such as "what's the weather going to tomorrow?", "how do I get to location A?", or "what buses are there to city B?", where the request information "what's the weather going to tomorrow?" means that the target object wants the terminal to check tomorrow's weather through a weather forecast website, the request information "how do I get to location A" means that the target object wants the terminal to open a map application to check a navigation route from the current location to location A, and the request information "what buses are there to city B?" means that the target object wants the terminal to check the passenger transport schedule from the current city to city B through a passenger transport website.

[0034] In one embodiment, the target object may be a user or an external terminal communicatively connected to the terminal executing the instruction execution method, but this is not a limitation. When the target object is a user, the request information of the target object may be text information directly input by the user, or text information obtained by recognizing voice information input by the user, but this is not a limitation. When the target object is an external terminal communicatively connected to the terminal executing the instruction execution method, the request information of the target object may be text information transmitted by the external terminal, or text information obtained by recognizing voice information transmitted by the external terminal, but this is not a limitation.

[0035] In one embodiment, when the request information is text information obtained by recognition based on the target object's voice information (e.g., voice information input by a user or an external terminal), the terminal can first receive the target object's input voice information, then invoke a pre-trained voice recognition model to perform voice recognition on the voice information to obtain text information, and then perform at least one of adding punctuation marks, correcting spelling, and correcting word order on the text information to obtain the request information.

[0036] Among them, the pre-trained speech recognition model is used to convert the voice frequency feature vector of the voice information into text information, and the speech recognition model may include models commonly used in the field, such as a Hidden Markov Model (HMM), a Gaussian mixture model, a deep neural network model, an n-gram language model, and other statistical models, but is not limited thereto herein.

[0037] In one embodiment, the speech recognition model may be further obtained by serially connecting an acoustic model and a language model, in which the acoustic model is used to classify acoustic features to correspond to (decoded) phonemes or words, and the language model is used to decode the phonemes or words into a complete text. For example, the acoustic model first converts the voice frequency feature vector of the speech information into intermediate recognition results (e.g., phonemes, phoneme strings, subwords, etc.), and then the language model converts the intermediate recognition results into text recognition results (e.g., words, word strings, code (symbol) sequences, etc.), and then outputs text information corresponding to the target speech information.

[0038] In one embodiment, the text information output from the speech recognition model may contain words, word strings, code sequences, etc., and therefore may not accurately represent the target object's request information, which may affect the accuracy of subsequent command recognition of the request information. To address this issue, after obtaining the text information output from the speech recognition model, the text information may be subjected to at least one of adding punctuation, correcting spelling, and correcting word order to obtain more accurate request information. For example, if the target object's input speech information is "I want to know what the weather will be like tomorrow. How can I find out what the weather will be like tomorrow?", the text information obtained after speech recognition using the speech recognition model may be "I want to know what the weather will be like tomorrow. How can I find out what the tianqi is tomorrow?". The text information lacks punctuation to separate sentences and contains spelling errors and word order errors, which may affect the accuracy of subsequent recognition of the text information. Therefore, in this case, punctuation may be added to the text information first, and the text information after the punctuation is added becomes "I want to know what the weather will be like tomorrow. How can I find out what the weather will be like tomorrow?" Next, the spelling mistake "Tianqi" in the text information is corrected to obtain the correct spelling "weather", and the text information after the spelling correction becomes "I want to know what the weather will be like tomorrow. How can I find out what the weather will be like tomorrow?" Next, the word order error in the text information "I want to know what the weather will be like tomorrow" is corrected, and the obtained correct word order becomes "I want to know what the weather will be like tomorrow", and at this time, the correct request information "I want to know what the weather will be like tomorrow. How can I find out what the weather will be like tomorrow?" can be obtained.

[0039] In this way, by using the above-mentioned method, speech recognition is performed on the speech information input by the target object to obtain text information, and then corresponding adjustment and correction processing is performed on the text information, thereby ensuring that the final text information accurately corresponds to the request information of the target object, and thus ensuring the accuracy of subsequent command recognition for the request information.

[0040] In one embodiment, when a pre-trained speech recognition model is called to perform speech recognition on the speech information to obtain text information, interference information such as environmental noise may be mixed in when receiving the speech information input by the target subject, or the speech information input by the target subject may be long, all of which may affect the speech recognition effect of the speech recognition model. In order to improve the accuracy of speech recognition by the speech recognition model, when calling the pre-trained speech recognition model to perform speech recognition on the speech information, the speech information may first be subjected to at least one of noise reduction and speech segmentation to obtain pre-processed speech information, then a speech feature extraction model may be called to perform speech feature extraction on the pre-processed speech information to obtain speech feature information, and then the pre-trained speech recognition model may be called to perform speech recognition on the speech feature information to obtain text information.

[0041] Among these, noise removal of speech information can filter out interference information such as environmental noise mixed in the speech information, thereby improving the purity of the speech information, and speech segmentation of speech information can segment a long speech input by a target subject into multiple short speeches, thereby reducing the processing difficulty when the speech feature extraction model subsequently extracts speech features, thereby improving the accuracy and efficiency of speech feature extraction.

[0042] In one embodiment, the purpose of invoking the audio feature extraction model to extract audio features from the pre-processed audio information is to convert the pre-processed audio information into audio feature information in the form of a digital signal (e.g., an audio frequency feature vector) so that the audio recognition model can subsequently perform more accurate recognition processing on the audio feature information. The audio feature extraction model may be a trained deep neural network model, a convolutional neural network model, or a mathematical model capable of audio feature extraction. An appropriate selection may be made according to actual application circumstances, and is not limited herein. For example, when the audio feature extraction model is a mathematical model capable of audio feature extraction, the audio feature extraction model may be a mathematical model capable of performing Mel Frequency Cepstral Coefficients (MFCC) processing on the pre-processed audio information, a mathematical model capable of performing Discrete Wavelet Transform (DWT) processing on the pre-processed audio information, or a mathematical model capable of performing Perceptual Linear Prediction (PLP) processing on the pre-processed audio information, and is not limited herein. Among them, MFCC processing involves obtaining MFCCs from pre-processed audio information. MFCCs are commonly used audio features and are used to represent the frequency spectrum characteristics of audio signals. MFCCs have fairly good human perception attributes and can effectively extract important features of audio signals. DWT processing is a signal processing method based on wavelet analysis that can be used for feature extraction and noise reduction of audio signals. PLP processing is a method for audio feature extraction that takes into account the physiological and psychological characteristics of audio signals based on the way the human ear perceives sound, effectively extracting important features of audio signals.

[0043] In one embodiment, the audio feature extraction model may further be a mathematical model that performs a combined process of MFCC and PLP on the pre-processed audio information. In such a case, when the audio feature extraction model is called to perform audio feature extraction on the pre-processed audio information, the audio feature information of the pre-processed audio information can be obtained by first performing MFCC processing and PLP processing on the pre-processed audio information, respectively, to obtain the MFCC and PLP feature parameters of the pre-processed audio information, and then obtaining the average values ​​of the MFCC and PLP feature parameters of the pre-processed audio information.

[0044] When the audio feature extraction model is called to perform MFCC processing on the pre-processed audio information, a fast Fourier transform is first performed on the time-domain signal of the pre-processed audio information to obtain a frequency-domain signal of the pre-processed audio information, a logarithmic operation is then performed on the frequency-domain signal of the pre-processed audio information to obtain a logarithmic spectrum of the pre-processed audio information, and a discrete cosine transform is then performed on the logarithmic spectrum of the pre-processed audio information to obtain the MFCC of the pre-processed audio information. When the audio feature extraction model is called to perform PLP processing on the pre-processed audio information, a fast Fourier transform is first performed on the time-domain signal of the pre-processed audio information to obtain a frequency-domain signal of the pre-processed audio information, a pre-emphasis and intensity-loudness transform are then performed on the frequency-domain signal of the pre-processed audio information to obtain loudness feature information of the pre-processed audio information, and then an inverse Fourier transform and linear prediction are performed on the loudness feature information of the pre-processed audio information to obtain PLP feature parameters of the pre-processed audio information. By performing MFCC processing and PLP processing on the pre-processed speech information to obtain the corresponding processing results, and then calculating the average of the two processing results to obtain the average value of the two processing results as speech feature information, the accuracy of speech feature extraction for the pre-processed speech information can be further improved, which is advantageous for the speech recognition model to subsequently perform accurate speech recognition for the speech feature information, and can improve the accuracy when the speech recognition model performs speech recognition on the speech information input by the target subject.

[0045] Step 320: Execute at least one round of target operations until a candidate execution instruction that passes the feasibility check is obtained; for the target operation of the first round, invoke a language model to perform command recognition on the request information based on predetermined guide presentation information to obtain a candidate execution instruction for the target operation of the first round; perform a feasibility check on the candidate execution instruction for the target operation of the first round to determine whether the candidate execution instruction for the target operation of the first round passes the feasibility check; for the target operation of the i-th round, invoke the language model to perform command recognition on the request information based on the candidate execution instruction for the target operation of the previous (i-1) round and the guide presentation information to obtain a candidate execution instruction for the target operation of the i-th round; perform a feasibility check on the candidate execution instruction for the target operation of the i-th round to determine whether the candidate execution instruction for the target operation of the i-th round passes the feasibility check, where i is an integer greater than 1.

[0046] In this embodiment, the target operation refers to an operation of calling a language model to perform command recognition for a request to obtain a candidate execution command, and then performing a feasibility check on the candidate execution command. In actual application, the terminal can perform one or multiple rounds of the target operation until it obtains a candidate execution command that passes the feasibility check. That is, if the terminal only performs one round of the target operation and obtains a candidate execution command that passes the feasibility check, the terminal does not need to continue to perform the target operation. However, if the terminal performs one or multiple rounds of the target operation but still does not obtain a candidate execution command that passes the feasibility check, the terminal needs to continue to perform the target operation until it obtains a candidate execution command that passes the feasibility check.

[0047] In one embodiment, after obtaining the request information of the target object, a language model can be invoked to perform command recognition on the request information based on predetermined guide presentation information to obtain candidate execution commands, and in a subsequent step, the terminal can automatically execute corresponding operation tasks based on the candidate execution commands, thereby avoiding multiple repeated interactions with the target object and improving the efficiency of interaction between the target object and the terminal.

[0048] In one embodiment, the language model for command recognition for request information based on guide presentation information may be an LLM model, an NLU model, etc., and an appropriate model may be selected according to the actual application situation, and no limitations are imposed here.

[0049] In one embodiment, the language model for performing command recognition for the request information based on the guidance presentation information may be deployed to a cloud server communicatively connected to the terminal, or may be deployed to the terminal, but this is not limited thereto. For example, when the terminal has a relatively small storage space and a relatively low processing power, the language model may be deployed to a cloud server communicatively connected to the terminal, thereby reducing the requirements for storage space and processing power of the terminal and improving the versatility of the command execution method based on the language model in this embodiment. Also, for example, when the terminal has a relatively large storage space and a relatively high processing power, the language model may be deployed to the terminal, thereby allowing the terminal to directly call up the language model to perform command recognition for the request information, thereby saving the interaction time between the terminal and the cloud server and improving the efficiency of responding to the request information.

[0050] In one embodiment, when invoking a language model to perform command recognition on request information based on predetermined guide presentation information to obtain candidate execution commands, model invocation request information is generated based on the request information and the predetermined guide presentation information, and then the model invocation request information is sent to a language model invocation interface to invoke the language model to perform command recognition on the request information based on the guide presentation information to obtain candidate execution commands.

[0051] In one embodiment, when the language model is deployed on the cloud server, the terminal can generate model call request information in HTTP (Hyper Text Transfer Protocol) message format based on the request information and predetermined guide presentation information, wherein the request information and guide presentation information are filled in the message body part of the HTTP message. Then, the terminal sends the model call request information in HTTP message format to the language model through the call interface of the language model. After the language model obtains the model call request information through the call interface, the language model first parses the model call request information in HTTP message format to extract the request information and guide presentation information from the message body part of the HTTP message. Then, it can perform command recognition on the request information based on the guide presentation information to obtain candidate execution commands. After the language model recognizes the candidate execution commands, it can return the candidate execution commands to the terminal in the form of an HTTP response message.

[0052] In another embodiment, when the language model is deployed in the terminal, the terminal generates model call request information in a signal format for transmission between each module inside the terminal based on the request information and the predetermined guide presentation information, and then the model call request information is sent to the call interface of the language model deployed in the terminal by the artificial intelligence interaction module inside the terminal, so that the model call request information is transmitted to the language model through the call interface; after the language model receives the model call request information through the call interface, the language model can first analyze the model call request information to extract the request information and the guide presentation information from the model call request information, and then perform command recognition on the request information based on the guide presentation information to obtain the candidate execution command; after the language model recognizes the candidate execution command, the language model can send the candidate execution command back to the artificial intelligence interaction module of the terminal through the call interface.

[0053] In this way, by using the above-mentioned method, model call request information is generated based on request information and guide presentation information, and a language model is called using the model call request information to perform command recognition, thereby improving the standardity of the command recognition flow.

[0054] In one embodiment, the prompt is a text segment or prompt provided to activate (start) the language model. The prompt may be a single word, a phrase, a sentence, a paragraph, or a paragraph, but is not limited thereto. After the language model receives the prompt, it can generate a complete text based on the prompt, and the generated text can be as consistent as possible with the theme, tone, and style of the prompt. Therefore, by invoking the language model to perform command recognition on the request information based on the specified prompt, the language model can perform command recognition on the request information more accurately according to the content of the prompt, thereby increasing the reliability of the candidate execution commands obtained by recognition and obtaining candidate execution commands that more closely match the request information.

[0055] In one embodiment, the guide presentation information is composed of multiple command presentation information, and each command presentation information may include two parts: a question presentation information part and an answer presentation information part. Of these, for each command presentation information, the question presentation information part describes in detail the application scenario of the current command and its corresponding question explanation, which can be used to guide the language model to recognize the meaning of the question content in the application scenario of the current command. The answer presentation information part describes in detail the command message structure of the current command (including the command name, command parameters, etc.) and its corresponding command interpretation, which can be used to guide the language model to output the corresponding answer content in the application scenario of the current command.

[0056] 4, which exemplifies the structural content of the guide presentation information. In FIG. 4, the guide presentation information includes first command presentation information 410 and second command presentation information 420, of which the first command presentation information 410 includes a first question presentation information portion 411 and a first answer presentation information portion 412 that match each other, and the second command presentation information 420 includes a second question presentation information portion 421 and a second answer presentation information portion 422 that match each other. The first question presentation information portion 411 and the second question presentation information portion 421 both include a question event presentation description and corresponding question content. For example, in the first question presentation information portion 411, the question event presentation description is “event”:“question”, and the corresponding question content is “content”:“I need to open a weather forecast webpage to find out what the weather is like today. What should I do?” The first answer presentation information portion 412 and the second answer presentation information portion 422 both include an answer event presentation description and corresponding answer content. For example, in the first answer presentation information portion 412, the answer event presentation description is “event”:“answer”, and the corresponding answer content is “content”:“This can be achieved by using a weather forecast plugin, and the command message structure is {“event”:“ / weathernew”,“parameter”:“How is the weather today?”}”, where “event” refers to the name of the plugin and “parameter” refers to the script parameter.

[0057] By calling a language model and learning the question presentation information portion and answer presentation information portion of each guide presentation information, the accuracy with which the language model recognizes commands for the target subject's request information can be improved, so that the candidate execution commands obtained by recognition can be more accurately matched to the application scenario when the target subject issues the request information.

[0058] In one embodiment, the guide presentation information may include presentation information for multiple different types of commands, each of which has a corresponding application scenario and usage method. For example, see Table 1 below, which illustrates examples of applications and classifications corresponding to multiple commands.

[0059] [Table 1] As can be seen from the contents in Table 1, the guide presentation information provided in this embodiment includes presentation information for multiple different types of commands, such as search commands, message commands, and application commands, and further includes application scenarios and usage methods corresponding to these commands, for example, performing a search using a search engine, sending a message or voice message using instant messaging software, checking news using a news engine, etc. Therefore, when the language model performs command recognition on the request information based on the guide presentation information, it can perform more accurate command recognition on the target object's request information, thereby effectively improving the reliability of the candidate execution commands obtained by recognition and obtaining candidate execution commands that are more closely matched with the request information.

[0060] In one embodiment, the language model for command recognition for the request information based on the guidance presentation information is a trained neural network-based natural language processing model. Therefore, when the language model is invoked to perform command recognition for the request information based on the guidance presentation information, it can effectively recognize the execution command corresponding to the request information. The process of training the language model may include the following four steps: (1) preparing training data; (2) generating a dictionary; (3) building a model; and (4) training the model.

[0061] In the training data preparation step, a publicly available sample dataset can be downloaded in advance via the network, or a sample dataset can be generated by a natural language generation model, where these sample datasets may include multiple different types of natural language corpora, such as web page content, news content, and novel content. Then, these sample datasets and predetermined guide presentation information are constructed as training sample data, and the training sample data can be subjected to pre-processing such as cleaning, word segmentation, and stop word removal to obtain a training sample set for training the language model.

[0062] In the dictionary generation step, statistics and counting can be performed on the words that appear in all the text information of the training sample set to generate a dictionary, and then it is advantageous to generate a corresponding unique number (identifier) ​​for each word in the dictionary, so that the dictionary can be used to train a language model in a subsequent step.

[0063] In the model construction step, a language model can be constructed using a neural network, for example, a recurrent neural network can be employed to construct the language model. Recurrent neural networks have memory capabilities and can memorize previous inputs, thereby influencing subsequent outputs. When constructing the language model, parameters can be set for the language model, such as the number of hidden layers and the learning rate. In one embodiment, the number of hidden layers in the language model can be set to 12, i.e., the language model can include 12 hidden layers, and the learning rate of the language model can be set to 50%, but these are not limited thereto.

[0064] In the model training step, the training sample set after the previous step is input into the language model for training, and the parameters of the language model are constantly adjusted during the training process to minimize the error between the predicted result of the language model and the actual result (i.e., label), and the training of the language model can be considered complete when the error between the predicted result of the language model and the actual result is smaller than a predetermined error threshold or the number of iterations of the language model reaches a predetermined threshold. After the training of the language model is completed, the language model can be used to perform command recognition on the request information of the target object.

[0065] In one embodiment, the language model training process employs processing methods such as random dropout and batch normalization (BN) to improve the training effect of the language model. Random dropout can be used to drop out some nodes in the language model during the language model training process, thereby simplifying the structure of the language model during the training process and preventing certain nodes in the language model from becoming overweighted during training, thereby reducing overfitting of the language model. Furthermore, batch normalization can be used during the language model training process to adjust the intermediate outputs of the language model using the mean and variance of small batches during the training process, thereby ensuring that the outputs between each network layer in the language model conform to a Gaussian distribution with the same mean and variance, thereby making the data more stable. Therefore, regardless of how the parameters of the hidden layer in the language model change, the mean and variance of the output data of the network in the previous layer can be determined to be known and fixed, effectively solving problems such as slow training and a small learning rate caused by constantly changing data distribution.

[0066] In one embodiment, after calling the language model to perform command recognition on the request information based on the guidance presentation information to obtain candidate execution commands, the candidate execution commands are not necessarily valid executable commands. If the candidate execution commands are not valid executable commands, the terminal cannot be subsequently controlled to perform the operation task corresponding to the request information based on the candidate execution commands, which may result in problems such as a decrease in the efficiency of interaction between the target object and the terminal and a poor user experience.

[0067] In order to improve the interaction efficiency between the target object and the terminal and enhance the user experience, after obtaining a candidate execution command, a feasibility check is first performed on the candidate execution command to obtain a feasibility check result, and then, based on the feasibility check result, it is determined whether the candidate execution command passes the feasibility check. If the feasibility check result is "the candidate execution command does not pass the feasibility check," this means that the candidate execution command obtained by the language model recognition for the current round of target operation is incorrect and is not a valid executable execution command. At this time, the next round of target operation can be initiated. That is, the language model is called to perform command recognition on the request information based on the candidate execution command and guide presentation information for the previous target operation to obtain a candidate execution command for the current round of target operation. Then, a feasibility check is performed on the candidate execution command to obtain a feasibility check result, and then, based on the feasibility check result, it is determined whether the candidate execution command passes the feasibility check. This process is repeated until the obtained candidate execution command passes the feasibility check.

[0068] The candidate execution command that passes the feasibility check is obtained by calling the language model multiple times to perform command recognition on the request information based on the guide presentation information and the candidate execution command in the target operation of the previous round, so that the reliability of the candidate execution command can be further improved. Furthermore, the process of calling the language model to perform command recognition on the request information again is determined according to the specific situation of the feasibility check result. Therefore, the process of calling the language model to perform command recognition on the request information again is completed automatically, and does not require the target object to perform any additional operations, nor does it require the target object to perform multiple repeated interactions with the terminal. Therefore, the interaction efficiency between the target object and the terminal can be improved, and the purpose of simplifying the operation of the target object can be achieved, and the user experience of the target object can be improved.

[0069] In one embodiment, when the language model is called to perform command recognition again on the request information based on the candidate execution command and guide presentation information in the previous target operation, the model call request information is regenerated based on the candidate execution command obtained by recognition, the request information of the target object, and the predetermined guide presentation information, and then the regenerated model call request information is sent to the call interface of the language model to call the language model and perform command recognition again on the request information based on the candidate execution command and the guide presentation information.

[0070] Among these, when the language model is called and command recognition is performed again for the request information multiple times, the model call request information is regenerated based on the multiple candidate execution commands obtained by the previous recognition, the request information, and the guide presentation information, and then the regenerated model call request information is sent to the call interface of the language model to call the language model and perform command recognition again for the request information based on the multiple candidate execution commands obtained by the previous recognition and the guide presentation information, so that the multiple candidate execution commands obtained by the previous recognition can improve the reliability of the candidate execution commands obtained by the new recognition. For example, assuming that for request information A, the language model is called and command recognition is performed three times for the request information A to obtain three previous candidate execution commands, when the language model is called and command recognition is performed for the request information A for the fourth time, the model call request information is regenerated based on the three candidate execution commands obtained by the previous three recognitions, the request information A, and predetermined guide presentation information, and then the regenerated model call request information is sent to the call interface of the language model to call the language model and perform command recognition again for the request information A based on the three candidate execution commands obtained by the previous three recognitions and the guide presentation information.

[0071] In one embodiment, when performing a feasibility check on a candidate executable instruction to obtain a feasibility check result, the instruction function in the candidate executable instruction can first be replaced with a target null function, and the function result of the target null function can always be true, thereby obtaining an executable instruction waiting to be checked; then, an instruction execution engine is called to simulate the execution of the executable instruction waiting to be checked to obtain a simulated execution result; and then, a feasibility check result is obtained based on the simulated execution result, i.e., it can be determined whether the candidate executable instruction has passed the feasibility check.

[0072] Among these, when performing a feasibility check on a candidate executable instruction, the operations corresponding to the instruction function in the candidate executable instruction may be relatively complicated. Therefore, if the instruction execution engine is directly called to simulate the execution of the candidate executable instruction, the execution time may be long, which may affect the efficiency of performing the feasibility check on the candidate executable instruction. In order to save the execution time of performing the feasibility check on the candidate executable instruction and improve the efficiency of performing the feasibility check on the candidate executable instruction, the instruction function in the candidate executable instruction can be replaced with a null function whose function result is always true to obtain a check-awaiting executable instruction. In this way, when the instruction execution engine is called to simulate the execution of the check-awaiting executable instruction, the feasibility of the check-awaiting executable instruction can be quickly checked, thereby improving the efficiency of performing the feasibility check on the candidate executable instruction.

[0073] Referring to Figure 5, Figure 5 exemplarily provides a process of replacing the instruction function function in the candidate execution instruction with a null function whose function result is always true. In Figure 5, if the candidate execution instruction 510 includes an instruction function function 511, first analyze the candidate execution instruction 510 to determine the instruction function function 511 in the candidate execution instruction 510, then obtain one null function 521 whose function result is always true from a predetermined function library 520, and then replace the instruction function function 511 in the candidate execution instruction 510 with the null function 521, thereby obtaining a check-waiting execution instruction 530 containing the null function 521 whose function result is always true.

[0074] In one embodiment, before performing a feasibility check on a candidate execution command, a format check may be performed on the candidate execution command first. If the format of the candidate execution command matches the command message structure in the guide presentation information, the candidate execution command has passed the format check, which means that the command execution engine can simulate the execution of the candidate execution command, and therefore a feasibility check can be performed. If the format of the candidate execution command does not match the command message structure in the guide presentation information, it means that the candidate execution command does not pass the format check, which means that the command execution engine cannot recognize the candidate execution command, and therefore the execution of the candidate execution command cannot be simulated and a feasibility check cannot be performed. In other words, for candidate execution commands that do not pass the format check, there is no need to perform a feasibility check on them, and by directly determining that they cannot pass the feasibility check, the efficiency of performing feasibility checks on candidate execution commands can be improved. By first performing a format check on the candidate executable instructions, candidate executable instructions that do not pass the format check can be directly determined as candidate executable instructions that do not pass the executable check, thereby saving the execution time required to perform an executable check on candidate executable instructions that do not pass the format check and improving the efficiency of performing an executable check on candidate executable instructions.

[0075] In one embodiment, when obtaining a feasibility check result based on a simulated execution result, if the simulated execution result is "the executable instruction waiting to be checked is executed successfully," it means that a null function whose function result is always true in the executable instruction waiting to be checked can be accurately recognized and executed. In this case, by replacing the null function in the executable instruction waiting to be checked with an instruction function, the executable instruction waiting to be checked can be changed to the original candidate executable instruction, and the instruction function in the candidate executable instruction can also be accurately recognized and executed, so a feasibility check result indicating that the candidate executable instruction passed the feasibility check can be obtained. If the simulated execution result is "the executable instruction waiting to be checked cannot be executed successfully," it means that a null function whose function result is always true in the executable instruction waiting to be checked cannot be accurately recognized and executed. In this case, by replacing the null function in the executable instruction waiting to be checked with an instruction function, the executable instruction waiting to be checked can be changed to the original candidate executable instruction, and the instruction function in the candidate executable instruction can also be accurately recognized and executed, so a feasibility check result indicating that the candidate executable instruction did not pass the feasibility check can be obtained.

[0076] In this way, by using the above-mentioned method to determine whether a candidate executable instruction passes the feasibility check based on the simulated execution result obtained by executing the executable instruction waiting to be checked, the accuracy and reliability of the determined feasibility check result can be ensured, and the feasibility check result can be guaranteed to be determined efficiently.

[0077] In one embodiment, if the result of the first feasibility check on the candidate execution command is "the candidate execution command passes the feasibility check," it means that the candidate execution command is an execution command that matches the request information of the target object, and there is no need to call the language model to perform command recognition on the request information again. Therefore, the terminal can be directly controlled to execute the operation task corresponding to the request information based on the candidate execution command.

[0078] Step 330: The candidate execution command that passes the feasibility check is set as the target execution command, and the terminal is controlled to execute the operation task corresponding to the request information according to the target execution command.

[0079] In one embodiment, when a candidate execution command that passes the feasibility check is obtained, the candidate execution command that passes the feasibility check can first be used as a target execution command, and then, based on the target execution command, the terminal can be controlled to execute the operation task corresponding to the request information to meet the usage requirements of the target object. The entire process from obtaining the candidate execution command that passes the feasibility check to controlling the terminal to execute the operation task corresponding to the request information based on the target execution command is automatically performed by the terminal, without requiring the target object to perform any additional operation or to perform multiple repeated interactions, thereby effectively improving the interaction efficiency between the target object and the terminal and achieving the purpose of simplifying the operation of the target object, and effectively improving the usage experience of the target object.

[0080] In one embodiment, when a terminal is controlled to perform an operation task corresponding to request information according to a target execution command, the corresponding command execution code is first obtained according to the target execution command, and then the command execution code is executed in a virtual environment, thereby controlling the terminal to perform the operation task corresponding to the request information. Wherein, each target execution command has a corresponding command execution code, and the command execution code describes the process of each step of executing the target execution command. Therefore, the command execution code corresponding to the target execution command is first obtained, and then the command execution code is executed in a virtual environment, thereby achieving the purpose of simulating the target object controlling the terminal to perform the operation task corresponding to request information.

[0081] In this way, by using the above method, the command execution code corresponding to the target execution command is obtained, and the command execution code is executed to control the terminal to execute the operation task corresponding to the request information, thereby ensuring the accurate and reliable execution of the operation task.

[0082] In one embodiment, to achieve the concurrency of different execution commands, multiple target execution commands are obtained, and the command execution code corresponding to each target execution command is called. After that, multiple virtual environments (for example, virtual mirrors of operating systems with desktops) are launched simultaneously, and the command execution code of different target execution commands can be executed simultaneously in different virtual environments. That is, the parallel execution of different operating tasks can be achieved in the manner of the simultaneous execution of multiple operating system mirrors.

[0083] In one embodiment, when executing an instruction execution code in a virtual environment to control a terminal to perform an operation task corresponding to the request information, an instruction execution process can be executed in the virtual environment first, and then the instruction execution process can be called to execute the instruction execution code to control the terminal to perform an operation task corresponding to the request information.

[0084] Wherein, when executing an instruction execution code in a virtual environment, an operating system running in each virtual environment can launch a corresponding instruction execution process, which can be used to automatically perform control operations on the terminal, for example, if the terminal is a computer with a mouse and keyboard, the instruction execution process can be used to automatically perform control operations on the computer through the mouse and keyboard. Wherein, controlling the terminal to perform an operation task corresponding to the request information can be controlling the terminal to perform various operation tasks, which can include multiple different types of operation tasks, such as opening an application, inputting text, single-clicking a button, dragging a window, etc.

[0085] In one embodiment, the command execution code called based on the target execution command can be used to realize different operation tasks, such as opening an application, inputting text, single-clicking a button, dragging a window, etc. See Table 2, which exemplarily shows various command execution codes that can realize different functions.

[0086] [Table 2] As can be seen from the contents in Table 2, the command execution code in this embodiment includes command execution code that can achieve various different functions, such as command execution code that can obtain the coordinates of the operation cursor, command execution code that can move the operation cursor, command execution code that can simulate a click operation, command execution code that can input characters, etc. Therefore, when the corresponding command execution code is obtained based on the target execution command, it can effectively simulate the control of the target object on the terminal and perform the operation task corresponding to the request information. In one embodiment, the command execution code in this embodiment can obtain the data and operation events corresponding to the simulation by calling the application programming interface of the operating system.

[0087] In one embodiment, when the terminal includes a display module for displaying an operation cursor, when controlling the terminal to perform an operation task corresponding to the request information, the initial coordinates of the operation cursor are first obtained, and then a target coordinate for performing the operation task corresponding to the request information is determined. Subsequently, the operation cursor is moved from the initial coordinate to the target coordinate, and the terminal is controlled to perform the operation task corresponding to the request information at the target coordinate.

[0088] The initial coordinates of the operating cursor can be obtained by first calling the application programming interface of the operating system to obtain the screen resolution of the display module, and then obtaining the initial coordinates of the operating cursor based on the screen resolution. For example, if the screen resolution of the display module is 1920*1080 and the operating cursor is located at the center of the display module screen, the initial coordinates of the operating cursor on the display module screen can be determined to be (960,540). When performing an operating task corresponding to the request information, the terminal automatically simulates the operation method of the target object. This process does not require the target object's participation or any additional operations. This effectively simplifies the operation of the target object on the terminal, thereby improving the user experience of the target object. Furthermore, controlling the movement of the operating cursor visualizes the process of the terminal performing the operating task corresponding to the request information, allowing the target object to understand the execution process of the operating task. The display module can be integrated into the display screen of the terminal or an external display device of the terminal, without any limitation herein. For example, when the terminal is a smartphone, the display module may be the display screen of the smartphone, and when the terminal is a personal computer, the display module may be the external display of the personal computer.

[0089] An example will be given below to specifically describe the process of controlling the terminal to execute the operation task corresponding to the request information.

[0090] For example, if the request information of the target object is to check today's weather conditions, in this case, the operation task corresponding to the request information may be to check today's weather conditions using a browser.In this case, the command execution process executed in the virtual environment executes the command execution code getSize, which can obtain the screen width and screen height of the terminal. At this time, the command execution code getSize can return one tuple data, which can be, for example, (1920, 1080), representing the screen resolution of the terminal. Then, the command execution process executes the command execution code getLocation, which can obtain the current coordinates (a, b) of the operation cursor on the terminal screen and the target coordinates (x, y) of the browser application icon on the terminal screen. Next, the command execution process executes the command execution code moveTo, which causes the operation cursor to move from the current coordinates (a, b) to the target coordinates (x, y) at a certain speed. Then, the command execution code click is executed to simulate the click operation of the target object on the browser application icon (for example, the left mouse button click performed by the target object on the browser application icon). The command execution process executes the command execution code getSize to obtain the coordinates (h, i) of the search input box on the browser page. Then, the command execution process executes the command execution code moveTo again to move the operation cursor from the target coordinates (x, y) to the coordinates (h, i) of the search input box at a constant speed. Then, the command execution code click is executed again to simulate the click operation of the target object on the search input box (for example, to simulate the target object performing a single click of the left mouse button on the search input box), so that the operation cursor is positioned in the search input box. Next, the command execution process executes the command execution code typeWrite to simulate the input operation of the target object on the search input box, so that the text input information "How's the weather today?" is entered into the search input box, thereby allowing the browser to obtain today's weather conditions.When performing an operation task corresponding to the request information, the terminal automatically realizes the operation method of the target object by simulating the operation method of the target object, so the entire process does not require the participation of the target object, nor does it require the target object to perform any complicated additional operations, which effectively simplifies the operation of the target object on the terminal and improves the use experience of the target object.

[0091] In this embodiment, by the language model-based command execution method including the above-mentioned steps 310 to 330, when the request information of the target object is received, at least one round of target operations is executed until a candidate execution command that passes the feasibility check is obtained. In the first round of target operations, the language model is first called to perform command recognition on the request information based on the predetermined guide presentation information to obtain a candidate execution command for the target operation of the first round. The candidate execution command is obtained by the language model performing command recognition on the request information based on the guide presentation information, so that the reliability of the candidate execution command can be improved, which is advantageous to obtain a candidate execution command that matches the request information more closely. Next, the executable candidate execution command for the candidate execution command of the target operation of the first round is obtained. A feasibility check is performed to determine whether the candidate execution command passes the feasibility check. If the candidate execution command does not pass the feasibility check, it indicates that the candidate execution command obtained by the language model recognition in the current round (this round) is not accurate. At this time, the target operation in the next round is continued. In the target operation in the next round, the language model is called to perform command recognition again on the request information based on the candidate execution command in the previous target operation and the guide presentation information to obtain a candidate execution command in the target operation in this round. Then, a feasibility check is performed again on the candidate execution command to determine whether the candidate execution command passes the feasibility check. In this way, several rounds of target operations are repeatedly performed until a candidate execution command that passes the feasibility check is obtained.Furthermore, after obtaining a candidate execution command that passes the feasibility check, the candidate execution command is taken as a target execution command, and the terminal is controlled to execute the operation task corresponding to the request information based on the target execution command. The candidate execution command that passes the feasibility check is obtained by calling the language model multiple times to perform command recognition on the request information based on the guide presentation information and the previous candidate execution command, so the reliability of the candidate execution command can be guaranteed. In addition, this process is completed automatically, and does not require the target object to perform any additional operations, nor does it require the target object to perform multiple repeated interactions. This can improve the interaction efficiency between the target object and the terminal and achieve the purpose of simplifying the operation of the target object, thereby improving the user experience of the target object.

[0092] The specific process of the instruction execution method based on the language model will be described below using a specific example.

[0093] Referring to Figure 6, Figure 6 is a diagram showing an interface through which a user interacts with a terminal through an AI interaction module. After a user inputs voice information 610 into a terminal through the interaction interface of the AI ​​interaction module, the terminal can first perform recognition on the voice information 610. To avoid displaying blank content while the user is waiting, the terminal can display first loading waiting instruction information 620 on the interaction interface to inform the user that recognition processing is being performed on the voice information 610. In the process of recognizing the voice information 610, the terminal can first invoke a pre-trained voice recognition model to perform voice recognition on the voice information 610 to obtain text information, and then perform at least one of adding punctuation, correcting spelling, and correcting word order on the text information to obtain user request information 630. At this time, the terminal completes the recognition processing on the voice information 610. Among them, the pre-trained speech recognition model is used to convert the voice frequency feature vector of the voice information into text information, and the speech recognition model may include models commonly used in the field, such as hidden Markov models, Gaussian mixture models, deep neural network models, n-gram language models, and other statistical models.

[0094] After the terminal completes the recognition of the voice information 610 and obtains the user's request information 630, it can display the request information 630 obtained by recognition in the interaction interface. For example, suppose the voice information 610 uttered by the user is "Can you check what the weather is like today?", and after the terminal accurately recognizes the voice information 610, it can display the request information 630, the content of which is "Can you check what the weather is like today?", in the interaction interface. After the terminal completes the recognition of the voice information 610, it can call the language model to perform command recognition on the content of the request information 630 based on the guide presentation information. At this time, in order to avoid displaying blank content while the user is waiting, the terminal may display second load waiting instruction information 640 in the interaction interface to notify the user that command recognition is being performed on the content of the request information 630.

[0095] In the process of the terminal calling the language model to perform command recognition on the content of the request information 630 based on the guide presentation information, it can first generate model call request information based on the guide presentation information and the request information 630, and then send the model call request information to the call interface of the language model to call the language model and perform command recognition on the request information 630 based on the guide presentation information.

[0096] The content structure of the guide presentation information can be seen in Figure 7, which illustrates an example of the guide presentation information. In Figure 7, the guide presentation information may include a type item 710, a question item 720, and an answer item 730, in which the type item 710 records the information type of the guide presentation information, for example, as shown in Figure 7, the information type of the guide presentation information is "content", the question item 720 records the question content of the guide presentation information, for example, as shown in Figure 7, the question content of the guide presentation information is "I need to open a browser and search for today's weather, what should I do?", and the answer item 730 records the answer content of the guide presentation information, for example, as shown in Figure 7, the answer content of the guide presentation information is It includes a structure description 732 and an interpretation description 733, of which the answer description 731 is "this can be achieved by using a browser plug-in", the command message structure description 732 is "command message structure: {"event":" / weathernew", "parameter":"what's the weather like today?"}", and the interpretation description 733 is "event is the plug-in name:weathernew, which represents opening a browser and entering the website www.xxxxxx.com, and parameter is the script parameter:"what's the weather like today?", which is the content to be searched in the browser". When generating guide presentation information, the information type, question content, and answer content corresponding to the question content of the guide presentation information are first obtained, and then the information type of the guide presentation information is filled into the type item 710, the question content of the guide presentation information is filled into the question item 720, and the answer content corresponding to the question content is filled into the answer item 730, and at this time the guide presentation information can be obtained.

[0097] After the terminal calls the language model and performs command recognition on the request information 630 based on the guide presentation information to obtain a target execution command, the terminal can further control the terminal to execute a corresponding operation task based on the target execution command, and in the process of controlling the terminal to execute the corresponding operation task, third load waiting instruction information 650 can be displayed on the interaction interface to indicate to the user that the operation task desired by the user is being executed, and after the terminal completes the execution of the corresponding operation task, it can display execution completion instruction information 660 on the interaction interface to inform the user that the operation task corresponding to the voice information 610 has been completed.

[0098] In the process where the terminal calls the language model to perform command recognition on the request information 630 based on the guide presentation information, it can currently perform a feasibility check on the candidate execution command obtained by recognition. If the candidate execution command currently obtained by recognition fails to pass the feasibility check, the terminal can call the language model to perform command recognition on the request information 630 again based on the candidate execution command and the guide presentation information to obtain a new candidate execution command, and then perform a feasibility check on the new candidate execution command again. In this way, the above process is repeated until a candidate execution command that passes the feasibility check is obtained. At this time, the terminal can take the candidate execution command that passes the feasibility check as the target execution command.

[0099] Wherein, when the terminal currently performs a feasibility check on the candidate execution command obtained by recognition, it can first replace the command function function in the candidate execution command with an empty function whose function result is always true to obtain an execution command to be checked, then call the command execution engine to simulate the execution of the execution command to be checked to obtain the simulated execution result, and then obtain the feasibility check result based on the simulated execution result. In addition, in the process of the terminal controlling the terminal to execute a corresponding operation task based on the target execution command, it can first obtain the corresponding command execution code based on the target execution command, then execute the command execution process in the virtual environment, and then call the command execution process to execute the command execution code to control the terminal to execute the operation task corresponding to the request information. In addition, after the terminal completes the execution of the corresponding operation task, it can display an interaction interface as shown in FIG. 8. In the interaction interface shown in FIG. 8, a pop-up window 810 can display the operation result after the execution of the corresponding operation task is completed, and the user can obtain the desired request content from the operation result displayed in the pop-up window 810. In another embodiment, the operation result after the execution of the corresponding operation task is completed can also be displayed by page jumping, and is not limited thereto.

[0100] In one embodiment, when the terminal controls the terminal to execute the operation task corresponding to the request information according to the target execution command, the terminal can further generate an execution record table including an execution command name table entry, an instruction execution code table entry, an execution status table entry and an execution result table entry, in which the execution command name table entry records the target execution command, and the instruction execution code table entry records the instruction execution code corresponding to the target execution command, and after the terminal controls the terminal to complete the execution of the operation task corresponding to the request information, the terminal can update the content of the execution status table entry in the execution record table to completed and write the corresponding execution result into the execution result table entry. By generating an execution record table to record the execution status of the target execution command, it can realize real-time management and control of the execution status of the target execution command, which is advantageous for the terminal to automatically execute the operation task corresponding to the request information.

[0101] In one embodiment, the command message structure of the target execution command obtained by recognition is {“event”:“ / weathernew”,“parameter”:“How's the weather today?”} Let us assume that:

[0102] Wherein, weathernew is the plugin_name, ie, the name of the plugin, and different plugins may have different instruction execution codes in the table entry steps.

[0103] In this case, when the terminal controls the terminal to execute the operation task corresponding to the request information based on the target execution command, the terminal can generate an execution record table in the command execution database, and the execution record table includes the contents of multiple table entries such as execution command name table entries, command execution code table entries, execution state table entries, and execution result table entries, and the execution record table can record information such as the flow relationship between multiple target execution commands and execution time through these table entries. Please refer to Table 3, which exemplifies the specific structure of the execution record table.

[0104] [Table 3] In Table 3, the content of the first row is the table entry name, the content of the second row is the table entry content, among which the table entry of plugin_name is the execution command name table entry, the table entry of steps is the command execution code table entry, the table entry of status is the execution status table entry, and the table entry of result is the execution result table entry. In the execution record table, the command execution code in the table entry of steps can be retrieved and loaded from the plugin execution code database based on the plugin name (i.e., the execution command name) recorded in the table entry of plugin_name. In this way, when controlling the terminal to perform the operation task corresponding to the request information based on the target execution command, the command execution engine can control the terminal to perform the operation task corresponding to the request information by executing step by step based on the specific content of the command execution code in the table entry of steps.

[0105] In one embodiment, all target execution commands obtained by recognition based on the language model may have a unified standard command message structure. In other words, when the language model outputs candidate execution commands obtained by recognition, it can output the candidate execution commands in the form of a unified standard command message structure. In this way, by standardizing the command message structure, a unified standard can be realized for all operation tasks, so that the execution environment for executing the target execution commands can be unified. This makes it possible to support parallel processing of multiple operation tasks in a multi-operating system mirror mode (i.e., multiple virtual environments), and effectively reduces the cost of executing multiple operation tasks.

[0106] In one embodiment, during the execution process of the system, the terminal further generates a corresponding log file and records the corresponding log information in detail in the log file, so as to manage and control the execution status of the terminal based on the log information in the log file, and realize daily troubleshooting and status recording during the execution of the terminal.

[0107] In one embodiment, the log information in the log file may have different log types, for example, the log information may have an error type, an alarm type, an information type, and a test type, and different types of log information may have different uses. The log information in the log file may also have different log contents, and according to the different log contents, the log information in the log file may be classified into a configuration log, an alarm log, a background log, etc. See Tables 4 and 5, where Table 4 provides an example of a description of different types of log information, and Table 5 provides an example of a description of different contents of log information.

[0108] [Table 4] As can be seen from the contents of Table 4, when error-type log information is recorded in the log file, administrators should pay special attention to the log information and resolve the faults in the log information in a timely manner to ensure the subsequent normal operation of the system. When alarm-type log information is recorded in the log file, administrators should pay close attention to the log information and resolve the faults in the log information as soon as possible to avoid abnormal risks in the system execution process. When information-type log information is recorded in the log file, administrators should pay attention to this log information in daily operation and maintenance work so that they can correct any abnormalities in important system information when they occur and maintain the normal operation of the system. When test-type log information is recorded in the log file, administrators can use this log information as reference information when testing the system, which can assist the test and improve the efficiency of administrator testing the system.

[0109] [Table 5] As can be seen from the contents in Table 5, when the setting log is recorded in the log file, the administrator can use the setting log to understand various different system execution statuses, such as user addition and deletion statuses and setting modification statuses. When the management and control log is recorded in the log file, the administrator can use the management and control log to understand the management and control actions regarding system execution. When the alarm log is recorded in the log file, the administrator can use the alarm log to understand alarm information that appears in the system execution process, so that the alarm information can be corrected in a timely manner to maintain normal system execution. When the background log is recorded in the log file, the administrator can use the background log to understand various system actions throughout the system's background execution process.

[0110] Hereinafter, the language model-based instruction execution method provided in the embodiment of the present application will be described in detail through a specific example.

[0111] 9, which is a flowchart illustrating an overall language model-based instruction execution method provided in one example, which may include, but is not limited to, steps 910 to 950.

[0112] Step 910: The terminal receives the target object's voice signal and converts the voice signal into request information that can be recognized by a language model.

[0113] This step is a step in which the terminal receives request information, and in this step, the terminal can receive the target voice signal through various voice receiving software or hardware.

[0114] In the step, when the terminal converts the voice signal into request information that can be recognized by the language model, it can sequentially perform operations such as voice signal pre-processing, voice feature extraction, voice recognition, and text post-processing on the target voice signal to obtain the request information that can be recognized by the language model, so that in the subsequent step, it can call the language model to perform command recognition on the request information.

[0115] In one embodiment, the process of the terminal receiving the target object's voice signal and converting the voice signal into request information that can be recognized by a language model may include the following steps 911 to 917.

[0116] Step 911: Acquire an audio signal.

[0117] In this step, collecting the audio signal refers to receiving audio information input by the target object. The terminal can collect the audio signal of the target object through an audio receiving module (e.g., a microphone installed inside the terminal) or through another external audio receiving module (e.g., an external device such as an external recording device, another smartphone, another tablet computer, or industrial equipment), and the present invention is not limited thereto.

[0118] Step 912: Pre-process the audio signal.

[0119] In this step, pre-processing the speech signal refers to performing at least one of noise reduction and speech segmentation on the speech information to obtain pre-processed speech information. After collecting the speech signal of the target object, the terminal can perform speech signal pre-processing on the speech signal of the target object. In one embodiment, the speech signal pre-processing performed by the terminal on the speech signal may include operations such as noise reduction and speech segmentation. Performing noise reduction on the speech information means filtering out interference information such as environmental noise mixed in the speech information, thereby improving the purity of the speech information. Performing speech segmentation on the speech information means segmenting a long speech input by the target object into multiple short speeches, thereby reducing the difficulty of the subsequent speech feature extraction process by the speech feature extraction model and improving the accuracy and efficiency of speech feature extraction.

[0120] Step 913: Extract audio features.

[0121] In this step, extracting voice features refers to calling a voice feature extraction model to extract voice features from pre-processed voice information to obtain voice feature information. After the terminal performs voice signal pre-processing on the voice signal to obtain pre-processed voice information, it can convert the voice signal into voice feature information in the form of a digital signal (e.g., voice frequency feature vector), which is advantageous for the subsequent voice recognition model to perform more accurate recognition processing on the voice feature information.

[0122] In one embodiment, audio feature extraction can be performed on the audio signal by calling an audio feature extraction model, which may be a trained deep neural network model, a convolutional neural network model, or a mathematical model capable of extracting audio features (e.g., a mathematical model that performs MFCC processing on the audio signal, a mathematical model that performs DWT processing on the audio signal, a mathematical model that performs PLP processing on the audio signal, etc.), and may be appropriately selected according to the actual application, and is not limited thereto herein.

[0123] Step 914: Build a speech recognition model.

[0124] In this step, constructing a speech recognition model refers to constructing a speech recognition model for performing speech recognition on speech information and training the speech recognition model using speech feature information. After the terminal performs speech feature extraction on the speech signal to obtain speech feature information in the form of a digital signal, it can construct a corresponding speech recognition model based on the obtained speech feature information, and thereby perform speech recognition on the speech feature information using the speech recognition model.

[0125] In one embodiment, the speech recognition model may be a model commonly used in the field, such as an HMM model, a Gaussian mixture model, a deep neural network model, an n-gram language model, or other statistical model, and is not limited thereto herein.

[0126] Step 915: The speech recognition model is called to perform speech recognition.

[0127] In this step, calling a speech recognition model to perform speech recognition refers to calling a trained speech recognition model to perform speech recognition on speech feature information to obtain text information. After the terminal builds the speech recognition model, the terminal can call the speech recognition model to perform speech recognition on the speech feature information to obtain text information corresponding to the target speech signal.

[0128] In one embodiment, when the speech recognition model includes an acoustic model and a language model, in the process of invoking the speech recognition model to perform speech recognition on speech feature information, the acoustic model is first invoked to convert the speech feature information into intermediate recognition results (e.g., phonemes, phoneme strings, subwords, etc.), and then the language model is invoked to convert the intermediate recognition results into text recognition results (e.g., words, word strings, code sequences, etc.), and then text information corresponding to the target speech signal is output to complete speech recognition on the speech feature information.

[0129] Step 916: Post-process the text.

[0130] In this step, performing text post-processing refers to performing at least one of adding punctuation, correcting spelling, and correcting word order on the text information obtained by speech recognition to obtain request information of the target object. After the terminal invokes the speech recognition model to perform speech recognition on the speech feature information to obtain corresponding text information, the terminal performs at least one of adding punctuation, correcting spelling, and correcting word order on the text information to obtain request information corresponding to the speech signal of the target object.

[0131] Step 917: Output the request information of the target object.

[0132] In this step, after the terminal completes text post-processing of the text information output by the speech recognition model, it can output the text content that has undergone text post-processing, and at this time, the text content is the request information of the target object.

[0133] Through the processing of steps 911 to 917, the target speech signal can be converted into request information that can be recognized by a language model, so that in subsequent steps, the language model can be called to perform command recognition on the request information to obtain the execution command corresponding to the target speech signal.

[0134] Step 920: Generate targeted guide presentation information, where the guide presentation information includes an important request description, an instruction message structure and an interpretation description.

[0135] In this step, the guide presentation information is a text segment or presentation information provided to activate the language model. The guide presentation information may be a single word, a single phrase, a single sentence, a single paragraph, or a single paragraph, but is not limited thereto. When the guide presentation information is input into the language model for language recognition processing, the language model can generate a complete text based on the guide presentation information, and the language model can ensure that the generated text matches the theme, tone, and style of the guide presentation information as much as possible. Therefore, by generating guide presentation information including an important request description, a command message structure, and an interpretation description, when the language model is invoked in a subsequent step to perform command recognition on the request information based on the guide presentation information, the language model can perform more accurate command recognition on the request information according to the content of the guide presentation information, thereby improving the reliability of the command message structure obtained by recognition, i.e., a command message structure that more closely matches the request information can be obtained.

[0136] In one embodiment, the guide presentation information is composed of multiple command presentation information, and each command presentation information may include a question presentation information portion and an answer presentation information portion, wherein, for each command presentation information, the question presentation information portion details the application scenario of the current command and the corresponding question description, which are used to guide the language model to perform semantic recognition on the question content in the application scenario of the current command, and the answer presentation information portion details the command message structure of the current command (including the command name, command parameters, etc.) and the corresponding command interpretation, which are used to guide the language model to output the corresponding answer content under the application scenario of the current command.

[0137] Step 930: Invoke the pre-trained language model to perform command recognition on the request information based on the guide presentation information to obtain an executable command message structure.

[0138] In this step, after obtaining the request information and generating the guide presentation information, a pre-trained language model can be invoked to perform command recognition on the request information based on the guide presentation information to obtain an executable command message structure, so that in a subsequent step, the terminal can automatically execute the corresponding operation task based on the executable command message structure, thereby avoiding the target object from performing repeated interactions multiple times and improving the interaction efficiency between the target object and the terminal.

[0139] In one embodiment, the language model may be an LLM model, an NLU model, etc., and an appropriate selection may be made according to the actual application situation, and there is no limitation thereto here.

[0140] In one embodiment, the process of invoking a pre-trained language model to perform command recognition on request information based on guide presentation information may include the following steps 931 to 935.

[0141] Step 931: Prepare training data.

[0142] In this step, preparing training data refers to obtaining a training sample set for training a language model. The terminal can download publicly available sample data sets via a network or generate sample data sets using a natural language generation model, where these sample data sets may include different types of natural language corpora, such as web page content, news content, and novel content. Then, these sample data sets and the guide presentation information are organized as training sample data, and the training sample data can be pre-processed, such as cleaning, word segmentation, and stop word removal, to obtain a training sample set for training a language model.

[0143] Step 932: Build a dictionary.

[0144] In this step, building a dictionary refers to building a dictionary for training a language model based on the text information in the training sample set. The terminal performs statistics and counting on the vocabulary appearing in all the text information in the training sample set to build a dictionary, and then generates a corresponding unique number for each word in the dictionary, so that the dictionary can be used to train a language model in subsequent steps.

[0145] Step 933: Build a language model.

[0146] In this step, constructing a language model refers to constructing a language model for command recognition of the target object's request information. The terminal can construct the language model using a neural network, for example, a recurrent neural network. Recurrent neural networks have memory capabilities, allowing them to memorize previous inputs and affect subsequent outputs. When constructing the language model, parameters can be configured for the language model, such as the number of hidden layers and the learning rate. In one embodiment, the number of hidden layers in the language model can be set to 12, i.e., the language model can include 12 hidden layers, and the learning rate of the language model can be set to 50%, but this is not a limitation.

[0147] Step 934: Train the language model.

[0148] In this step, training a language model refers to training a language model using a training sample set. The terminal can input the training sample set after the previous step into the language model to perform training. In the training process, the parameters of the language model are constantly adjusted to minimize the error between the predicted result of the language model and the actual result (i.e., label). When the error between the predicted result of the language model and the actual result is smaller than a predetermined error threshold, or when the number of iterative training of the language model reaches a predetermined threshold, the training of the language model can be considered to be completed.

[0149] Step 935: Call the language model to perform command recognition on the request information based on the guide presentation information.

[0150] In this step, after the training of the language model is completed, the language model can be used to perform command recognition on the request information of the target object based on the guidance presentation information.

[0151] Through the processing of steps 931 to 935, a language model is constructed and training is completed for the language model, so that the language model can perform command recognition on the request information of the target object based on the guidance presentation information to obtain an executable command message structure, so that in subsequent steps, the terminal can automatically perform corresponding operation tasks based on the executable command message structure.

[0152] Step 940: Reading the command message structure and executing the command execution code corresponding to the command message structure in the virtual environment, thereby simulating the target object controlling the terminal to perform the operation task corresponding to the request information.

[0153] In this step, a language model is called and command recognition is performed on the request information based on the guide presentation information to obtain an executable command message structure, and then the contents of the command message structure are read and a command execution code corresponding to the command message structure is obtained. Then, the command execution code is executed in a virtual environment, thereby simulating the target object controlling the terminal to perform an operation task corresponding to the request information.

[0154] In one embodiment, the process of simulating the target object controlling the terminal to perform the operation task corresponding to the request information by executing the command execution code corresponding to the command message structure in the virtual environment may include the following steps 941 to 945.

[0155] Step 941: Load the function plug-in.

[0156] In this step, loading a function plug-in refers to loading the command execution code corresponding to the command message structure into the generated execution record table. Before executing the command execution code corresponding to the command message structure, the terminal can first generate an execution record table in the command execution database, which can be used to record information such as the command execution flow relationship and execution time. Then, based on the command name (or plug-in name) in the command message structure, the terminal can obtain the corresponding command execution code (i.e., function plug-in) from the command execution code database and load the command execution code into the execution record table, so that in a subsequent step, the terminal can read the command execution code from the execution record table and execute the command execution code.

[0157] Step 942: The virtual environment is started.

[0158] In this step, launching a virtual environment refers to launching a virtual environment for executing command execution code on a terminal. A terminal can simultaneously launch multiple virtual environments (such as virtual mirrors of operating systems with desktops) to achieve the concurrency of different execution commands, thereby achieving the simultaneous execution of different command execution codes in different virtual environments, thereby achieving the simultaneous execution mode of multiple operating system mirrors, and realizing the purpose of parallel execution of different operating tasks.

[0159] Step 943: Execute the instruction execution code.

[0160] In this step, executing the command execution code refers to executing a command execution process in the activated virtual environment and calling the command execution process to execute the command execution code. When the terminal executes the command execution code in the virtual environment to simulate the target object controlling the terminal to perform an operation task corresponding to the request information, the terminal can first execute the command execution process in the virtual environment and then call the command execution process to execute the command execution code to control the terminal to perform an operation task corresponding to the request information. When the command execution code is executed in the virtual environment, a corresponding command execution process can be started in the operating system executed in each virtual environment, and the command execution process can be used to automatically simulate the target object performing control operations on the terminal. For example, if the terminal is a computer with a mouse and keyboard, the command execution process can be used to automatically simulate the target object performing control operations on the computer using the mouse and keyboard. Simulating the target object controlling the terminal to perform an operation task corresponding to the request information may mean that the target object can control various operation tasks performed by the terminal, including multiple different types of operation tasks such as opening an application, entering text, clicking a button, and dragging a window.

[0161] Step 944: Update the execution state.

[0162] In this step, updating the execution status refers to updating the content of the execution status table entry in the execution record table to "completed" and writing the execution result into the execution result table entry after simulating that the target object controls the terminal's execution completion for the operation task corresponding to the request information. After simulating that the target object controls the terminal's execution completion for the operation task corresponding to the request information, the terminal can update the execution status of the command execution code in the execution record table to "completed" and write the corresponding execution result into the execution result table entry in the execution record table. By using the execution record table to record and update the execution status of the command execution code, real-time management and control of the execution status of the command execution code can be realized, which is advantageous for the terminal to automatically execute the operation task corresponding to the request information.

[0163] Step 945: Turn off the virtual environment.

[0164] In this step, turning off the virtual environment refers to the terminal turning off the virtual environment for executing the instruction execution code after simulating the target object controlling the terminal to complete the execution of the operation task corresponding to the request information, which can prevent resource waste by turning off the virtual environment after simulating the target object controlling the terminal to complete the execution of the operation task corresponding to the request information.

[0165] The processing of steps 941 to 945 allows the terminal to simulate the target object controlling the terminal to perform the operation task corresponding to the request information, and this process is completed automatically, without the target object needing to perform any additional operations, and without the target object needing to perform multiple repeated interactions, thereby improving the interaction efficiency between the target object and the terminal and achieving the purpose of simplifying the operation of the target object, thereby improving the user experience of the target object.

[0166] Step 950: After simulating the target object's control so that the terminal completes the execution of the operation task corresponding to the request information, write the execution result into the database and record the system-wide log information into a log file.

[0167] In this step, the execution results of simulating the target object controlling the terminal to execute the operation task corresponding to the request information are written to the database, and the log information of the entire system is stored in a log file, thereby realizing the recording of information on the entire processing process. This allows the execution status of the terminal to be managed and controlled based on the log information in the log file, thereby realizing daily troubleshooting and status recording of the execution of the terminal.

[0168] In this embodiment, a language model-based command execution method including the aforementioned steps 910 to 950 is used to, when receiving a voice signal of a target object, first convert the voice signal into request information that can be recognized by a language model, then construct targeted guidance presentation information, and then invoke the pre-trained language model to perform command recognition on the request information based on the guidance presentation information to obtain an executable command message structure; the executable command message structure is obtained by using the language model to perform command recognition on the request information based on the guidance presentation information, thereby improving the reliability of the command message structure obtained by recognition and obtaining a command message structure that is more closely matched with the request information. Furthermore, after obtaining the executable command message structure, a command execution code corresponding to the command message structure is first executed in a virtual environment to simulate the target object controlling the terminal to perform an operation task corresponding to the request information. After simulating the target object completing the execution of the operation task corresponding to the request information by the terminal, the execution result is written to a database and the log information of the entire system is recorded in a log file. By writing the execution result of simulating the target object controlling the terminal to perform the operation task corresponding to the request information to a database and recording the log information of the entire system in a log file, information recording of the entire processing process is realized, and the execution status of the terminal can be managed and controlled based on the log information in the log file, thereby realizing daily troubleshooting and status recording of terminal execution. Furthermore, because the command execution method is automatically completed by the terminal, it does not require the target object to perform any additional operations or perform multiple repeated interactions, which improves the efficiency of interaction between the target object and the terminal and achieves the purpose of simplifying the operation of the target object, thereby improving the user experience of the target object.

[0169] 10, which is a flowchart of steps of a language model-based instruction execution method provided in one embodiment, which may include, but is not limited to, steps 1001 to 1009.

[0170] Step 1001: Receive request information from a target object.

[0171] Step 1002: Model call request information is generated based on the request information and predetermined guide presentation information.

[0172] Step 1003: Send the model call request information to a language model call interface to call the language model, and perform command recognition on the request information based on the guide presentation information to obtain candidate execution commands.

[0173] Step 1004: Obtain an execution instruction waiting to be checked by replacing the instruction function function in the candidate execution instruction with an empty function whose function result is always true, then call the instruction execution engine to simulate the execution of the execution instruction waiting to be checked to obtain the simulated execution result, and then obtain the feasibility check result based on the simulated execution result.

[0174] In addition, when performing a feasibility check on a candidate executable instruction, the operation corresponding to the instruction function function in the candidate executable instruction may be relatively complicated, so if the instruction execution engine is directly called to simulate the execution of the candidate executable instruction, there is a problem that the execution time will be long, which may affect the efficiency of performing the feasibility check on the candidate executable instruction. Therefore, in order to save the time for performing the feasibility check on the candidate executable instruction and improve the efficiency of performing the feasibility check on the candidate executable instruction, the instruction function function in the candidate executable instruction may be replaced with a null function whose function result is always true to obtain a check-awaiting executable instruction. In this way, when the instruction execution engine is called to simulate the execution of the check-awaiting executable instruction, the executableness of the check-awaiting executable instruction can be quickly verified, thereby improving the efficiency of performing the feasibility check on the candidate executable instruction.

[0175] Step 1005: Determine whether the candidate execution instruction passes the feasibility check based on the result of the feasibility check. If the candidate execution instruction does not pass the feasibility check, execute step 1006. If the candidate execution instruction passes the feasibility check, execute step 1007.

[0176] In one embodiment, if the result of the feasibility check is "the candidate execution command fails to pass the feasibility check", it indicates that the candidate execution command currently (in this round) obtained by recognition using the language model is incorrect and is not a valid executable command, so step 1006 can be executed, i.e., the language model can be called to perform command recognition again on the request information based on the candidate execution command and the guide presentation information; and if the result of the feasibility check is "the candidate execution command passes the feasibility check", it indicates that the candidate execution command currently obtained by recognition based on the language model is accurate and is a valid executable command, so step 1007 can be executed.

[0177] Step 1006: Call the language model to perform command recognition again on the request information based on the candidate execution commands and the guide presentation information to obtain new candidate execution commands, and then return to the execution of step 1004.

[0178] In one embodiment, when the language model is called to perform command recognition again on the request information based on the candidate execution commands and the guide presentation information, the model call request information can be regenerated based on the candidate execution commands, the target request information, and the predetermined guide presentation information obtained by the recognition, and then the regenerated model call request information can be sent to the call interface of the language model to call the language model and perform command recognition again on the request information based on the candidate execution commands and the guide presentation information. Whereas, when the language model is called to perform command recognition again on the request information multiple times, the model call request information can be regenerated based on the multiple candidate execution commands, the request information, and the guide presentation information obtained by the previous recognition, and then the regenerated model call request information can be sent to the call interface of the language model to call the language model, and command recognition can be regenerated on the request information based on the multiple candidate execution commands and the guide presentation information obtained by the previous recognition, and then the multiple candidate execution commands obtained by the previous recognition can be used to improve the reliability of the candidate execution commands newly obtained by the recognition.

[0179] Step 1007: The candidate execution instruction that passes the feasibility check is set as the target execution instruction.

[0180] In one embodiment, when the feasibility check result obtained by performing a feasibility check on a candidate execution command is "the candidate execution command passes the feasibility check," it means that the candidate execution command is an execution command that matches the request information of the target object, so there is no need to call a language model and perform command recognition on the request information again, and the terminal can be directly controlled to execute the operation task corresponding to the request information based on the candidate execution command. Therefore, by setting the candidate execution command that passes the feasibility check as the target execution command, in a subsequent step, the terminal can be controlled to execute the operation task corresponding to the request information based on the target execution command.

[0181] Step 1008: Obtain corresponding command execution code according to the target execution command, execute the command execution process in the virtual environment, and then call the command execution process to execute the command execution code to simulate the target object controlling the terminal to perform the operation task corresponding to the request information.

[0182] In one embodiment, there is a corresponding command execution code for each target execution command, and the command execution code records the process of each step of executing the target execution command. Therefore, by first obtaining the command execution code corresponding to the target execution command, and then executing the command execution code in a virtual environment, the purpose of simulating the target object controlling the terminal to perform the operation task corresponding to the request information can be achieved.

[0183] In one embodiment, to achieve the concurrency of different execution commands, multiple target execution commands are obtained, and the command execution code corresponding to each target execution command is called. Then, multiple virtual environments (for example, virtual mirrors of operating systems with desktops) are launched simultaneously, and the command execution code of different target execution commands can be executed simultaneously in different virtual environments, so that the parallel execution of different operating tasks can be realized in the manner of simultaneous execution in multiple operating system mirrors.

[0184] Step 1009: Generate an execution record table including a target execution command, a command execution code, an execution status table entry, and an execution result table entry. After controlling the terminal to complete the execution of the operation task corresponding to the request information, update the content of the execution status table entry in the execution record table to completed, and write the execution result into the execution result table entry.

[0185] In one embodiment, an execution record table is used to record and update the execution status of the command execution code, thereby realizing real-time management and control of the execution status of the command execution code, which is advantageous for the terminal to automatically execute the operation task corresponding to the request information.

[0186] According to the language model-based command execution method including the above-mentioned steps 1001 to 1009, when request information of a target object is received, at least one round of target operations is executed until a candidate execution command that passes the feasibility check is obtained. In the first round of target operations, the language model is first invoked to perform command recognition on the request information based on predetermined guide presentation information to obtain candidate execution commands for the target operation of the first round. The candidate execution commands are obtained by the language model performing command recognition on the request information based on the guide presentation information, so that the reliability of the candidate execution commands can be improved, which is advantageous to obtaining candidate execution commands that match the request information more closely. Next, the feasibility check is performed on the candidate execution commands for the target operation of the first round. A check is performed to determine whether the candidate execution command passes the feasibility check; if the candidate execution command does not pass the feasibility check, it indicates that the candidate execution command obtained by the language model recognition at the current (current round) is not accurate; at this time, the target operation of the next round is continued; in the target operation of the next round, the language model is called to perform command recognition again on the request information based on the candidate execution command in the previous target operation and the guide presentation information to obtain a candidate execution command for the target operation of this round; then a feasibility check is performed again on the candidate execution command to determine whether the candidate execution command passes the feasibility check; in this way, several rounds of target operations are repeatedly performed until a candidate execution command that passes the feasibility check is obtained.Furthermore, after obtaining a candidate execution command that passes the feasibility check, the candidate execution command is taken as a target execution command, and the terminal is controlled to execute the operation task corresponding to the request information based on the target execution command. The candidate execution command that passes the feasibility check is obtained by calling the language model multiple times to perform command recognition on the request information based on the guide presentation information and the previous candidate execution command, so the reliability of the candidate execution command can be guaranteed. In addition, this process is completed automatically, and does not require the target object to perform any additional operations, nor does it require the target object to perform multiple repeated interactions. This effectively improves the interaction between the target object and the terminal, and achieves the purpose of simplifying the operation of the target object, thereby improving the user experience of the target object.

[0187] Below, some practical examples are given to illustrate application scenarios of the embodiments of the present application.

[0188] It should be noted that the language model-based command execution method provided in the embodiments of the present application can be applied to different scenarios, such as web search, voice chat, etc. The following description will be given taking the web search scenario and voice chat scenario as examples.

[0189] <Scenario 1 (Web search)> The language model-based command execution method provided in the embodiments of the present application can be applied to web search scenarios. For example, suppose a user issues a request to search for weather information on a web page to a terminal through an AI interaction module of the terminal. In this case, the terminal first receives the request information through the AI ​​interaction module, and then executes the command execution method provided in the embodiments of the present application to obtain a target execution command corresponding to the request information, and controls the terminal to perform an operation task corresponding to the request information based on the target execution command. When controlling the terminal to perform an operation task corresponding to the request information, it first invokes an application programming interface to obtain the screen resolution of the display module and obtains the initial coordinates of the operation cursor based on the screen resolution, and then determines the target coordinates of the search input box on the web page. Then, it simulates the user controlling the operation cursor to move from the initial coordinates to the target coordinates, so that the operation cursor is positioned in the search input box on the web page. This simulates the user checking the weather information through the search input box on the web page. At this time, the web page can jump to a page displaying the weather information, and the user can obtain the desired weather information through the page.

[0190] <Scenario 2 (Voice Chat)> The language model-based command execution method provided in the embodiments of the present application can also be applied to a voice chat scenario. For example, suppose a user chats with a friend through an instant communication application of a terminal. In this case, when the user inputs request information for sending chat information in the instant communication application through an AI interaction module of the terminal, the terminal first receives the request information through the AI ​​interaction module, and then executes the command execution method provided in the embodiments of the present application to obtain a target execution command corresponding to the request information, and controls the terminal to perform an operation task corresponding to the request information according to the target execution command. When controlling the terminal to perform an operation task corresponding to the request information, first call an application programming interface to obtain the screen resolution of the display module, obtain the initial coordinates of the operation cursor based on the screen resolution, then determine the target coordinates of the information input box in the instant communication application, and then simulate the user controlling the operation cursor to move from the initial coordinates to the target coordinates, thereby causing the operation cursor to be positioned in the information input box in the instant communication application, and simulate the user sending chat information through the information input box, and at this time, the chat information can be displayed in the chat interface of the instant communication application.

[0191] As can be understood, although the steps in the above flowcharts are displayed sequentially according to the arrows, these steps are not necessarily executed sequentially according to the order of the arrows. Unless otherwise clearly stated in this embodiment, the execution of these steps is not restricted to the order, and these steps may be executed in other orders. Furthermore, at least some of the steps in the above flowcharts may include multiple steps or multiple stages, and these steps or stages may not necessarily be executed at the same time but may be executed at different times. The execution of these steps or stages may also not necessarily be executed sequentially, and may, for example, be executed alternately with other steps or at least some of the steps or stages in other steps.

[0192] Referring to FIG. 11, an embodiment of the present application further discloses an instruction execution device based on a language model, the instruction execution device 1100 can realize the instruction execution method based on a language model in the above embodiment, and the instruction execution device 1100 includes: An information receiving unit 1110: receives request information of a target object; a command processing unit 1120: executing at least one round of target operations until obtaining a candidate execution command that passes a feasibility check; for a first round of target operations, invoking the language model to perform command recognition on the request information based on predetermined guide presentation information to obtain a candidate execution command for the target operation of the first round; performing the feasibility check on the candidate execution command for the target operation of the first round to determine whether the candidate execution command for the target operation of the first round passes the feasibility check; for an i-th round of target operations, invoking the language model to perform command recognition on the request information based on the candidate execution command for the target operation of the previous i-1 rounds and the guide presentation information to obtain a candidate execution command for the target operation of the i-th round; performing the feasibility check on the candidate execution command for the target operation of the i-th round to determine whether the candidate execution command for the target operation of the i-th round passes the feasibility check, wherein i is an integer greater than 1; and The command execution unit 1130 determines the candidate command that passes the feasibility check as a target command, and controls the terminal to execute the operation task corresponding to the request information according to the target command.

[0193] In one embodiment, the instruction processing unit 1120 is specifically used to: Replace the instruction function function in the candidate execution instruction with the target empty function to obtain the check-waiting execution instruction; Invoke the instruction execution engine to simulate the execution of the check-waiting execution instruction and obtain the simulated execution result; and Based on the simulated execution results, it is determined whether the candidate execution instruction passes the feasibility check.

[0194] In one embodiment, the instruction processing unit 1120 is specifically used to: determining that the candidate execution instruction has passed the feasibility check when the simulated execution result is "the execution instruction waiting for check has been successfully executed"; or When the simulated execution result is "the execution instruction waiting to be checked cannot be executed successfully," it is determined that the candidate execution instruction cannot pass the feasibility check.

[0195] In one embodiment, the instruction execution unit 1130 is further used to: Obtaining a corresponding instruction execution code according to the target execution instruction; and The command execution code is executed in the virtual environment to control the terminal to perform the operation task corresponding to the request information.

[0196] In one embodiment, the instruction execution unit 1130 is further used to: Running an instruction execution process in a virtual environment; and The command execution process is called to execute the command execution code, and controls the terminal to perform the operation task corresponding to the request information.

[0197] In one embodiment, the terminal includes a display module, the display module is used to display an operation cursor, and the instruction execution unit 1140 is further used to: Get the initial coordinates of the operation cursor; Determine a target coordinate, where the target coordinate is a coordinate when performing an operation task corresponding to the request information; Controlling the operation cursor to move from the initial coordinates to the target coordinates; and The terminal is controlled to execute an operation task corresponding to the request information at the target coordinates.

[0198] In one embodiment, the instruction execution unit 1130 is further used to: Calling the application programming interface to obtain the screen resolution of the display module; and Get the initial coordinates of the control cursor based on the screen resolution.

[0199] In one embodiment, the instruction execution unit 1100 further includes: a record table generating unit: generating an execution record table including a target execution instruction and an instruction execution code, the execution record table further including an execution state table entry and an execution result table entry; and A record table updating unit: after the terminal completes the execution of the operation task corresponding to the request information, the execution record table updates the content of the execution status table entry to "completed" and controls to write the execution result to the execution result table entry.

[0200] In one embodiment, the instruction processing unit 1120 is further used to: Generate model call request information based on the request information and the predetermined guide presentation information; and The model call request information is sent to a language model call interface to call the language model, and command recognition is performed on the guide presentation information request information to obtain candidate execution commands.

[0201] In one embodiment, the information receiving unit 1110 is further used to: receiving audio information input by a target subject; Invoke a pre-trained speech recognition model to perform speech recognition on the speech information to obtain text information; and The text information is processed by adding punctuation, correcting spelling, and correcting word order to obtain target subject request information.

[0202] In one embodiment, the information receiving unit 1110 is further used to: performing at least one of noise reduction and speech segmentation on the speech information to obtain pre-processed speech information; Invoke a speech feature extraction model to perform speech feature extraction on the pre-processed speech information to obtain speech feature information; and A pre-trained speech recognition model is called to perform speech recognition on the speech feature information to obtain text information.

[0203] Since the language model-based instruction execution device 1100 of this embodiment can realize the language model-based instruction execution method of the above-mentioned embodiment, the language model-based instruction execution device 1100 of this embodiment has the same technical principles and technical effects as the language model-based instruction execution method of the above-mentioned embodiment, and detailed description thereof will be omitted here.

[0204] Referring to FIG. 12 , an embodiment of the present application further discloses a computer device 1200 for performing the language model-based instruction execution method provided in the embodiment of the present application, the computer device 1200 including: at least one processor 1201; and At least one memory 1202 is coupled to the processor 1201 and stores at least one program.

[0205] The at least one program, when executed by the at least one processor 1201, may implement the instruction execution method based on the language model described above.

[0206] An embodiment of the present application further discloses a computer-readable storage medium having stored therein a computer program executable by a processor, which, when executed by the processor, may realize the instruction execution method based on the above-mentioned language model.

[0207] An embodiment of the present application further discloses a computer program product, which includes a computer program or computer instructions, the computer program or computer instructions being stored in a computer-readable storage medium, and a processor of a language model-based instruction execution device reading the computer program or computer instructions from the computer-readable storage medium and executing the computer program or computer instructions, thereby causing the instruction execution device to realize the above-mentioned language model-based instruction execution method.

[0208] The terms "first," "second," "third," "fourth," etc. (when present) in the specification of this application and in the above-mentioned drawings are used to distinguish between similar objects and are not used to limit a particular order. It will be understood that data used in this manner can be interchanged where appropriate, so that embodiments of the application not described herein may, for example, be performed according to orders other than those illustrated or described herein. Also, the terms "comprise" and "have," and any variations thereof, are intended to cover non-exclusive inclusions; for example, a process, method, system, product, or apparatus comprising a series of steps or units is not necessarily limited to the explicitly listed steps or units, and may include other steps or units not explicitly listed or inherent to those processes, methods, products, or apparatus.

[0209] As will be understood, in this application, "at least one" refers to one or more, and "multiple" refers to two or more. "And / or" is used to describe a relationship between related objects and indicates that there are three types of relationships. For example, "A and / or B" can represent three cases: A only exists, B only exists, and A and B simultaneously exist, where A and B may occur one or more times. The character " / " generally indicates that the related objects before and after it have an "or" relationship. "At least one of the following terms" or similar expressions refers to any combination of these terms, including any combination of one term or multiple terms. For example, at least one of a, b, or c can represent a, b, c, "a and b," "a and c," "b and c," or "a and b and c," where a, b, and c may occur one or more times.

[0210] In some embodiments provided in this application, it is understood that the disclosed systems, devices, and methods may be realized in other ways. For example, the above-described device embodiments are merely examples, and the division of the units is merely a logical division. In actual implementation, other division methods may be used. For example, multiple units or assemblies may be integrated or integrated into another system, or some features may be omitted or not implemented. Furthermore, the connections, direct connections, or communicative connections between each other shown or described may be indirect connections or communicative connections between several interfaces, devices, or units, and may be electrical, mechanical, or other types.

[0211] The above units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, i.e., located in one place or distributed across multiple network units. Depending on actual needs, some or all of the units may be selected to achieve the objectives of the technical solution of this embodiment.

[0212] Furthermore, each functional unit in each embodiment of the present application may be integrated into one processing unit, each unit may exist physically alone, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be realized in the form of hardware or in the form of a software functional unit.

[0213] The above-mentioned integrated unit can be realized in the form of a software functional unit and stored in a computer-readable storage medium when sold or used as an independent product. Based on this understanding, the technical solution of the present application, or all or part of the technical solution that has contributed to the prior art, may be embodied in the form of a software product, and the computer software product is stored in a storage medium and includes multiple instructions for causing a computer device (which may be a personal computer, a server, a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. In addition, the above-mentioned storage medium includes various media that can store program code, such as a U disk, a mobile hard disk, a ROM (Read-Only Memory), a RAM (Random Access Memory), a magnetic disk, and an optical disk.

[0214] Although the preferred embodiment of the present application has been described above, the present application is not limited to this embodiment, and any modification to the present application falls within the technical scope of the present application as long as it does not depart from the spirit of the present application.

Claims

1. 1. A method for executing instructions based on a language model, executed by a computing device, comprising: receiving target object request information; a step of executing at least one round of target operations until a candidate execution instruction that passes a feasibility check is obtained, wherein for a first round of target operations, the language model is invoked to perform command recognition on the request information based on predetermined guide presentation information to obtain a candidate execution instruction for the target operation of the first round, and the feasibility check is performed on the candidate execution instruction for the target operation of the first round to determine whether the candidate execution instruction for the target operation of the first round passes the feasibility check; for an i-th round of target operations, the language model is invoked to perform command recognition on the request information based on the candidate execution instruction for the target operation of the previous (i-1) round and the guide presentation information to obtain a candidate execution instruction for the target operation of the i-th round, and the feasibility check is performed on the candidate execution instruction for the target operation of the i-th round to determine whether the candidate execution instruction for the target operation of the i-th round passes the feasibility check, wherein i is an integer greater than 1; and The method includes a step of determining the candidate execution command that passes the feasibility check as a target execution command, and controlling the terminal to execute the operation task corresponding to the request information based on the target execution command.

2. 2. The method of claim 1, The feasibility check includes: replacing the instruction function function in the candidate execution instruction with a target empty function to obtain a check-waiting execution instruction; calling an instruction execution engine to simulate the execution of the check-waiting execution instruction and obtain a simulated execution result; and determining whether the candidate execution instruction passed the feasibility check based on the simulated execution results.

3. 3. The method of claim 2, determining whether the candidate execution instruction passes the feasibility check based on the simulated execution result, determining that the candidate execution instruction has passed the feasibility check if the simulated execution result is successful execution of the execution instruction awaiting check; or determining that the candidate execution instruction cannot pass the feasibility check if the simulated execution result is that the check-awaiting execution instruction cannot be successfully executed.

4. 2. The method of claim 1, The step of controlling the terminal to execute an operation task corresponding to the request information according to the target execution command includes: Obtaining a corresponding instruction execution code according to the target execution instruction; and The method includes a step of executing the instruction execution code in a virtual environment to control the terminal to perform an operation task corresponding to the request information.

5. 5. The method of claim 4, The step of executing the command execution code in the virtual environment to control the terminal to perform an operation task corresponding to the request information includes: executing an instruction execution process in the virtual environment; and The method includes the step of calling the instruction execution process to execute the instruction execution code, and controlling the terminal to perform an operation task corresponding to the request information.

6. 5. The method of claim 4, The terminal includes a display module, the display module is used to display an operation cursor; The step of controlling the terminal to execute an operation task corresponding to the request information includes: obtaining the initial coordinates of the operation cursor; determining a target coordinate, the target coordinate being a coordinate when performing an operation task corresponding to the request information; Controlling the operation cursor to move from the initial coordinates to the target coordinates; and The method includes controlling the terminal to perform an operation task corresponding to the request information at the target coordinates.

7. 7. The method of claim 6, The step of acquiring the initial coordinates of the operation cursor includes: calling an application programming interface to obtain the screen resolution of the display module; and The method includes obtaining initial coordinates of the manipulation cursor based on the screen resolution.

8. 5. The method of claim 4, The method further comprises: generating an execution record table including the target execution instruction and the instruction execution code, the execution record table further including an execution state table entry and an execution result table entry; and a step of controlling the terminal to update the content of the execution status table entry in the execution record table to "completed" after the terminal completes execution of the operation task corresponding to the request information, and to write the execution result to the execution result table entry.

9. 2. The method of claim 1, The step of calling the language model and performing command recognition on the request information based on predetermined guide presentation information to obtain candidate execution commands for the target operation of the first round includes: generating model call request information based on the request information and the guide presentation information; The method includes a step of sending the model invocation request information to an invocation interface of the language model to invoke the language model, and causing the language model to perform command recognition on the request information based on the guide presentation information to obtain the candidate execution command.

10. 2. The method of claim 1, The step of receiving the target request information includes: receiving audio information input by the target object; Invoking a pre-trained speech recognition model to perform speech recognition on the speech information to obtain text information; and The method includes performing at least one of adding punctuation, correcting spelling, and correcting word order on the text information to obtain the request information.

11. 11. The method of claim 10, Invoking the pre-trained speech recognition model to perform speech recognition on the speech information to obtain text information, performing at least one of noise reduction and speech segmentation on the speech information to obtain pre-processed speech information; Invoking a speech feature extraction model to perform speech feature extraction on the pre-processed speech information to obtain speech feature information; and Invoking the pre-trained speech recognition model to perform speech recognition on the speech feature information to obtain the text information.

12. 1. An apparatus for executing instructions based on a language model, comprising: an information receiving unit for receiving request information of the target object; an instruction processing unit for executing at least one round of target operations until a candidate execution instruction that passes a feasibility check is obtained, wherein for a first round of target operations, the instruction processing unit invokes the language model to perform instruction recognition on the request information based on predetermined guide presentation information to obtain a candidate execution instruction for the target operation of the first round, and performs the feasibility check on the candidate execution instruction for the target operation of the first round to determine whether the candidate execution instruction for the target operation of the first round passes the feasibility check; for an i-th round of target operations, the instruction processing unit invokes the language model to perform instruction recognition on the request information based on the candidate execution instruction for the target operation of the previous (i-1) round and the guide presentation information to obtain a candidate execution instruction for the target operation of the i-th round, and performs the feasibility check on the candidate execution instruction for the target operation of the i-th round to determine whether the candidate execution instruction for the target operation of the i-th round passes the feasibility check, wherein i is an integer greater than 1; and the candidate execution instruction that passes the feasibility check is set as a target execution instruction, and an instruction execution unit is configured to control a terminal to execute an operation task corresponding to the request information based on the target execution instruction.

13. A computer device comprising: a processor; and a memory coupled to the processor; The storage device stores a computer program, A computing device, wherein the processor is configured to execute a computer program to implement the method of any one of claims 1 to 11.

14. A program for causing a computer to execute the method according to any one of claims 1 to 11.

Images