Instruction execution method and apparatus based on language model, and storage medium

Abstract

Disclosed in the present application are an instruction execution method and apparatus based on a language model, and a storage medium. The method comprises: when requirement information for a target object is received, calling a language model to perform instruction identification on the requirement information according to guide prompt information, so as to obtain a candidate execution instruction; performing an executability check on the candidate execution instruction, wherein when the candidate execution instruction does not pass the executability check, the language model is called to perform another instance of instruction identification on the requirement information according to the candidate execution instruction and the guide prompt information, so as to obtain a new candidate execution instruction, and perform another instance of the executability check on the new candidate execution instruction until the executability check is passed; and taking a candidate execution instruction which passes the executable check as a target execution instruction, and controlling, according to the target execution instruction, a terminal to execute an operation task corresponding to the requirement information. By means of the embodiments of the present application, the interaction efficiency and the usage experience of a user can be improved. The embodiments of the present application can be applied to various application scenarios in which interaction is performed with artificial intelligence.

Description

Instruction execution method, device and storage medium based on language model

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on June 30, 2023, with application number 2023108010693 and application name “Instruction execution method, device and storage medium based on language model”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of artificial intelligence technology, and in particular to natural language processing technology. Background Art

[0003] With the development of artificial intelligence technology, many terminal devices are able to support interaction with users. For example, users can interact with terminal devices through voice or text. When the terminal device recognizes the user's demand instructions, the terminal device can perform different operations such as opening applications, checking the weather, or playing music according to the recognized demand instructions.

[0004] Terminal devices typically identify user requests based on neural network models. However, neural network models are not always able to accurately identify user requests. When the neural network model cannot accurately identify the user's request, the user needs to re-interact with the terminal device using another description of the request until the neural network model can accurately identify the user's request. During this process, the user needs to interact with the terminal device repeatedly in order for the terminal device to correctly perform the user's desired operation, resulting in low interaction efficiency and poor user experience.

[0005] Summary of the Invention

[0006] The embodiments of the present application provide a language model-based instruction execution method, device, and storage medium, which can improve interaction efficiency and thus improve the user experience.

[0007] In one aspect, an embodiment of the present application provides a method for executing instructions based on a language model, which is executed by a computer device and includes the following steps:

[0008] Receive demand information from the target object;

[0009] Execute at least one round of target operations until candidate execution instructions that pass the feasibility check are obtained; in the first round of target operations, call the language model, perform instruction recognition on the requirement information according to preset guidance prompt information, obtain candidate execution instructions for the first round of target operations, perform the executable check on the candidate execution instructions for the first round of target operations, and determine whether the candidate execution instructions for the first round of target operations pass the feasibility check; in the i-th round of target operations, call the language model, perform instruction recognition on the requirement information according to the candidate execution instructions of the previous i-1 rounds of target operations and the guidance prompt information, obtain candidate execution instructions for the i-th round of target operations, perform the executable check on the candidate execution instructions for the i-th round of target operations, and determine whether the candidate execution instructions for the i-th round of target operations pass the feasibility check, where i is an integer greater than 1;

[0010] The candidate execution instruction that passes the executable check is used as the target execution instruction, and according to the target execution instruction, the terminal is controlled to execute the operation task corresponding to the requirement information.

[0011] On the other hand, an embodiment of the present application further provides an instruction execution device based on a language model, comprising:

[0012] An information receiving unit, configured to receive demand information from a target object;

[0013] an instruction processing unit configured to execute at least one round of target operations until candidate execution instructions that pass the feasibility check are obtained; in a first round of target operations, calling the language model, performing instruction recognition on the requirement information according to preset guidance prompt information, obtaining candidate execution instructions for the first round of target operations, performing the executable check on the candidate execution instructions for the first round of target operations, and determining whether the candidate execution instructions for the first round of target operations pass the feasibility check; in an i-th round of target operations, calling the language model, performing instruction recognition on the requirement information according to the candidate execution instructions for the i-1th round of target operations and the guidance prompt information, obtaining candidate execution instructions for the i-th round of target operations, performing the executable check on the candidate execution instructions for the i-th round of target operations, and determining whether the candidate execution instructions for the i-th round of target operations pass the feasibility check, where i is an integer greater than 1;

[0014] The instruction execution unit is used to take the candidate execution instruction that passes the executable check as the target execution instruction, and control the terminal to execute the operation task corresponding to the requirement information according to the target execution instruction.

[0015] On the other hand, an embodiment of the present application further provides a computer device, including:

[0016] at least one processor;

[0017] at least one memory for storing at least one program;

[0018] When at least one of the programs is executed by at least one of the processors, the above-mentioned instruction execution method based on the language model is implemented.

[0019] On the other hand, an embodiment of the present application further provides a computer-readable storage medium, which stores a computer program executable by a processor. When the processor-executable computer program is executed by the processor, it is used to implement the instruction execution method based on the language model as described above.

[0020] On the other hand, an embodiment of the present application also provides a computer program product, including a computer program or computer instructions, wherein the computer program or the computer instructions are stored in a computer-readable storage medium, and the processor of the instruction execution device based on the language model reads the computer program or the computer instructions from the computer-readable storage medium, and the processor executes the computer program or the computer instructions, so that the instruction execution device executes the instruction execution method based on the language model as described above.

[0021] The embodiments of the present application include at least the following beneficial effects: when the demand information of the target object is received, at least one round of target operation is performed until a candidate execution instruction that passes the feasibility check is obtained; in the first round of target operation, the language model is first called to perform instruction recognition on the demand information according to the preset guidance prompt information, and the candidate execution instruction of the first round of target operation is obtained. Since the candidate execution instruction is obtained by the language model performing instruction recognition on the demand information according to the guidance prompt information, the confidence of the candidate execution instruction can be improved, thereby facilitating the acquisition of a candidate execution instruction with a higher degree of matching with the demand information; then, the candidate execution instruction of the first round of target operation is subjected to the feasibility check. Test to determine whether the candidate execution instruction passes the feasibility check. If the candidate execution instruction does not pass the executable check, it means that the candidate execution instruction identified by the current language model is not accurate. At this time, continue to the next round of target operation. In the next round of target operation, call the language model to re-identify the instruction of the requirement information according to the candidate execution instruction in the previous target operation and the guidance prompt information to obtain the candidate execution instruction in this round of target operation, and then re-perform the executable check on the candidate execution instruction to determine whether the candidate execution instruction passes the feasibility check. It iterates and executes several rounds of target operation until a candidate execution instruction that passes the executable check is obtained. Moreover, after obtaining the candidate execution instruction that passes the executable check, the candidate execution instruction is used as the target execution instruction, and the terminal is controlled to execute the operation task corresponding to the demand information according to the target execution instruction. Since the candidate execution instruction that passes the executable check is obtained by multiple calls to the language model to perform instruction recognition on the demand information based on the guidance prompt information and the previous candidate execution instructions, the confidence of the candidate execution instruction can be guaranteed. Moreover, since this process is completed automatically, the target object does not need to perform any additional operations, nor does it require the target object to perform multiple repeated interactions. Therefore, the interaction efficiency between the target object and the terminal can be improved, so as to achieve the purpose of simplifying the use of the target object, thereby improving the use experience of the target object. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] FIG1 is a schematic diagram of an implementation environment provided by an embodiment of the present application;

[0023] FIG2 is a schematic diagram of another implementation environment provided by an embodiment of the present application;

[0024] FIG3 is a flow chart of a method for executing instructions based on a language model according to an embodiment of the present application;

[0025] FIG4 is a schematic diagram of the structure and content of the guidance prompt information provided in an embodiment of the present application;

[0026] FIG5 is a schematic diagram of a process of replacing an instruction function with an empty function according to an embodiment of the present application;

[0027] FIG6 is a schematic diagram of an interface for interacting with a terminal through an artificial intelligence interaction module provided by one embodiment of the present application;

[0028] FIG7 is a schematic diagram of the content of a guidance prompt information provided in an embodiment of the present application;

[0029] FIG8 is a schematic diagram of an interface for interacting with a terminal through an artificial intelligence interaction module provided by another embodiment of the present application;

[0030] FIG9 is an overall flow chart of a method for executing instructions based on a language model according to an example of the present application;

[0031] FIG10 is a flowchart of a method for executing instructions based on a language model according to a specific example of the present application;

[0032] FIG11 is a schematic diagram of an instruction execution device based on a language model provided in an embodiment of the present application;

[0033] FIG12 is a schematic diagram of another instruction execution device based on a language model provided in an embodiment of the present application. DETAILED DESCRIPTION

[0034] The present application is further described below in conjunction with the accompanying drawings and specific embodiments. The described embodiments should not be considered as limiting the present application. All other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0035] When a user uses a terminal device to perform an operation, they generally need to manually operate the terminal device to complete the operation, such as manually dragging a window, manually opening an application, manually entering a query in a browser to obtain the desired search results, etc. Because these operations require the user to perform them manually, there are problems with low intelligent interaction and low interaction efficiency. Moreover, when the operation path of the operation task is long, the user needs to perform more complex operation steps to complete the operation task, resulting in a poor user experience.

[0036] To address these issues, related art proposes enabling interaction between users and terminal devices through voice or text, enabling the terminal device to recognize user requests during the interaction and, based on the recognized requests, perform various tasks such as opening an application, checking the weather, or playing music. In this related art, neural network models are generally used to identify user requests. However, the output of neural network models is probabilistic, and they may not always accurately identify user requests. When the neural network model fails to accurately identify a user request, the user is required to interact with the terminal device again using another representation of the request. For example, if the neural network model fails to accurately recognize the user's voice request "Play some music," the user is required to interact with the terminal device again using the voice command "Play music" until the neural network model accurately recognizes the user's request. This requires repeated interaction with the terminal device in order for the terminal device to correctly perform the desired task, resulting in low interaction efficiency and a poor user experience.

[0037] In order to improve the interaction efficiency between the user and the terminal device, and improve the user experience, the embodiment of the present application provides an instruction execution method based on a language model. In this method, when the demand information of the target object is received, at least one round of target operation is performed until a candidate execution instruction that passes the feasibility check is obtained; in the first round of target operation, the language model is first called to perform instruction recognition on the demand information according to the preset guidance prompt information, and the candidate execution instruction of the first round of target operation is obtained. Since the candidate execution instruction is obtained by the language model performing instruction recognition on the demand information according to the guidance prompt information, the confidence of the candidate execution instruction can be improved, thereby helping to obtain a candidate execution instruction with a higher degree of match with the demand information; then, The candidate execution instructions of the first round of target operations are checked for executableness to determine whether the candidate execution instructions pass the feasibility check. If the candidate execution instructions do not pass the executableness check, it means that the candidate execution instructions identified by the current language model are not accurate. At this time, continue to the next round of target operations. In the next round of target operations, call the language model to re-identify the instructions on the requirement information based on the candidate execution instructions in the previous target operations and the guidance prompt information to obtain the candidate execution instructions in this round of target operations. Then, re-check the executableness of the candidate execution instructions to determine whether the candidate execution instructions pass the feasibility check. It is iteratively performed for several rounds of target operations until a candidate execution instruction that passes the executable check is obtained. Moreover, after obtaining the candidate execution instruction that passes the executable check, the candidate execution instruction is used as the target execution instruction, and the terminal is controlled to execute the operation task corresponding to the demand information according to the target execution instruction. Since the candidate execution instruction that passes the executable check is obtained by multiple calls to the language model to perform instruction recognition on the demand information based on the guidance prompt information and the previous candidate execution instructions, the confidence of the candidate execution instruction can be guaranteed. Moreover, since this process is completed automatically, the target object does not need to perform any additional operations, nor does it require the target object to perform multiple repeated interactions. Therefore, the interaction efficiency between the target object and the terminal can be improved, so as to achieve the purpose of simplifying the use of the target object, thereby improving the use experience of the target object.

[0038] Referring to FIG. 1 , FIG. 1 is a schematic diagram of an implementation environment provided in an embodiment of the present application. The implementation environment may include a first user terminal 110 and a first server 120. The first user terminal 110 and the first server 120 may be directly or indirectly connected via wired or wireless communication. The first user terminal 110 and the first server 120 may be nodes in a blockchain, which is not specifically limited in this embodiment.

[0039] The first server 120 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. A trained language model can be deployed in the first server 120. The language model can be a large language model (LLM) that can interact by learning and understanding human language, or a natural language understanding model (NLU) that can perform semantic recognition on voice or text information input by the user, etc.

[0040] The first user terminal 110 may include, but is not limited to, a smartphone, a tablet, a computer, an intelligent voice interaction device, a smart home appliance, an in-vehicle terminal, an aircraft, etc. Optionally, the first user terminal 110 may be provided with an artificial intelligence interaction module 111. Through the artificial intelligence interaction module 111, interaction between the user and the first user terminal 110 may be achieved, so that the first user terminal 110 automatically performs the operation task desired by the user based on the demand information input by the user. For example, the artificial intelligence interaction module 111 may call a language model in the first server 120 to identify the demand information input by the user and obtain an execution instruction corresponding to the demand information, so that the first user terminal 110 can perform the operation task corresponding to the demand information based on the execution instruction.

[0041] Referring to Figure 2, which is a schematic diagram of another implementation environment provided by an embodiment of the present application, the implementation environment may include a second user terminal 210, wherein the second user terminal 210 may be a node in the blockchain, which is not specifically limited in this embodiment.

[0042] The second user terminal 210 may include, but is not limited to, a smartphone, tablet, computer, intelligent voice interaction device, smart home appliance, vehicle-mounted terminal, aircraft, etc. Optionally, the second user terminal 210 may be provided with an artificial intelligence interaction module 211, and may be deployed with a trained language model. This language model may be an LLM model that can interact by learning and understanding human language, or an NLU model that can perform semantic recognition of user-entered voice or text information. The artificial intelligence interaction module 211 enables interaction between the user and the second user terminal 210, allowing the second user terminal 210 to automatically execute the user's desired operation task based on the user's input request information. For example, the artificial intelligence interaction module 211 may call the language model in the second user terminal 210 to identify the user's input request information and obtain an execution instruction corresponding to the request information, so that the second user terminal 210 can execute the operation task corresponding to the request information according to the execution instruction.

[0043] Referring to Figure 1 or Figure 2, in an application scenario, after the first user terminal 110 or the second user terminal 210 receives the demand information input by the user, the first user terminal 110 can call the language model in the first server 120 through the artificial intelligence interaction module 111, and perform instruction recognition on the demand information according to the preset guidance prompt information to obtain candidate execution instructions; the second user terminal 210 can call the language model in the second user terminal 210 through the artificial intelligence interaction module 211, and perform instruction recognition on the demand information according to the preset guidance prompt information to obtain candidate execution instructions.

[0044] When the user inputs voice information, the first user terminal 110 or the second user terminal 210 first calls a pre-trained voice recognition model to perform voice recognition on the voice information to obtain text information, and then performs at least one of adding punctuation, spelling correction, or word order correction on the text information to obtain the required information. In addition, when calling the pre-trained voice recognition model to perform voice recognition on the voice information, the first user terminal 110 or the second user terminal 210 first performs at least one of noise removal and voice segmentation on the voice information to obtain pre-processed voice information, then calls the pre-trained voice feature extraction model to extract voice features from the pre-processed voice information to obtain voice feature information, and then calls the pre-trained voice recognition model to perform voice recognition on the voice feature information to obtain text information.

[0045] When the first user terminal 110 or the second user terminal 210 calls the language model to perform instruction recognition on the demand information according to the preset guidance prompt information to obtain candidate execution instructions, it can first generate model call request information based on the demand information and the preset guidance prompt information, and then send the model call request information to the calling interface of the language model in the first artificial intelligence content generation server 120 or the calling interface of the language model in the second user terminal 210, so as to call the language model according to the model call request information to perform instruction recognition on the demand information according to the guidance prompt information, thereby obtaining candidate execution instructions.

[0046] After the first user terminal 110 or the second user terminal 210 obtains the candidate execution instruction obtained by language model recognition, the first user terminal 110 or the second user terminal 210 first performs an executable check on the candidate execution instruction to obtain an executable check result. Among them, when the first user terminal 110 or the second user terminal 210 performs the executable check on the candidate execution instruction, it first replaces the instruction function function in the candidate execution instruction with an empty function whose function result is Yongzhen to obtain the execution instruction to be checked, and then calls the instruction execution engine to simulate the execution of the execution instruction to be checked to obtain a simulated execution result. If the simulated execution result is that the execution instruction to be checked is successfully executed, the executable check result that the candidate execution instruction passed the executable check can be obtained. If the simulated execution result is that the execution instruction to be checked is not successfully executed, the executable check result that the candidate execution instruction did not pass the executable check can be obtained.

[0047] After the first user terminal 110 or the second user terminal 210 performs an executable check on the candidate execution instruction and obtains an executable check result, if the executable check result is that the candidate execution instruction fails the executable check, the language model in the first server 120 or the second user terminal 210 is called, and the instruction is re-identified on the requirement information based on the candidate execution instruction and the guidance prompt information to obtain a new candidate execution instruction, and the executable check is re-performed on the new candidate execution instruction, and the above process is repeated until a candidate execution instruction that passes the feasibility check is obtained.

[0048] Next, the first user terminal 110 or the second user terminal 210 uses the candidate execution instruction that has passed the executable check as the target execution instruction, and the artificial intelligence interaction module 111 in the first user terminal 110 or the artificial intelligence interaction module 211 in the second user terminal 210 controls the first user terminal 110 or the second user terminal 210 to perform the operation task corresponding to the requirement information according to the target execution instruction. When the first user terminal 110 or the second user terminal 210 performs the operation task corresponding to the requirement information according to the target execution instruction, the corresponding instruction execution code can be first obtained according to the target execution instruction, and then the instruction execution code can be executed in a virtual environment to simulate a user controlling the user terminal to perform the operation task corresponding to the requirement information. In the process of the first user terminal 110 or the second user terminal 210 executing the instruction execution code in the virtual environment to simulate the user controlling the user terminal to perform the operation task corresponding to the requirement information, the first user terminal 110 or the second user terminal 210 first runs an instruction execution process in the virtual environment, and then calls the instruction execution process to run the instruction execution code to simulate the user controlling the user terminal to perform the operation task corresponding to the requirement information.

[0049] In the case where the first user terminal 110 or the second user terminal 210 includes a display module for displaying an operation cursor, the first user terminal 110 or the second user terminal 210 will first call the application programming interface to obtain the screen resolution of the display module when simulating the user controlling the user terminal to perform the operation task corresponding to the demand information, and obtain the initial coordinates of the operation cursor based on the screen resolution, and then determine the target coordinates when performing the operation task corresponding to the demand information, and then simulate the user controlling the operation cursor to move from the initial coordinates to the target coordinates, and simulate the user controlling the user terminal to perform the operation task corresponding to the demand information at the target coordinates.

[0050] In addition, when the artificial intelligence interaction module 111 or the artificial intelligence interaction module 211 controls the user terminal to execute the operation task corresponding to the demand information according to the target execution instruction, an execution record table including the target execution instruction and the instruction execution code will be created. The execution record table also includes an execution status table item and an execution result table item. When the first user terminal 110 or the second user terminal 210 completes the operation task corresponding to the demand information, the first user terminal 110 or the second user terminal 210 will update the content of the execution status table item in the execution record table to completed, and write the execution result into the execution result table item.

[0051] It should be noted that in each specific embodiment of the present application, when it comes to the need to perform relevant processing on data related to the characteristics of the target object such as the attribute information or attribute information set of the target object (such as a user, etc.), the permission or consent of the target object will be obtained first, and the collection, use and processing of such data will comply with relevant laws, regulations and standards. In addition, when the embodiment of the present application needs to obtain the attribute information of the target object, it will obtain the separate permission or separate consent of the target object through a pop-up window or jump to a confirmation page, etc., and after clearly obtaining the separate permission or separate consent of the target object, the relevant data of the target object used to enable the normal operation of the embodiment of the present application will be obtained.

[0052] FIG3 is a flowchart of a language model-based instruction execution method provided in an embodiment of the present application. The language model-based instruction execution method can be executed by a computer device, for example, by a terminal, or by a terminal and a server. In the embodiment of the present application, the method is described as being executed by a terminal. Referring to FIG3 , the language model-based instruction execution method may include, but is not limited to, steps 310 to 330.

[0053] Step 310: Receive the target object's demand information.

[0054] In one embodiment, the target object's demand information can be a direct control instruction from the target object to the terminal, or it can be a question that the target object asks the terminal to provide a corresponding answer, which is not specifically limited here. For example, when the demand information is a direct control instruction from the target object to the terminal, the demand information can be a command to directly control the terminal, such as "play music," "open a browser," or "set an alarm for 10:00 a.m."; for another example, when the demand information is a question that the target object asks the terminal to provide a corresponding answer, the demand information can be a question such as "what's the weather like tomorrow," "how do I get to location A," or "what buses are available to city B." The demand information "what's the weather like tomorrow" indicates that the target object wants the terminal to check tomorrow's weather conditions on a weather forecast website, the demand information "how do I get to location A" indicates that the target object wants the terminal to open a map application to query a navigation route from the current location to location A, and the demand information "what buses are available to city B" indicates that the target object wants the terminal to query bus schedules from the current city to city B on a bus website.

[0055] In one embodiment, the target object may be a user or an external terminal in communication with the terminal executing the instruction execution method, and the present invention is not specifically limited here. When the target object is a user, the target object's demand information may be text information directly input by the user, or may be text information obtained by recognizing voice information input by the user, and the present invention is not specifically limited here. When the target object is an external terminal in communication with the terminal executing the instruction execution method, the target object's demand information may be text information sent by the external terminal, or may be text information obtained by recognizing voice information sent by the external terminal, and the present invention is not specifically limited here.

[0056] In one embodiment, when the demand information is text information obtained by recognizing the voice information of the target object (for example, voice information input by a user or an external terminal), the terminal can first receive the voice information input by the target object, and then call a pre-trained voice recognition model to perform voice recognition on the voice information to obtain text information, and then add at least one of punctuation marks, spelling correction or word order correction to the text information to obtain the demand information.

[0057] Among them, the pre-trained speech recognition model is used to convert the audio feature vector of speech information into text information. The speech recognition model may include a hidden Markov model (HMM), a Gaussian mixture model, a deep neural network model, an n-gram language model or other statistical models commonly used in this field, and is not specifically limited here.

[0058] In one embodiment, the speech recognition model can also be obtained by combining an acoustic model and a language model in series, wherein the acoustic model is used to classify acoustic features into (decode) phonemes or words, and the language model is used to decode phonemes or words into a complete text. For example, the audio feature vector of the speech information can be first converted into an intermediate recognition result (such as a phoneme, a phoneme string or a sub-word, etc.) through the acoustic model, and then the intermediate recognition result can be converted into a text recognition result (such as a word, a word string or a symbol sequence, etc.) through the language model, and then the text information corresponding to the speech information of the target object is output.

[0059] In one embodiment, since the text information output by the speech recognition model is content such as words, word strings, or symbol sequences, there may be a problem that the demand information of the target object cannot be accurately expressed, which may easily affect the accuracy of subsequent command recognition of the demand information. To solve this problem, after obtaining the text information output by the speech recognition model, at least one of the following processes can be performed on the text information: adding punctuation marks, spelling correction, or word order correction, to obtain more accurate demand information. For example, assume that the speech information input by the target object is "I want to know what the weather will be like tomorrow. How should I inquire about the weather situation tomorrow?", then after the speech recognition model performs speech recognition on this speech information, the obtained text information may be "I want to know what the weather will be like tomorrow. How should I inquire about the weather situation tomorrow?", because there are no punctuation marks for sentence segmentation in this text information, and there are problems such as spelling mistakes and word order mistakes in this text information, so it will affect the recognition accuracy of this text information in the future. Therefore, at this time, punctuation marks can be added to this text information first. The text information after adding punctuation marks is "I want to know what the weather will be like tomorrow. How should I inquire about the weather situation tomorrow?", then, the spelling mistake "weather situation" in this text information is corrected to obtain the correct spelling "weather situation", and the text information after spelling correction is "I want to know what the weather will be like tomorrow. How should I inquire about the weather situation tomorrow?", then, the word order mistake "I want to know what the weather will be like tomorrow" in this text information is corrected to obtain the correct word order "I want to know what the weather will be like tomorrow", at this time, the accurate demand information can be obtained as "I want to know what the weather will be like tomorrow. How should I inquire about the weather situation tomorrow?".

[0060] In this way, through the above method, speech recognition is performed on the speech information input by the target object to obtain text information, and corresponding adjustment and correction processes are performed on the text information, which can ensure that the finally obtained text information accurately corresponds to the demand information of the target object, and correspondingly, ensure the accuracy of subsequent command recognition of the demand information.

[0061] In one embodiment, when calling a pre-trained speech recognition model to perform speech recognition on the speech information to obtain text information, since the speech information input by the target object may be mixed with interference information such as environmental noise, and the speech information input by the target object may be a long speech, these situations will affect the speech recognition effect of the speech recognition model. In order to improve the speech recognition accuracy of the speech recognition model, when calling the pre-trained speech recognition model to perform speech recognition on the speech information, the speech information can be first subjected to at least one of noise removal and speech segmentation processing to obtain pre-processed speech information, and then the speech feature extraction model is called to perform speech feature extraction on the pre-processed speech information to obtain speech feature information, and then the pre-trained speech recognition model is called to perform speech recognition on the speech feature information to obtain text information.

[0062] Among them, noise removal of speech information can filter out interference information such as environmental noise mixed in the speech information, thereby improving the purity of the speech information; speech segmentation of speech information can segment the long speech input by the target object into multiple short speech segments, thereby reducing the processing difficulty of speech feature extraction by the subsequent speech feature extraction model, and thus improving the accuracy and efficiency of speech feature extraction.

[0063] In one embodiment, a speech feature extraction model is called to extract speech features from the pre-processed speech information, with the purpose of converting the pre-processed speech information into speech feature information (such as an audio feature vector) in the form of a digital signal, so that the speech recognition model can subsequently perform more accurate recognition processing on the speech feature information. The speech feature extraction model can be a trained deep neural network model or a convolutional neural network model, or a mathematical model with speech feature extraction capabilities. It can be appropriately selected according to the actual application situation and is not specifically limited here. For example, when the speech feature extraction model is a mathematical model with speech feature extraction capabilities, the speech feature extraction model can be a mathematical model that can perform Mel Frequency Cepstral Coefficients (MFCC) processing on the pre-processed speech information, or it can be a mathematical model that can perform Discrete Wavelet Transform (DWT) processing on the pre-processed speech information, or it can be a mathematical model that can perform Perceptual Linear Prediction (PLP) processing on the pre-processed speech information, and is not specifically limited here. Mel-frequency cepstral coefficient processing involves obtaining the Mel-frequency cepstral coefficients of preprocessed speech information. Mel-frequency cepstral coefficients are a commonly used speech feature that can be used to represent the spectral characteristics of speech signals. Mel-frequency cepstral coefficients have excellent human perceptual properties and can effectively extract key features of speech signals. Discrete wavelet transform processing is a signal processing method based on wavelet analysis that can be used for feature extraction and noise reduction of speech signals. Perceptual linear prediction processing is a method for speech feature extraction that is based on the way the human ear perceives sound and takes into account the physiological and psychological characteristics of speech signals, effectively extracting important features of speech signals.

[0064] In one embodiment, the speech feature extraction model can also be a mathematical model that combines the mel-frequency cepstral coefficients and perceptual linear prediction of the preprocessed speech information. In this case, when the speech feature extraction model is called to perform speech feature extraction on the preprocessed speech information, the preprocessed speech information can be first subjected to mel-frequency cepstral coefficient processing and perceptual linear prediction processing respectively to obtain the mel-frequency cepstral coefficients and perceptual linear prediction feature parameters of the preprocessed speech information, and then the average values ​​of the mel-frequency cepstral coefficients and perceptual linear prediction feature parameters of the preprocessed speech information are calculated to obtain the speech feature information of the preprocessed speech information.

[0065] When the speech feature extraction model is called to perform Mel-frequency cepstral coefficient processing on the preprocessed speech information, a fast Fourier transform can be first performed on the time domain signal of the preprocessed speech information to obtain the frequency domain signal of the preprocessed speech information, and then a logarithmic operation can be performed on the frequency domain signal of the preprocessed speech information to obtain the logarithmic spectrum of the preprocessed speech information. Then, a discrete cosine transform can be performed on the logarithmic spectrum of the preprocessed speech information to obtain the Mel-frequency cepstral coefficients of the preprocessed speech information. When the speech feature extraction model is called to perform perceptual linear prediction processing on the preprocessed speech information, a fast Fourier transform can be first performed on the time domain signal of the preprocessed speech information to obtain the frequency domain signal of the preprocessed speech information, and then pre-emphasis and intensity-loudness conversion can be performed on the frequency domain signal of the preprocessed speech information to obtain loudness feature information of the preprocessed speech information. Then, an inverse Fourier transform and linear prediction can be performed on the loudness feature information of the preprocessed speech information to obtain perceptual linear prediction feature parameters of the preprocessed speech information. By performing Mel-frequency cepstral coefficient processing and perceptual linear prediction processing on the preprocessed speech information respectively, the corresponding processing results are obtained, and then the two processing results are averaged to obtain the average value of the two processing results. This is used as speech feature information, which can better improve the accuracy of speech feature extraction of the preprocessed speech information, thereby facilitating the subsequent speech recognition model to accurately recognize the speech feature information, and can improve the speech recognition accuracy of the speech recognition model for the speech information input by the target object.

[0066] Step 320: Execute at least one round of target operations until candidate execution instructions that pass the feasibility check are obtained; in the first round of target operations, call the language model, perform instruction recognition on the requirement information according to the preset guidance prompt information, obtain candidate execution instructions for the first round of target operations, perform an executable check on the candidate execution instructions for the first round of target operations, and determine whether the candidate execution instructions for the first round of target operations pass the feasibility check; in the i-th round of target operations, call the language model, perform instruction recognition on the requirement information according to the candidate execution instructions of the previous i-1 rounds of target operations and the guidance prompt information, obtain candidate execution instructions for the i-th round of target operations, perform an executable check on the candidate execution instructions for the i-th round of target operations, and determine whether the candidate execution instructions for the i-th round of target operations pass the feasibility check, where i is an integer greater than 1.

[0067] In this embodiment, the target operation refers to the operation of calling a language model to perform instruction recognition on the demand, obtaining candidate execution instructions, and performing feasibility verification on the candidate execution instructions. In actual application, the terminal may perform one or more rounds of the target operation until a candidate execution instruction that passes the feasibility verification is obtained. That is, if the terminal performs one round of the target operation and obtains a candidate execution instruction that passes the feasibility verification, the terminal does not need to continue to perform the target operation. If the terminal performs one or more rounds of the target operation and still does not obtain a candidate execution instruction that passes the feasibility verification, the terminal needs to continue to perform the target operation until a candidate execution instruction that passes the feasibility verification is obtained.

[0068] In one embodiment, after obtaining the demand information of the target object, the language model can be called to perform instruction recognition on the demand information according to the preset guidance prompt information to obtain candidate execution instructions, so that in subsequent steps the terminal can automatically perform the corresponding operation tasks according to the candidate execution instructions, avoiding multiple repeated interactions with the target object, thereby improving the interaction efficiency between the target object and the terminal.

[0069] In one embodiment, the language model used to perform instruction recognition on the demand information according to the guidance prompt information may be an LLM model or an NLU model, etc., which may be appropriately selected according to the actual application situation and is not specifically limited here.

[0070] In one embodiment, the language model used to identify the instructions of the demand information according to the guidance prompt information can be deployed in a cloud server connected to the terminal for communication, or can be deployed in the terminal, which is not specifically limited here. For example, when the storage space of the terminal is small and the processing power is weak, the language model can be deployed in the cloud server connected to the terminal for communication, which can reduce the storage space requirements and processing power requirements of the terminal, thereby improving the universal applicability of the instruction execution method based on the language model of this embodiment. For another example, when the storage space of the terminal is large and the processing power is high, the language model can be deployed in the terminal, so that the language model can be directly called in the terminal to realize the instruction identification of the demand information, which can save the interaction time between the terminal and the cloud server, thereby improving the response efficiency to the demand information.

[0071] In one embodiment, when calling a language model to perform instruction recognition on the requirement information according to preset guidance prompt information to obtain candidate execution instructions, a model call request information can be generated based on the requirement information and the preset guidance prompt information, and then the model call request information is sent to the calling interface of the language model, and the language model is called to perform instruction recognition on the requirement information according to the guidance prompt information to obtain candidate execution instructions.

[0072] In one embodiment, when the language model is deployed in a cloud server, the terminal can generate a model call request information in a Hyper Text Transfer Protocol (HTTP) message format based on the demand information and preset guidance prompt information, wherein the demand information and guidance prompt information are filled in the message body of the HTTP message. Then, the terminal sends the model call request information in the HTTP message format to the language model through the language model's call interface. When the language model receives the model call request information through the call interface, the language model will first parse the model call request information in the HTTP message format, extract the demand information and guidance prompt information from the message body of the HTTP message, and then perform instruction recognition on the demand information based on the guidance prompt information to obtain candidate execution instructions. When the language model recognizes the candidate execution instructions, the language model will return the candidate execution instructions to the terminal via an HTTP response message.

[0073] In another embodiment, when the language model is deployed in the terminal, the terminal can generate model call request information in a signal format for transmission between various modules inside the terminal based on the demand information and preset guidance prompt information, and then send the model call request information to the calling interface of the language model deployed inside the terminal through the artificial intelligence interaction module inside the terminal, so that the model call request information is transmitted to the language model through the calling interface. When the language model receives the model call request information through the calling interface, the language model will first parse the model call request information, extract the demand information and guidance prompt information from the model call request information, and then perform instruction recognition on the demand information according to the guidance prompt information to obtain candidate execution instructions. When the language model recognizes the candidate execution instructions, the language model will return the candidate execution instructions to the artificial intelligence interaction module of the terminal through the calling interface.

[0074] In this way, through the above method, model call request information is generated according to the demand information and the guidance prompt information, and the language model is called through the model call request information to perform instruction recognition, thereby improving the standardization of the instruction recognition process.

[0075] In one embodiment, the guidance prompt information (Prompt) is a text fragment or prompt information provided to start the language model. The guidance prompt information can be a word, a phrase, a sentence, a paragraph or a whole article, which is not specifically limited here. After the language model receives the guidance prompt information, the language model can generate a complete text based on the guidance prompt information, and the language model will make the generated text conform to the theme, tone and style of the guidance prompt information as much as possible. Therefore, calling the language model to perform instruction recognition on the demand information according to the preset guidance prompt information can enable the language model to more accurately recognize the instruction of the demand information according to the content in the guidance prompt information, thereby improving the confidence of the identified candidate execution instructions, and then obtaining candidate execution instructions with a higher degree of match with the demand information.

[0076] In one embodiment, the guidance prompt information can be composed of multiple instruction prompt information, and each instruction prompt information includes two parts: a question prompt information part and an answer prompt information part. Among them, for each instruction prompt information, the question prompt information part describes in detail the usage scenario of the current instruction and the corresponding question description, which can be used to guide the language model to semantically recognize the question content in the usage scenario of the current instruction, while the answer prompt information part describes in detail the instruction message structure of the current instruction (including instruction name and instruction parameters, etc.) and the corresponding instruction interpretation, which can be used to guide the language model to output the corresponding answer content in the usage scenario of the current instruction.

[0077] As shown in Figure 4, Figure 4 exemplarily shows the structural content of the guidance prompt information. In Figure 4, the guidance prompt information includes a first instruction prompt information 410 and a second instruction prompt information 420, wherein the first instruction prompt information 410 includes a first question prompt information part 411 and a first answer prompt information part 412 that match each other, and the second instruction prompt information 420 includes a second question prompt information part 421 and a second answer prompt information part 422 that match each other. In the first question prompt information part 411 and the second question prompt information part 421, both include an action prompt description of the question and the corresponding question content. For example, in the first question prompt information part 411, the action prompt description of the question is "action": "Ask a question", and the corresponding question content is "content": "I need to open the weather forecast webpage and search what the weather is like today? What should I do?" The first answer prompt information part 412 and the second answer prompt information part 422 both include an answer action prompt description and a corresponding answer content. For example, in the first answer prompt information part 412, the answer action prompt description is "action": "Answer", and the corresponding answer content is "content": "It is achieved by using the weather forecast plug-in. The instruction message structure is: {"event": " / weathernew", "parameter": "What's the weather like today?"}", where event refers to the name of the plug-in and parameter refers to the script parameter.

[0078] By calling the language model to learn the question prompt information part and the answer prompt information part in each guidance prompt information, the accuracy of the language model in identifying instructions for the target object's demand information can be improved, so that the identified candidate execution instructions can more accurately meet the usage scenario when the target object issues the demand information.

[0079] In one embodiment, the guidance prompt information may include prompt information of multiple different types of instructions, wherein each instruction may correspond to a corresponding application scenario and usage method. For example, as shown in Table 1 below, Table 1 exemplifies the applications corresponding to multiple instructions and their classification examples.

[0080] Table 1

[0081] According to the content in Table 1, the guidance prompt information provided by this embodiment includes prompt information of various types of instructions, such as search instructions, message instructions, and application instructions, and also includes the application scenarios and usage methods corresponding to these instructions, such as searching through a retrieval engine, sending messages or voice messages of instant messaging software, and viewing news through a news engine. Therefore, when the language model performs instruction recognition on the demand information based on the guidance prompt information, it can perform more accurate instruction recognition on the demand information of the target object, thereby effectively improving the confidence of the identified candidate execution instructions and obtaining candidate execution instructions that have a higher degree of match with the demand information.

[0082] In one embodiment, the language model used to identify instructions for demand information based on guidance prompt information is a trained natural language processing model based on a neural network. Therefore, when the language model is called to identify instructions for demand information based on guidance prompt information, the execution instructions corresponding to the demand information can be effectively identified. The process of training the language model may include the following four steps: (1) preparing training data; (2) establishing a dictionary; (3) constructing a model; and (4) training the model.

[0083] Among them, in the training data preparation step, a public sample data set can be downloaded in advance from the Internet, or a sample data set can be generated through a natural language generation model. These sample data sets can include various types of natural language materials such as web page content, news content or novel content. Then, these sample data sets and preset guidance prompt information are constructed into training sample data, and the training sample data is preprocessed by cleaning, word segmentation, stop word removal, etc. to obtain a training sample set for training the language model.

[0084] In the dictionary building step, the words appearing in all the text information of the training sample set can be counted and counted to build a dictionary, and then a corresponding unique number is generated for each word in the dictionary so that the dictionary can be used to train the language model in subsequent steps.

[0085] In the model building step, a neural network can be used to build a language model. For example, a recurrent neural network can be used to build a language model. Among them, the recurrent neural network has a memory ability and can remember the previous input, thereby affecting the subsequent output. Among them, when building the language model, the parameters of the language model can be set, for example, the number of hidden layers and the learning rate of the language model can be set. In one embodiment, the number of hidden layers of the language model can be set to 12, that is, the language model can include 12 hidden layers. In addition, the learning rate of the language model can be set to 50%, which is not specifically limited here.

[0086] In the model training step, the training sample set processed in the previous step is input into the language model for training. During the training process, the parameters of the language model are continuously adjusted to minimize the error between the language model's prediction results and the actual results (i.e., labels). When the error between the language model's prediction results and the actual results is less than a preset error threshold, or the number of iterative training of the language model reaches a preset threshold, the language model training can be considered complete. Once the language model training is completed, the language model can be used to identify the target object's demand information.

[0087] In one embodiment, during the training of the language model, a processing method such as dropout or batch normalization (BN) can be used to improve the training effect of the language model. In the process of training the language model, random dropout can inactivate some nodes in the language model, which not only simplifies the structure of the language model during the training process, but also prevents the language model from overweighting a certain node during training, thereby reducing the overfitting of the language model. In addition, during the training of the language model, by using batch normalization, the intermediate output of the language model can be adjusted using the mean and variance of the small batch during the training process, so that the outputs between each network layer in the language model can conform to a Gaussian distribution with the same mean and variance, thereby making the data more stable. Regardless of how the parameters of the hidden layer in the language model change, the mean and variance of the output data of the previous layer of the network can be determined to be known and fixed, thereby effectively solving the problems of slow training and small learning rate caused by the continuous change of data distribution.

[0088] In one embodiment, after calling the language model to perform instruction recognition on the demand information according to the guidance prompt information to obtain a candidate execution instruction, the candidate execution instruction is not necessarily a legally executable execution instruction. If the candidate execution instruction is not a legally executable execution instruction, it will be impossible to control the terminal to perform the operation task corresponding to the demand information according to the candidate execution instruction, which will lead to problems such as low interaction efficiency between the target object and the terminal and poor user experience.

[0089] In order to improve the interaction efficiency between the target object and the terminal and improve the user experience, after obtaining the candidate execution instruction, the candidate execution instruction can be firstly subjected to an executable verification to obtain an executable verification result. Then, whether the candidate execution instruction passes the executable verification is determined based on the executable verification result. If the executable verification result is that the candidate execution instruction fails to pass the executable verification, it means that the candidate execution instruction obtained by the language model recognition in this round of target operation is not accurate and is not a legally executable execution instruction. At this time, the next round of target operation can be started, that is, the language model is called to re-recognize the instruction of the requirement information based on the candidate execution instruction in the previous target operation and the guidance prompt information to obtain the candidate execution instruction in this round of target operation, and then the candidate execution instruction is re-performed with an executable verification to obtain an executable verification result. Then, whether the candidate execution instruction passes the executable verification is determined based on the executable verification result. This cycle is repeated until the obtained candidate execution instruction passes the executable verification.

[0090] Since the candidate execution instructions that pass the executable check are obtained by calling the language model multiple times to identify the instruction of the requirement information according to the guidance prompt information and the candidate execution instructions in the previous round of target operations, the confidence of the candidate execution instructions can be further improved. Moreover, since the process of calling the language model to re-identify the instruction of the requirement information is determined based on the specific situation of the executable check result, the process of calling the language model to re-identify the instruction of the requirement information is completed automatically, and does not require the target object to perform any additional operations, nor does it require the target object to interact with the terminal repeatedly. Therefore, the interaction efficiency between the target object and the terminal can be improved, so as to achieve the purpose of simplifying the use of the target object, thereby improving the use experience of the target object.

[0091] In one embodiment, when the language model is called to re-identify the instruction of the requirement information based on the candidate execution instructions and the guidance prompt information in the previous target operation, the model call request information can be regenerated based on the identified candidate execution instructions, the requirement information of the target object and the preset guidance prompt information, and then the regenerated model call request information is sent to the calling interface of the language model, and the language model is called to re-identify the instruction of the requirement information based on the candidate execution instructions and the guidance prompt information.

[0092] When the language model is called multiple times to re-identify the instruction for the requirement information, a model call request message can be regenerated based on the multiple candidate execution instructions previously identified, the requirement information, and the guidance prompt information, and then the regenerated model call request message can be sent to the language model call interface, thereby calling the language model to re-identify the instruction for the requirement information based on the multiple candidate execution instructions previously identified and the guidance prompt information. In this way, the confidence of the newly identified candidate execution instructions can be improved by using the multiple candidate execution instructions previously identified. For example, assuming that for requirement information A, the language model is called to perform instruction recognition on requirement information A three times and obtains three prior candidate execution instructions, then when the language model is called to perform instruction recognition on requirement information A for the fourth time, the three candidate execution instructions obtained by the first three identifications, requirement information A, and the preset guidance prompt information will be regenerated into a model call request message, and then the regenerated model call request message will be sent to the language model call interface, and the language model will be called to re-identify the instruction for requirement information A based on the three candidate execution instructions obtained by the first three identifications and the guidance prompt information.

[0093] In one embodiment, when performing an executable check on a candidate execution instruction to obtain an executable check result, the instruction function function in the candidate execution instruction can be first replaced with a target empty function, the function result of the target empty function can be always true, thereby obtaining the execution instruction to be checked, and then calling the instruction execution engine to simulate the execution of the execution instruction to be checked to obtain a simulated execution result, and then obtaining the executable check result based on the simulated execution result, that is, determining whether the candidate execution instruction passes the feasibility check.

[0094] Among them, when performing an executable check on a candidate execution instruction, since the operation corresponding to the instruction function function in the candidate execution instruction may be relatively complex, if the instruction execution engine is directly called to simulate the execution of the candidate execution instruction, there may be a problem of long execution time, which will affect the efficiency of the executable check on the candidate execution instruction. In order to save the time spent on the executable check on the candidate execution instruction, thereby improving the efficiency of the executable check on the candidate execution instruction, the instruction function function in the candidate execution instruction can be replaced with an empty function whose function result is always true to obtain the execution instruction to be checked. In this way, when the instruction execution engine is called to simulate the execution of the execution instruction to be checked, the executableness of the execution instruction to be checked can be verified more quickly, thereby improving the efficiency of the executable check on the candidate execution instruction.

[0095] Referring to FIG5 , FIG5 exemplifies the process of replacing the instruction function function in the candidate execution instruction with an empty function whose function result is always true. In FIG5 , assuming that the candidate execution instruction 510 includes an instruction function function 511, the candidate execution instruction 510 can be parsed first to determine the instruction function function 511 in the candidate execution instruction 510, and then an empty function 521 whose function result is always true is obtained from the preset function library 520. Then, the instruction function function 511 in the candidate execution instruction 510 is replaced with the empty function 521, and the execution instruction to be verified 530 including the empty function 521 whose function result is always true is obtained.

[0096] In one embodiment, before performing an executable check on a candidate execution instruction, a format check can be performed on the candidate execution instruction first. If the format of the candidate execution instruction is consistent with the instruction message structure in the boot prompt information, it means that the candidate execution instruction has passed the format check, and the instruction execution engine can simulate the execution of the candidate execution instruction, so the executable check can be performed; if the format of the candidate execution instruction is inconsistent with the instruction message structure in the boot prompt information, it means that the candidate execution instruction has not passed the format check, and the instruction execution engine cannot recognize the candidate execution instruction, so it cannot simulate the execution of the candidate execution instruction, so it cannot perform an executable check. In other words, for candidate execution instructions that have not passed the format check, there is no need to perform an executable check on them, and they can be directly considered to have not passed the executable check, thereby improving the efficiency of performing executable checks on candidate execution instructions. By first performing a format check on the candidate execution instruction, the candidate execution instruction that has not passed the format check can be directly confirmed as a candidate execution instruction that has not passed the executable check, thereby saving the time spent on performing executable checks on candidate execution instructions that have not passed the format check, and thus effectively improving the efficiency of performing executable checks on candidate execution instructions.

[0097] In one embodiment, when obtaining an executable verification result based on a simulation execution result, if the simulation execution result is that the execution instruction to be verified is successfully executed, it means that the empty function with a function result of always being true in the execution instruction to be verified can be accurately identified and executed. Then, if the empty function in the execution instruction to be verified is replaced with the instruction function function, so that the execution instruction to be verified is changed back to the original candidate execution instruction, the instruction function function in the candidate execution instruction can also be accurately identified and executed. Therefore, an executable verification result indicating that the candidate execution instruction passes the executable verification can be obtained. If the simulation execution result is that the execution instruction to be verified is not successfully executed, it means that the empty function with a function result of always being true in the execution instruction to be verified cannot be accurately identified and cannot be executed. Then, if the empty function in the execution instruction to be verified is replaced with the instruction function function, so that the execution instruction to be verified is changed back to the original candidate execution instruction, the instruction function function in the candidate execution instruction cannot be accurately identified and cannot be executed. Therefore, an executable verification result indicating that the candidate execution instruction fails the executable verification can be obtained.

[0098] In this way, through the above method, whether the candidate execution instruction passes the feasibility check is determined based on the simulated execution result obtained by executing the execution instruction to be checked, which can ensure the accuracy and reliability of the determined feasibility check result and ensure that the feasibility check result is determined efficiently.

[0099] In one embodiment, if the executable verification result obtained by the first executable verification of the candidate execution instruction is that the candidate execution instruction has passed the executable verification, it means that the candidate execution instruction is an execution instruction that matches the requirement information of the target object. There is no need to call the language model to re-identify the instruction of the requirement information. The terminal can be directly controlled to execute the operation task corresponding to the requirement information according to the candidate execution instruction.

[0100] Step 330: The candidate execution instruction that has passed the executable check is used as the target execution instruction, and the terminal is controlled to execute the operation task corresponding to the demand information according to the target execution instruction.

[0101] In one embodiment, upon obtaining a candidate execution instruction that has passed the executable verification, the candidate execution instruction that has passed the executable verification can be first used as the target execution instruction, and then the terminal can be controlled to perform the operation task corresponding to the requirement information according to the target execution instruction, thereby meeting the usage needs of the target object. Since the process from obtaining the candidate execution instruction that has passed the executable verification to controlling the terminal to perform the operation task corresponding to the requirement information according to the target execution instruction is all performed automatically by the terminal, no additional operations are required by the target object, nor does the target object need to perform multiple repeated interactions. Therefore, the interaction efficiency between the target object and the terminal can be effectively improved, the operation of the target object can be simplified, and the user experience of the target object can be effectively improved.

[0102] In one embodiment, when controlling a terminal to execute an operation task corresponding to requirement information according to a target execution instruction, the corresponding instruction execution code can be first obtained according to the target execution instruction, and then the instruction execution code can be executed in a virtual environment to control the terminal to execute the operation task corresponding to the requirement information. Each target execution instruction has a corresponding instruction execution code, which records each step of executing the target execution instruction. Therefore, by first obtaining the instruction execution code corresponding to the target execution instruction and then executing the instruction execution code in the virtual environment, the target object can be simulated to control the terminal to execute the operation task corresponding to the requirement information.

[0103] In this way, by obtaining the instruction execution code corresponding to the target execution instruction in the above manner, and by running the instruction execution code to control the terminal to perform the operation task corresponding to the demand information, the operation task can be ensured to be executed accurately and reliably.

[0104] In one embodiment, in order to achieve simultaneous concurrency of different execution instructions, after obtaining multiple target execution instructions and calling the instruction execution code corresponding to each target execution instruction, multiple virtual environments (for example, a virtual image of an operating system with a desktop) can be started at the same time, and the instruction execution codes of different target execution instructions can be executed simultaneously in different virtual environments. The parallel execution of different operation tasks is achieved by running multiple operating system images at the same time.

[0105] In one embodiment, when running the instruction execution code in the virtual environment to control the terminal to perform the operation task corresponding to the demand information, the instruction execution process can be run in the virtual environment first, and then the instruction execution process is called to run the instruction execution code to control the terminal to perform the operation task corresponding to the demand information.

[0106] When the instruction execution code is executed in a virtual environment, a corresponding instruction execution process can be started in the operating system running in each virtual environment. This instruction execution process can be used to automatically control the terminal. For example, if the terminal is a computer connected to a mouse and keyboard, the instruction execution process can be used to automatically control the computer via the mouse and keyboard. The operation task corresponding to the demand information for controlling the execution of the terminal can be various operation tasks performed by the control terminal, such as opening an application, entering text, clicking a button, dragging a window, and other different types of operation tasks.

[0107] In one embodiment, the instruction execution code called by the target execution instruction can be used to implement different operation tasks, such as opening an application, entering text, clicking a button, or dragging a window. Referring to Table 2, Table 2 exemplifies various instruction execution codes that can implement different functions.

[0108] Table 2

[0109] As shown in Table 2, the instruction execution codes in this embodiment include various functions, such as those for acquiring the coordinates of the operating cursor, moving the operating cursor, simulating click operations, and inputting characters. Therefore, when the corresponding instruction execution code is acquired based on the target execution instruction, the target object's control over the terminal can be effectively simulated to execute the operation task corresponding to the requested information. In one embodiment, the instruction execution code in this embodiment can acquire data and simulate the corresponding operation action by calling the operating system's application programming interface.

[0110] In one embodiment, when the terminal includes a display module for displaying an operation cursor, when controlling the terminal to perform an operation task corresponding to the requirement information, the initial coordinates of the operation cursor can be obtained first, and then the target coordinates when performing the operation task corresponding to the requirement information can be determined. Then, the operation cursor is controlled to move from the initial coordinates to the target coordinates, and the terminal is controlled to perform the operation task corresponding to the requirement information at the target coordinates.

[0111] Among them, when obtaining the initial coordinates of the operation cursor, the application programming interface of the operating system can be called first to obtain the screen resolution of the display module, and then the initial coordinates of the operation cursor can be obtained according to the screen resolution. For example, assuming that the screen resolution of the display module is 1920*1080, and the operation cursor is at the center of the screen of the display module, then the initial coordinates of the operation cursor in the screen of the display module can be determined to be (960, 540). Since when executing the operation task corresponding to the demand information, it is automatically realized by the terminal simulating the operation mode of the target object, the entire process does not require the participation of the target object, nor does it require the target object to perform any additional operations. Therefore, it can effectively simplify the target object's use of the terminal, thereby improving the target object's user experience, and by controlling the movement of the operation cursor, the process of the terminal executing the operation task corresponding to the demand information can be visualized, so that the target object can know the execution process of the operation task. It should be noted that the display module can be a display screen integrated in the terminal, or it can be an external display device of the terminal, which is not specifically limited here. For example, when the terminal is a smart phone, the display module can be the display screen of the smart phone; when the terminal is a computer, the display module can be the external display of the computer.

[0112] The following uses an example to specifically illustrate the process of controlling the terminal to execute the operation task corresponding to the demand information.

[0113] For example, assuming that the target object's demand information is to query today's weather conditions, then the operation task corresponding to the demand information can be to query today's weather conditions through a browser. In this case, the instruction execution process running in the virtual environment can run the getSize instruction execution code to obtain the terminal's screen width and screen height through the getSize instruction execution code. At this time, the getSize instruction execution code will return a tuple data, which can be (1920, 1080), indicating the terminal's screen resolution; then, the instruction execution process can run the getLocation instruction execution code to obtain the current coordinates (a, b) of the operation cursor on the terminal screen and the target coordinates (x, y) of the browser application's icon on the terminal screen through the getLocation instruction execution code; then, the instruction execution process can run the moveTo instruction execution code to move the operation cursor from the current coordinates (a, b) to the target coordinates (x, y) at a certain speed, and run the click instruction execution code to simulate the target object's click operation on the browser application's icon (for example, simulate the target The target object performs a double-click of the left mouse button on the browser application icon to open the browser application. After the browser application is opened, the instruction execution process can execute the getSize instruction to obtain the coordinates (h, i) of the search input box on the browser page. Then, the instruction execution process can execute the moveTo instruction again to move the operating cursor from the target coordinates (x, y) to the coordinates (h, i) of the search input box at a certain speed. The instruction execution process can also execute the click instruction again to simulate the target object's click operation on the search input box (for example, simulating the target object's left-click operation on the search input box), so that the operating cursor is positioned in the search input box. Next, the instruction execution process can execute the typeWrite instruction to simulate the target object's input operation in the search input box, entering the text message "What's the weather like today?" in the search input box, thereby obtaining today's weather conditions through the browser query. Since the terminal simulates the target object's operation mode when executing the operation task corresponding to the demand information, the entire process does not require the target object's participation or any complex additional operations. This can effectively simplify the target object's use of the terminal, thereby improving the target object's user experience.

[0114] In this embodiment, through the instruction execution method based on the language model including the previous steps 310 to 330, when the demand information of the target object is received, at least one round of target operation is executed until the candidate execution instruction that passes the feasibility check is obtained; in the first round of target operation, the language model is first called to perform instruction recognition on the demand information according to the preset guidance prompt information to obtain the candidate execution instruction of the first round of target operation. Since the candidate execution instruction is obtained by the language model performing instruction recognition on the demand information according to the guidance prompt information, the confidence of the candidate execution instruction can be improved, which is conducive to obtaining the candidate execution instruction with a higher degree of matching with the demand information; then, the candidate execution instruction of the first round of target operation is selected. The executable check is performed on the candidate execution instruction to determine whether the candidate execution instruction passes the feasibility check. If the candidate execution instruction does not pass the executable check, it means that the candidate execution instruction identified by the current language model is not accurate. At this time, continue to the next round of target operation. In the next round of target operation, call the language model to re-identify the instruction of the requirement information according to the candidate execution instruction in the previous target operation and the guidance prompt information to obtain the candidate execution instruction in this round of target operation, and then re-perform the executable check on the candidate execution instruction to determine whether the candidate execution instruction passes the feasibility check. It is iteratively performed for several rounds of target operation until a candidate execution instruction that passes the executable check is obtained. Moreover, after obtaining the candidate execution instruction that passes the executable check, the candidate execution instruction is used as the target execution instruction, and the terminal is controlled to execute the operation task corresponding to the demand information according to the target execution instruction. Since the candidate execution instruction that passes the executable check is obtained by multiple calls to the language model to perform instruction recognition on the demand information based on the guidance prompt information and the previous candidate execution instructions, the confidence of the candidate execution instruction can be guaranteed. Moreover, since this process is completed automatically, the target object does not need to perform any additional operations, nor does it require the target object to perform multiple repeated interactions. Therefore, the interaction efficiency between the target object and the terminal can be improved, so as to achieve the purpose of simplifying the use of the target object, thereby improving the use experience of the target object.

[0115] The specific process of the instruction execution method based on the language model is described below with a specific example.

[0116] Referring to Figure 6, Figure 6 is a schematic diagram of the interface for a user to interact with a terminal through an artificial intelligence interaction module. When a user inputs a voice message 610 into the terminal through the interactive interface of the artificial intelligence interaction module, the terminal will first recognize the voice message 610. To avoid displaying blank content while the user is waiting, the terminal can display a first loading wait indicator 620 in the interactive interface to prompt the user that the voice message 610 is being recognized. During the process of recognizing the voice message 610, the terminal can first call a pre-trained speech recognition model to perform speech recognition on the voice message 610 to obtain text information. Then, the text information is processed by adding punctuation, correcting spelling errors, or correcting word order to obtain user demand information 630. At this point, the terminal has completed the recognition process of the voice message 610. The pre-trained speech recognition model is used to convert the audio feature vector of the voice information into text information. The speech recognition model can include a hidden Markov model, a Gaussian mixture model, a deep neural network model, an n-gram language model, or other statistical models commonly used in the art.

[0117] After the terminal completes recognition of the voice message 610 and obtains the user's demand information 630, it can display the recognized demand information 630 on the interactive interface. For example, assuming that the user sends a voice message 610 saying "What's the weather like today?", after the terminal accurately recognizes the voice message 610, it can display the demand information 630 with the content "What's the weather like today?" on the interactive interface. After the terminal completes recognition of the voice message 610, it calls the language model to perform command recognition on the content of the demand information 630 based on the guidance prompt information. At this time, in order to avoid displaying blank content while the user is waiting, the terminal can display a second loading waiting indicator 640 on the interactive interface to prompt the user that the command recognition of the content of the demand information 630 is being performed.

[0118] In the process of calling the language model to perform instruction recognition on the content of the requirement information 630 according to the guidance prompt information, the terminal can first generate model call request information based on the guidance prompt information and the requirement information 630, and then send the model call request information to the calling interface of the language model, and call the language model to perform instruction recognition on the requirement information 630 according to the guidance prompt information.

[0119] Among them, the content structure of the guidance prompt information can be shown in Figure 7, which exemplarily shows an example of the guidance prompt information. In Figure 7, the guidance prompt information may include a type item 710, a question item 720 and an answer item 730, wherein the type item 710 records the information type of the guidance prompt information, for example, as shown in Figure 7, the information type of the guidance prompt information is "content"; the question item 720 records the question content of the guidance prompt information, for example, as shown in Figure 7, the question content of the guidance prompt information is "I need to open the browser to search what the weather is like today? What should I do?"; the answer item 730 records the answer content of the guidance prompt information, for example, as shown in Figure 7, the answer content of the guidance prompt information includes an answer description 731, an instruction message structure description 732 and an explanation description 733, wherein the answer description 731 is "achieved by using a browser plug-in", and the instruction message structure description 732 is "The instruction message structure is: {"event": " / weathernew", "parameter": "What is the weather like today? ”}”, explanation description 733 is “event refers to the plug-in name: weathernew, which means opening the browser and entering the URL www.xxxxxx.com; parameter refers to the script parameter: “What’s the weather like today?”, which is the content searched in the browser”. When generating the guidance prompt information, you can first obtain the information type, question content and answer content corresponding to the question content of the guidance prompt information, and then fill the information type of the guidance prompt information into the type item 710, fill the question content of the guidance prompt information into the question item 720, and fill the answer content corresponding to the question content into the answer item 730. At this time, the guidance prompt information can be obtained.

[0120] After the terminal calls the language model to identify the instruction of the demand information 630 according to the guidance prompt information to obtain the target execution instruction, the terminal will further control the terminal to perform the corresponding operation task according to the target execution instruction. In the process of controlling the terminal to perform the corresponding operation task, the third loading waiting indication information 650 can be displayed in the interactive interface to prompt the user that the operation task desired by the user is being performed. Moreover, after the terminal completes the corresponding operation task, the execution completion indication information 660 can be displayed in the interactive interface to inform the user that the operation task corresponding to the voice information 610 has been completed.

[0121] In the process of calling the language model to identify the instruction of the requirement information 630 according to the guidance prompt information, the terminal will perform an executable check on the candidate execution instruction currently identified. If the candidate execution instruction currently identified does not pass the executable check, the terminal will call the language model to re-identify the instruction of the requirement information 630 according to the candidate execution instruction and the guidance prompt information to obtain a new candidate execution instruction, and then re-check the executableness of the new candidate execution instruction. The above process is repeated until a candidate execution instruction that passes the feasibility check is obtained. At this time, the terminal will use the candidate execution instruction that passes the executable check as the target execution instruction.

[0122] Among them, when the terminal performs an executable verification on the candidate execution instruction currently identified, it can first replace the instruction function in the candidate execution instruction with an empty function whose function result is a permanent true function to obtain the execution instruction to be verified, and then call the instruction execution engine to simulate the execution of the execution instruction to be verified to obtain the simulated execution result, and then obtain the executable verification result based on the simulated execution result. In addition, when the terminal controls the terminal to perform the corresponding operation task according to the target execution instruction, it can first obtain the corresponding instruction execution code according to the target execution instruction, then run the instruction execution process in the virtual environment, and then call the instruction execution process to run the instruction execution code to control the terminal to perform the operation task corresponding to the demand information. In addition, after the terminal completes the corresponding operation task, it will display the interactive interface shown in Figure 8. In the interactive interface shown in Figure 8, the operation result after the corresponding operation task is completed can be displayed through a pop-up window 810. Through the operation result displayed in the pop-up window 810, the user can obtain the desired content. In addition, in another embodiment, the operation result after the corresponding operation task is completed can also be displayed by page jump, which is not specifically limited here.

[0123] In one embodiment, when controlling a terminal to execute an operation task corresponding to demand information according to a target execution instruction, the terminal may also create an execution record table including an execution instruction name table item, an instruction execution code table item, an execution status table item, and an execution result table item, wherein the execution instruction name table item records the target execution instruction, and the instruction execution code table item records the instruction execution code corresponding to the target execution instruction; after the control terminal completes the execution of the operation task corresponding to the demand information, the terminal may update the content of the execution status table item in the execution record table to completed, and write the corresponding execution result into the execution result table item. By creating an execution record table to record the execution status of the target execution instruction, real-time control of the execution status of the target execution instruction can be achieved, which is conducive to the terminal's automated execution of the operation task corresponding to the demand information.

[0124] In one embodiment, it is assumed that the instruction message structure of the identified target execution instruction is:

[0125] {"event":" / weathernew","parameter":"What's the weather like today?"}

[0126] Among them, weathernew is plugin_name, that is, the name of the plug-in. Different plug-ins will have different instruction execution codes in the steps table item.

[0127] When the terminal executes the operation task corresponding to the target execution instruction according to the required information, the terminal can create an execution record table in the instruction execution database. This execution record table includes multiple entries, such as the execution instruction name table, the instruction execution code table, the execution status table, and the execution result table. Through these entries, the execution record table can record information such as the process relationship between multiple target execution instructions and the execution time. Table 3 shows an exemplary structure of the execution record table.

[0128] Table 3

[0129] In Table 3, the first row contains the table item name, and the second row contains the table item content. The plugin_name table item is the execution instruction name table item, the steps table item is the instruction execution code table item, the status table item is the execution status table item, and the result table item is the execution result table item. In the execution record table, the instruction execution code in the steps table item can be retrieved and loaded from the plugin execution code database based on the plugin name (i.e., the execution instruction name) recorded in the plugin_name table item. In this way, when controlling the terminal to execute the operation task corresponding to the demand information according to the target execution instruction, the instruction execution engine can execute the instruction execution code step by step according to the specific content of the steps table item, thereby controlling the terminal to execute the operation task corresponding to the demand information.

[0130] In one embodiment, all target execution instructions obtained based on language model identification can have a unified standard instruction message structure. That is, when the language model outputs the identified candidate execution instructions, it can output the candidate execution instructions in the form of a unified standard instruction message structure. In this way, the standardization of the instruction message structure can achieve a unified standard for all operation tasks, so that the operating environment for executing the target execution instructions can be unified, so that the multi-operating system image mode (i.e., multiple virtual environments) can support the concurrency of multiple operation tasks, thereby effectively reducing the cost of executing multiple operation tasks.

[0131] In one embodiment, during the system operation, the terminal will also generate a corresponding log file and record the corresponding log information in detail in the log file, so as to facilitate the management and control of the terminal operation status based on the log information in the log file, and realize daily troubleshooting and status recording of the terminal operation.

[0132] In one embodiment, the log information in the log file can have different log types. For example, the log information can have error type, warning type, information type and debugging type, and different types of log information can have different purposes. In addition, the log information in the log file can also have different log contents. Wherein, according to the difference in log contents, the log information in the log file can be classified into configuration log, warning log and background log etc. With reference to shown in Table 4 and Table 5, Table 4 exemplarily provides the description of the different types of log information, and Table 5 exemplarily provides the description of the different contents of log information.

[0133] Table 4

[0134] According to the content in Table 4, when error-type log information is recorded in the log file, the administrator needs to pay special attention to the log information and should promptly resolve the fault problems in the log information to ensure the normal operation of the subsequent system. When alarm-type log information is recorded in the log file, the administrator needs to pay more attention to the log information and should promptly resolve the fault problems in the log information to avoid abnormal risks during system operation. When information-type log information is recorded in the log file, the administrator can pay certain attention to the log information in daily operation and maintenance work, and can make timely corrections when abnormalities occur in the key information of the system to maintain the normal operation of the system. When debugging-type log information is recorded in the log file, the administrator can use the log information as reference information when debugging the system, which can play a role in assisting debugging and improve the administrator's debugging efficiency.

[0135] Table 5

[0136] As shown in Table 5, when configuration logs are recorded in the log file, administrators can use them to understand various system operation conditions, such as user additions, deletions, and configuration modifications. When control logs are recorded in the log file, administrators can use them to understand control actions taken regarding system operation. When alarm logs are recorded in the log file, administrators can use them to understand alarm information that occurs during system operation, allowing them to promptly correct alarms and maintain normal system operation. When background logs are recorded in the log file, administrators can use them to understand various system behaviors throughout the entire system's background operation.

[0137] The following is a detailed description of the language model-based instruction execution method provided in the embodiments of the present application using specific examples.

[0138] 9 , which is a flow chart of an example of a method for executing instructions based on a language model, includes but is not limited to steps 910 to 950 .

[0139] Step 910: The terminal receives a speech signal from a target object and converts the speech signal into demand information that can be recognized by a language model.

[0140] This step is a step in which the terminal receives demand information. In this step, the terminal can receive the voice signal of the target object through various voice receiving software or hardware.

[0141] In this step, when the terminal converts the voice signal into demand information that can be recognized by the language model, it can perform voice signal preprocessing, voice feature extraction, voice recognition, and text post-processing on the voice signal of the target object in sequence to obtain demand information that can be recognized by the language model, so that the language model can be called in subsequent steps to perform instruction recognition on the demand information.

[0142] In one embodiment, the process of the terminal receiving the speech signal of the target object and converting the speech signal into demand information that can be recognized by the language model may include the following steps 911 to 917.

[0143] Step 911: Collect voice signals.

[0144] In this step, collecting a voice signal refers to receiving voice information input by a target subject. The terminal may collect the target subject's voice signal through an internal voice receiving module (e.g., a microphone provided within the terminal), or through another external voice receiving module (e.g., an external recording device, another mobile phone, another tablet, or an external device such as industrial equipment), without specific limitations herein.

[0145] Step 912: Speech signal preprocessing.

[0146] In this step, speech signal preprocessing refers to performing at least one of noise removal and speech segmentation on speech information to obtain preprocessed speech information. After collecting the speech signal of the target object, the terminal can perform speech signal preprocessing on the speech signal of the target object. In one embodiment, the speech signal preprocessing performed by the terminal on the speech signal may include operations such as noise removal and speech segmentation. Among them, performing noise removal on speech information can filter out interference information such as environmental noise mixed in the speech information, thereby improving the purity of the speech information; performing speech segmentation on speech information can segment the long speech input by the target object into multiple short speech, thereby reducing the processing difficulty of the subsequent speech feature extraction model to extract speech features, thereby improving the accuracy and efficiency of speech feature extraction.

[0147] Step 913: Speech feature extraction.

[0148] In this step, speech feature extraction refers to calling a speech feature extraction model to extract speech features from the preprocessed speech information to obtain speech feature information. After preprocessing the speech signal to obtain preprocessed speech information, the terminal can perform speech feature extraction on the preprocessed speech information, converting the speech signal into speech feature information in the form of a digital signal (e.g., an audio feature vector). This facilitates the subsequent speech recognition model to more accurately recognize and process this speech feature information.

[0149] In one embodiment, speech features can be extracted from a speech signal by calling a speech feature extraction model. The speech feature extraction model can be a trained deep neural network model or a convolutional neural network model, or a mathematical model with speech feature extraction capabilities (e.g., a mathematical model that performs MFCC processing on a speech signal, a mathematical model that performs DWT processing on a speech signal, a mathematical model that performs PLP processing on a speech signal, etc.). The model can be appropriately selected based on the actual application and is not specifically limited here.

[0150] Step 914: Establish a speech recognition model.

[0151] In this step, establishing a speech recognition model refers to constructing a speech recognition model for speech recognition and training the speech recognition model using speech feature information. After extracting speech features from the speech signal and obtaining speech feature information in the form of a digital signal, the terminal can establish a corresponding speech recognition model based on the obtained speech feature information, thereby facilitating speech recognition using the speech recognition model.

[0152] In one embodiment, the speech recognition model may be an HMM model, a Gaussian mixture model, a deep neural network model, an n-gram language model or other statistical models commonly used in the art, and is not specifically limited here.

[0153] Step 915: Call the speech recognition model to perform speech recognition.

[0154] In this step, calling the speech recognition model for speech recognition refers to calling the trained speech recognition model to perform speech recognition on speech feature information to obtain text information. Once the terminal has established the speech recognition model, the terminal can call the speech recognition model to perform speech recognition on the speech feature information to obtain text information corresponding to the target subject's speech signal.

[0155] In one embodiment, when the speech recognition model includes an acoustic model and a language model, in the process of calling the speech recognition model to perform speech recognition on speech feature information, the acoustic model can be called first to convert the speech feature information into an intermediate recognition result (such as phonemes, phoneme strings or sub-words, etc.), and then the language model can be called to convert the intermediate recognition result into a text recognition result (such as words, word strings or symbol sequences, etc.), and then the text information corresponding to the speech signal of the target object is output to complete the speech recognition of the speech feature information.

[0156] Step 916: Text post-processing.

[0157] In this step, text post-processing refers to adding punctuation, correcting spelling, or correcting word order to the text information obtained through speech recognition to obtain the target subject's desired information. After the terminal uses the speech recognition model to perform speech recognition on the speech feature information to obtain the corresponding text information, the terminal can add punctuation, correct spelling, or correct word order to the text information to obtain the desired information corresponding to the target subject's speech signal.

[0158] Step 917: Output the target object's demand information.

[0159] In this step, after the terminal completes the text post-processing of the text information output by the speech recognition model, it can output the text content after the text post-processing. The text content at this time is the demand information of the target object.

[0160] Through the processing of steps 911 to 917, the speech signal of the target object can be converted into demand information that can be recognized by the language model, so that in subsequent steps the language model can be called to perform instruction recognition on the demand information to obtain the execution instruction corresponding to the speech signal of the target object.

[0161] Step 920: Construct targeted guidance prompt information, wherein the guidance prompt information includes key requirement description, instruction message structure and explanation.

[0162] In this step, the guidance prompt information is a text fragment or prompt information provided to start the language model. The guidance prompt information can be a word, a phrase, a sentence, a paragraph or an entire article, which is not specifically limited here. When the guidance prompt information is input into the language model for language recognition processing, the language model can generate a complete text based on the guidance prompt information, and the language model will make the generated text conform to the theme, tone and style of the guidance prompt information as much as possible. Therefore, a guidance prompt information including a key requirement description, an instruction message structure and an explanation is constructed. When the language model is called in a subsequent step to perform instruction recognition on the requirement information according to the guidance prompt information, the language model can perform instruction recognition on the requirement information more accurately according to the content in the guidance prompt information, thereby improving the confidence of the instruction message structure obtained by recognition, that is, an instruction message structure with a higher degree of matching with the requirement information can be obtained.

[0163] In one embodiment, the guidance prompt information can be composed of multiple instruction prompt information, and each instruction prompt information includes a question prompt information part and an answer prompt information part. For each instruction prompt information, the question prompt information part describes in detail the usage scenario of the current instruction and the corresponding question description, which can be used to guide the language model to perform semantic recognition of the question content in the usage scenario of the current instruction, while the answer prompt information part describes in detail the instruction message structure of the current instruction (including instruction name and instruction parameters, etc.) and the corresponding instruction interpretation, which can be used to guide the language model to output the corresponding answer content in the usage scenario of the current instruction.

[0164] Step 930: Call the pre-trained language model to perform instruction recognition on the requirement information according to the guidance prompt information to obtain an executable instruction message structure.

[0165] In this step, after obtaining the demand information and constructing the guidance prompt information, the pre-trained language model can be called to perform instruction recognition on the demand information based on the guidance prompt information to obtain an executable instruction message structure, so that in subsequent steps the terminal can automatically perform the corresponding operation task according to the executable instruction message structure, avoiding multiple repeated interactions with the target object, thereby improving the interaction efficiency between the target object and the terminal.

[0166] In one embodiment, the language model may be an LLM model or an NLU model, etc., and may be appropriately selected according to actual application conditions, and is not specifically limited here.

[0167] In one embodiment, the process of calling a pre-trained language model to perform instruction recognition on the demand information according to the guidance prompt information may include the following steps 931 to 935.

[0168] Step 931: Prepare training data.

[0169] In this step, training data preparation refers to obtaining a training sample set for training the language model. The terminal can download a publicly available sample dataset from the internet or generate a sample dataset using a natural language generation model. These sample datasets can include various types of natural language materials, such as webpage content, news content, or novel content. These sample datasets and guidance prompts are then combined to form training sample data. This training sample data is then preprocessed, including cleaning, word segmentation, and stop word removal, to obtain the training sample set used for training the language model.

[0170] Step 932: Create a dictionary.

[0171] In this step, building a dictionary refers to creating a dictionary for language model training based on the text information in the training sample set. The terminal can count the words that appear in all the text information in the training sample set to create a dictionary. Then, for each word in the dictionary, a unique number is generated, so that the dictionary can be used to train the language model in subsequent steps.

[0172] Step 933: Build a language model.

[0173] In this step, constructing a language model refers to constructing a language model for identifying instructions for the demand information of the target object. The terminal can use a neural network to construct a language model. For example, a recurrent neural network can be used to construct a language model. The recurrent neural network has a memory ability and can memorize the previous input, thereby affecting the subsequent output. When constructing the language model, parameters of the language model can be set. For example, parameters such as the number of hidden layers and the learning rate of the language model can be set. In one embodiment, the number of hidden layers of the language model can be set to 12, that is, the language model can include 12 hidden layers. In addition, the learning rate of the language model can be set to 50%, which is not specifically limited here.

[0174] Step 934: Train the language model.

[0175] In this step, training the language model refers to training the language model using a training sample set. The terminal can input the training sample set processed in the previous step into the language model for training. During the training process, the language model parameters are continuously adjusted to minimize the error between the language model's predicted results and the actual results (i.e., labels). When the error between the language model's predicted results and the actual results is less than a preset error threshold, or the number of iterative training of the language model reaches a preset threshold, the language model training can be considered complete.

[0176] Step 935: Call the language model to perform instruction recognition on the demand information according to the guidance prompt information.

[0177] In this step, once the language model is trained, the language model can be used to perform instruction recognition on the target object's demand information according to the guidance prompt information.

[0178] Through the processing of steps 931 to 935, a language model can be constructed and the training of the language model can be completed, so that the language model can perform instruction recognition on the target object's demand information based on the guidance prompt information, and obtain an executable instruction message structure, so that in subsequent steps the terminal can automatically perform corresponding operation tasks according to the executable instruction message structure.

[0179] Step 940: Read the instruction message structure, run the instruction execution code corresponding to the instruction message structure in the virtual environment, and control the terminal to execute the operation task corresponding to the demand information.

[0180] In this step, after calling the language model to perform instruction recognition on the requirement information according to the guidance prompt information to obtain an executable instruction message structure, the content in the instruction message structure can be read, and the instruction execution code corresponding to the instruction message structure can be obtained, and then the instruction execution code can be run in a virtual environment, so that the terminal can simulate the target object to control the terminal to execute the operation task corresponding to the requirement information.

[0181] In one embodiment, running the instruction execution code corresponding to the instruction message structure in a virtual environment to simulate the process of the target object controlling the terminal to execute the operation task corresponding to the demand information may include the following steps 941 to 945.

[0182] Step 941: Load the function plug-in.

[0183] In this step, loading the function plug-in refers to loading the instruction execution code corresponding to the instruction message structure into the created execution record table. Before running the instruction execution code corresponding to the instruction message structure, the terminal can first create an execution record table in the instruction execution database. The execution record table can be used to record information such as the process relationship and execution time of the instruction execution. Then, based on the instruction name (or plug-in name) in the instruction message structure, the terminal obtains the corresponding instruction execution code (i.e., the function plug-in) from the instruction execution code database and loads the instruction execution code into the execution record table so that the subsequent steps can read the instruction execution code from the execution record table and run the instruction execution code.

[0184] Step 942: Start the virtual environment.

[0185] In this step, launching a virtual environment refers to launching a virtual environment for running instruction execution code in the terminal. To enable the concurrent execution of different instructions, the terminal can simultaneously launch multiple virtual environments (e.g., virtual images of an operating system with a desktop), so that different instruction execution codes can be executed simultaneously in different virtual environments. This achieves the purpose of parallel execution of different operating tasks by running multiple operating system images simultaneously.

[0186] Step 943: Run the instruction execution code.

[0187] In this step, running the instruction execution code refers to running the instruction execution process in the already activated virtual environment and calling the instruction execution process to run the instruction execution code. When the terminal runs the instruction execution code in the virtual environment to simulate the target object controlling the terminal to perform the operation task corresponding to the demand information, the instruction execution process can be first run in the virtual environment, and then the instruction execution process can be called to run the instruction execution code to control the terminal to perform the operation task corresponding to the demand information. When running the instruction execution code in the virtual environment, a corresponding instruction execution process can be started in the operating system running in each virtual environment. This instruction execution process can be used to automatically simulate the target object performing control operations on the terminal. For example, assuming that the terminal is a computer connected to a mouse and keyboard, the instruction execution process can be used to automatically simulate the target object controlling the computer using the mouse and keyboard. The operation task corresponding to the demand information simulated by the target object controlling the terminal can be various operation tasks that the target object can control the terminal to perform, such as opening an application, entering text, clicking a button, dragging a window, and other different types of operation tasks.

[0188] Step 944: Update the execution status.

[0189] In this step, updating the execution status means that after the simulation target object control terminal completes the execution of the operation task corresponding to the demand information, the content of the execution status table item in the execution record table is updated to completed, and the execution result is written into the execution result table item. After running the instruction execution code to simulate the target object control terminal to complete the execution of the operation task corresponding to the demand information, the terminal can update the execution status of the instruction execution code in the execution record table to completed, and write the corresponding execution result into the execution result table item in the execution record table. By using the execution record table to record and update the execution status of the instruction execution code, real-time control of the execution status of the instruction execution code can be achieved, which is conducive to the terminal's automatic execution of the operation task corresponding to the demand information.

[0190] Step 945: Close the virtual environment.

[0191] In this step, closing the virtual environment means that after the simulated target object control terminal completes the execution of the operation task corresponding to the demand information, the terminal closes the virtual environment used to run the instruction execution code. When the simulated target object control terminal completes the execution of the operation task corresponding to the demand information, the virtual environment can be closed to avoid wasting resources.

[0192] Through the processing of steps 941 to 945, the terminal can simulate the target object to control the terminal to execute the operation task corresponding to the demand information. Since this process is completed automatically, the target object does not need to perform any additional operations, nor does it require the target object to perform multiple repeated interactions. Therefore, the interaction efficiency between the target object and the terminal can be improved, thereby achieving the purpose of simplifying the use of the target object, thereby improving the user experience of the target object.

[0193] Step 950: When the simulated target object control terminal completes the execution of the operation task corresponding to the demand information, the execution result is written into the database, and the log information of the entire system is recorded in the log file.

[0194] In this step, by writing the execution results of the operation tasks corresponding to the execution requirement information of the simulated target object control terminal into the database, and recording the log information of the entire system in the log file, the information of the entire processing process can be recorded, so that the operation status of the terminal can be managed and controlled according to the log information in the log file, and daily troubleshooting and status recording of the terminal operation can be realized.

[0195] In this embodiment, through the instruction execution method based on the language model including the previous steps 910 to 950, when the voice signal of the target object is received, the voice signal is first converted into demand information that can be recognized by the language model, and then targeted guidance prompt information is constructed, and then the pre-trained language model is called to perform instruction recognition on the demand information according to the guidance prompt information to obtain an executable instruction message structure. Since the executable instruction message structure is obtained by the language model performing instruction recognition on the demand information according to the guidance prompt information, the confidence of the recognized instruction message structure can be improved, so that an instruction message structure with a higher matching degree with the demand information can be obtained; after obtaining the executable instruction message structure After the structure is constructed, the instruction execution code corresponding to the instruction message structure is first run in the virtual environment to simulate the target object control terminal to execute the operation task corresponding to the demand information. After the target object control terminal completes the execution of the operation task corresponding to the demand information, the execution result is written into the database, and the log information of the entire system is recorded in the log file. By writing the execution result of the operation task corresponding to the demand information executed by the simulated target object control terminal into the database and recording the log information of the entire system in the log file, the information recording of the entire processing process can be realized, so that the operation status of the terminal can be controlled according to the log information in the log file, and daily troubleshooting and status recording of the terminal operation can be realized. In addition, since the instruction execution method is completed automatically by the terminal, it does not require the target object to perform any additional operations, nor does it require the target object to perform multiple repeated interactions. Therefore, the interaction efficiency between the target object and the terminal can be improved, and the purpose of simplifying the use of the target object can be achieved, thereby improving the user experience of the target object.

[0196] 10 , which is a flowchart of a method for executing an instruction based on a language model, is a specific example of a method for executing an instruction based on a language model. In FIG10 , the method for executing an instruction based on a language model may include, but is not limited to, steps 1001 to 1009 .

[0197] Step 1001: Receive demand information of a target object.

[0198] Step 1002: Generate model call request information based on demand information and preset guidance prompt information.

[0199] Step 1003: Send the model call request information to the call interface of the language model, call the language model to perform instruction recognition on the requirement information according to the guidance prompt information, and obtain candidate execution instructions.

[0200] Step 1004: Replace the instruction function in the candidate execution instruction with an empty function whose function result is always true to obtain the execution instruction to be verified, then call the instruction execution engine to simulate the execution of the execution instruction to be verified to obtain the simulation execution result, and then obtain the executable verification result based on the simulation execution result.

[0201] It should be noted that when performing an executable check on a candidate execution instruction, since the operations corresponding to the instruction function functions in the candidate execution instruction may be relatively complex, if the instruction execution engine is directly called to simulate the execution of the candidate execution instruction, there may be a problem of long execution time, which will affect the efficiency of the executable check on the candidate execution instruction. Therefore, in order to save the time spent on the executable check on the candidate execution instruction and thus improve the efficiency of the executable check on the candidate execution instruction, the instruction function functions in the candidate execution instruction can be replaced with an empty function whose function result is always true to obtain the execution instruction to be checked. In this way, when the instruction execution engine is called to simulate the execution of the execution instruction to be checked, the executableness of the execution instruction to be checked can be verified more quickly, thereby improving the efficiency of the executable check on the candidate execution instruction.

[0202] Step 1005: Determine whether the candidate execution instruction passes the executable check based on the executable check result. If the candidate execution instruction does not pass the executable check, execute step 1006; if the candidate execution instruction passes the executable check, execute step 1007.

[0203] In one embodiment, if the result of the executable check is that the candidate execution instruction fails the executable check, it means that the candidate execution instruction identified by the current language model is not accurate and is not a legally executable execution instruction. Therefore, step 1006 can be executed to call the language model to re-identify the instruction based on the candidate execution instruction and the guidance prompt information; if the result of the executable check is that the candidate execution instruction passes the executable check, it means that the candidate execution instruction identified by the current language model is an accurate and legally executable execution instruction. Therefore, step 1007 can be executed.

[0204] Step 1006 : Call the language model to re-identify the instruction of the requirement information according to the candidate execution instruction and the guidance prompt information, obtain a new candidate execution instruction, and return to step 1004 .

[0205] In one embodiment, when the language model is called to re-identify the instruction of the demand information based on the candidate execution instructions and the guidance prompt information, the model call request information can be regenerated based on the identified candidate execution instructions, the target object's demand information and the preset guidance prompt information, and then the regenerated model call request information is sent to the calling interface of the language model, and the language model is called to re-identify the instruction of the demand information based on the candidate execution instructions and the guidance prompt information. Wherein, when the number of times the language model is called to re-identify the instruction of the demand information is multiple times, the model call request information can be regenerated based on the multiple candidate execution instructions previously identified, the demand information and the guidance prompt information, and then the regenerated model call request information is sent to the calling interface of the language model, so that the language model is called to re-identify the instruction of the demand information based on the multiple candidate execution instructions previously identified and the guidance prompt information, and then the confidence of the newly identified candidate execution instructions can be improved through the multiple candidate execution instructions previously identified.

[0206] Step 1007: The candidate execution instruction that passes the executable check is used as the target execution instruction.

[0207] In one embodiment, if the executable check result obtained by performing the executable check on the candidate execution instruction is that the candidate execution instruction passes the executable check, it means that the candidate execution instruction is an execution instruction that matches the requirement information of the target object. There is no need to call the language model to re-identify the instruction on the requirement information. The terminal can be directly controlled to execute the operation task corresponding to the requirement information according to the candidate execution instruction. Therefore, the candidate execution instruction that passes the executable check can be used as the target execution instruction, so that the subsequent steps can control the terminal to execute the operation task corresponding to the requirement information according to the target execution instruction.

[0208] Step 1008: Obtain the corresponding instruction execution code according to the target execution instruction, run the instruction execution process in the virtual environment, and then call the instruction execution process to run the instruction execution code to simulate the target object control terminal to execute the operation task corresponding to the demand information.

[0209] In one embodiment, for each target execution instruction, there will be a corresponding instruction execution code, which records each step process of executing the target execution instruction. Therefore, by first obtaining the instruction execution code corresponding to the target execution instruction, and then running the instruction execution code in a virtual environment, the purpose of simulating the target object control terminal to execute the operation task corresponding to the demand information can be achieved.

[0210] In one embodiment, in order to achieve simultaneous concurrency of different execution instructions, after obtaining multiple target execution instructions and calling the instruction execution code corresponding to each target execution instruction, multiple virtual environments (for example, a virtual image of an operating system with a desktop) can be started at the same time, and the instruction execution codes of different target execution instructions can be executed simultaneously in different virtual environments. The parallel execution of different operation tasks is achieved by running multiple operating system images at the same time.

[0211] Step 1009: Create an execution record table including the target execution instruction, instruction execution code, execution status table item and execution result table item. When the control terminal completes the execution of the operation task corresponding to the demand information, the content of the execution status table item in the execution record table is updated to completed, and the execution result is written into the execution result table item.

[0212] In one embodiment, by using an execution record table to record and update the execution status of the instruction execution code, real-time control of the execution status of the instruction execution code can be achieved, thereby facilitating the terminal's automated execution of the operation task corresponding to the demand information.

[0213] Through the above-mentioned instruction execution method based on the language model from step 1001 to step 1009, when the demand information of the target object is received, at least one round of target operation is executed until a candidate execution instruction that passes the feasibility check is obtained; in the first round of target operation, the language model is first called to perform instruction recognition on the demand information according to the preset guidance prompt information to obtain the candidate execution instruction of the first round of target operation. Since the candidate execution instruction is obtained by the language model performing instruction recognition on the demand information according to the guidance prompt information, the confidence of the candidate execution instruction can be improved, which is conducive to obtaining a candidate execution instruction with a higher degree of matching with the demand information; then, the candidate execution instruction of the first round of target operation is selected. The executable instruction is checked to determine whether the candidate execution instruction passes the feasibility check. If the candidate execution instruction does not pass the executable check, it means that the candidate execution instruction identified by the current language model is not accurate. At this time, continue to the next round of target operation. In the next round of target operation, call the language model to re-identify the instruction of the requirement information according to the candidate execution instruction in the previous target operation and the guidance prompt information to obtain the candidate execution instruction in this round of target operation, and then re-check the executable instruction for the candidate execution instruction to determine whether the candidate execution instruction passes the feasibility check. It iterates and executes several rounds of target operation until the candidate execution instruction passes the executable check. Moreover, after obtaining the candidate execution instruction that passes the executable check, the candidate execution instruction is used as the target execution instruction, and the terminal is controlled to execute the operation task corresponding to the demand information according to the target execution instruction. Since the candidate execution instruction that passes the executable check is obtained by multiple calls to the language model to perform instruction recognition on the demand information based on the guidance prompt information and the previous candidate execution instructions, the confidence of the candidate execution instruction can be guaranteed. Moreover, since this process is completed automatically, the target object does not need to perform any additional operations, nor does it require the target object to perform multiple repeated interactions. Therefore, the interaction efficiency between the target object and the terminal can be improved, so as to achieve the purpose of simplifying the use of the target object, thereby improving the use experience of the target object.

[0214] The following are some practical examples to illustrate the application scenarios of the embodiments of the present application.

[0215] It should be noted that the language model-based instruction execution method provided in the embodiment of the present application can be applied to different application scenarios such as web search and voice chat. The following is an illustration using web search scenarios and voice chat scenarios as examples.

[0216] Scene 1

[0217] The instruction execution method based on the language model provided in the embodiment of the present application can be applied to web page search scenarios. For example, assuming that a user sends a request information to the terminal through the artificial intelligence interaction module of the terminal to search for weather conditions on a web page, the terminal will first receive the request information through the artificial intelligence interaction module, and then execute the instruction execution method provided in the embodiment of the present application, obtain the target execution instruction corresponding to the request information, and control the terminal to perform the operation task corresponding to the request information according to the target execution instruction. When the terminal is controlled to perform the operation task corresponding to the request information, it will first call the application programming interface to obtain the screen resolution of the display module, and obtain the initial coordinates of the operation cursor based on the screen resolution, and then determine the target coordinates of the search input box in the web page, and then simulate the user controlling the operation cursor to move from the initial coordinates to the target coordinates, so that the operation cursor is positioned in the search input box in the web page, and simulate the user querying the weather conditions through the search input box in the web page. At this time, the web page will jump to the page for displaying the weather conditions, and the user can obtain the weather conditions they want through this page.

[0218] Scene 2

[0219] The instruction execution method based on the language model provided in the embodiment of the present application can also be applied to voice chat scenarios. For example, assuming that a user chats with a friend through the instant messaging application of the terminal, when the user inputs the demand information for sending a chat message in the instant messaging application through the artificial intelligence interaction module in the terminal, the terminal will first receive the demand information through the artificial intelligence interaction module, and then execute the instruction execution method provided in the embodiment of the present application, obtain the target execution instruction corresponding to the demand information, and control the terminal to execute the operation task corresponding to the demand information according to the target execution instruction. When the terminal is controlled to execute the operation task corresponding to the demand information, it will first call the application programming interface to obtain the screen resolution of the display module, and obtain the initial coordinates of the operation cursor based on the screen resolution, and then determine the target coordinates of the information input box in the instant messaging application, and then simulate the user controlling the operation cursor to move from the initial coordinates to the target coordinates, so that the operation cursor is positioned in the information input box in the instant messaging application, and simulate the user sending a chat message through the information input box. At this time, the chat message will be displayed in the chat interface of the instant messaging application.

[0220] It will be appreciated that, although the various steps in the above-mentioned various flow charts are shown in sequence according to the indication of the arrows, these steps are not necessarily performed in sequence according to the order indicated by the arrows. Unless clearly stated in the present embodiment, the execution of these steps does not have strict order restrictions, and these steps can be performed in other orders. Moreover, at least a portion of the steps in the above-mentioned flow charts can include multiple steps or multiple stages, and these steps or stages are not necessarily performed at the same time, but can be performed at different times, and the execution order of these steps or stages is not necessarily performed in sequence, but can be performed in turn or alternately with at least a portion of the steps or stages in other steps or other steps.

[0221] 11 , an embodiment of the present application further discloses an instruction execution device based on a language model. The instruction execution device 1100 can implement the instruction execution method based on a language model in the previous embodiment. The instruction execution device 1100 includes:

[0222] The information receiving unit 1110 is used to receive the demand information of the target object;

[0223] The instruction processing unit 1120 is configured to execute at least one round of target operations until candidate execution instructions that pass the feasibility check are obtained; in the first round of target operations, the language model is called to perform instruction recognition on the requirement information according to preset guidance prompt information to obtain candidate execution instructions for the first round of target operations, the executable check is performed on the candidate execution instructions for the first round of target operations, and it is determined whether the candidate execution instructions for the first round of target operations pass the feasibility check; in the i-th round of target operations, the language model is called to perform instruction recognition on the requirement information according to the candidate execution instructions for the i-1th round of target operations and the guidance prompt information to obtain candidate execution instructions for the i-th round of target operations, the executable check is performed on the candidate execution instructions for the i-th round of target operations, and it is determined whether the candidate execution instructions for the i-th round of target operations pass the feasibility check, where i is an integer greater than 1;

[0224] The instruction execution unit 1130 is configured to use the candidate execution instruction that has passed the executable check as the target execution instruction, and control the terminal to execute the operation task corresponding to the demand information according to the target execution instruction.

[0225] In one embodiment, the instruction processing unit 1120 is specifically configured to:

[0226] Replace the instruction function in the candidate execution instruction with the target empty function to obtain the execution instruction to be verified;

[0227] Calling the instruction execution engine to simulate the execution of the instruction to be verified and obtain the simulation execution result;

[0228] According to the simulation execution result, it is determined whether the candidate execution instruction passes the feasibility check.

[0229] In one embodiment, the instruction processing unit 1120 is specifically configured to:

[0230] When the simulation execution result is that the instruction to be verified is successfully executed, it is determined that the candidate instruction has passed the executable verification;

[0231] Alternatively, when the simulation execution result is that the instruction to be verified is not successfully executed, it is determined that the candidate execution instruction has not passed the executable verification.

[0232] In one embodiment, the instruction execution unit 1130 is further configured to:

[0233] Obtain the corresponding instruction execution code according to the target execution instruction;

[0234] Run the instruction execution code in the virtual environment to control the terminal to perform the operation tasks corresponding to the demand information.

[0235] In one embodiment, the instruction execution unit 1130 is further configured to:

[0236] Run the command execution process in a virtual environment;

[0237] The instruction execution process is called to run the instruction execution code, and the control terminal executes the operation task corresponding to the demand information.

[0238] In one embodiment, the terminal includes a display module, which is used to display an operation cursor; the instruction execution unit 1140 is further used to:

[0239] Get the initial coordinates of the operation cursor;

[0240] Determine the target coordinates, which are the coordinates when executing the operation task corresponding to the demand information;

[0241] Control the cursor to move from the initial coordinates to the target coordinates;

[0242] Execute the operation task corresponding to the demand information at the target coordinate control terminal.

[0243] In one embodiment, the instruction execution unit 1130 is further configured to:

[0244] Call the application programming interface to obtain the screen resolution of the display module;

[0245] Get the initial coordinates of the operation cursor according to the screen resolution.

[0246] In one embodiment, the instruction execution device 1100 further includes:

[0247] A record table creation unit, configured to create an execution record table including a target execution instruction and an instruction execution code, wherein the execution record table further includes an execution status table entry and an execution result table entry;

[0248] The record table updating unit is used to control the terminal to update the content of the execution status table item to completed in the execution record table after completing the operation task corresponding to the demand information, and write the execution result into the execution result table item.

[0249] In one embodiment, the instruction processing unit 1120 is further configured to:

[0250] Generate model call request information based on demand information and preset guidance prompt information;

[0251] The model call request information is sent to the calling interface of the language model, and the language model is called to perform instruction recognition on the requirement information according to the guidance prompt information to obtain candidate execution instructions.

[0252] In one embodiment, the information receiving unit 1110 is further configured to:

[0253] Receiving voice information input by the target object;

[0254] Call the pre-trained speech recognition model to perform speech recognition on the speech information and obtain text information;

[0255] At least one of adding punctuation marks, correcting spelling errors, or correcting word order is performed on the text information to obtain the target object's demand information.

[0256] In one embodiment, the information receiving unit 1110 is further configured to:

[0257] performing at least one of noise removal and speech segmentation on the speech information to obtain preprocessed speech information;

[0258] Calling the speech feature extraction model to extract speech features from the preprocessed speech information to obtain speech feature information;

[0259] Call the pre-trained speech recognition model to perform speech recognition on the speech feature information to obtain text information.

[0260] It should be noted that since the language model-based instruction execution device 1100 of this embodiment can implement the language model-based instruction execution method of the previous embodiment, the language model-based instruction execution device 1100 of this embodiment and the language model-based instruction execution method of the previous embodiment have the same technical principles and the same beneficial effects. In order to avoid repetition, they will not be repeated here.

[0261] 12 , the embodiment of the present application further discloses a computer device for executing the instruction execution method based on the language model provided in the embodiment of the present application. The computer device 1200 includes:

[0262] at least one processor 1201;

[0263] At least one memory 1202, configured to store at least one program;

[0264] When at least one program is executed by at least one processor 1201 , the above-mentioned instruction execution method based on the language model is implemented.

[0265] An embodiment of the present application further discloses a computer-readable storage medium, which stores a computer program executable by a processor. When the computer program executable by the processor is executed by the processor, it is used to implement the instruction execution method based on the language model as described above.

[0266] An embodiment of the present application also discloses a computer program product, including a computer program or computer instructions, which are stored in a computer-readable storage medium. The processor of the instruction execution device based on the language model reads the computer program or computer instructions from the computer-readable storage medium, and the processor executes the computer program or computer instructions, so that the instruction execution device executes the instruction execution method based on the language model as described above.

[0267] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can, for example, be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0268] It should be understood that in this application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0269] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0270] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0271] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0272] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0273] The step numbers in the above method embodiment are only provided for the convenience of explanation and description, and do not limit the order of the steps. The execution order of each step in the embodiment can be adaptively adjusted according to the understanding of those skilled in the art.

Claims

1. A method for executing instructions based on a language model, executed by a computer device, comprising the following steps: Receive demand information from target objects; Execute at least one round of target operation until a candidate execution instruction that passes the feasibility check is obtained; In the first round of target operations, the language model is called to perform instruction recognition on the requirement information according to preset guidance prompt information to obtain candidate execution instructions for the first round of target operations, and the executable verification is performed on the candidate execution instructions for the first round of target operations to determine whether the candidate execution instructions for the first round of target operations pass the feasibility verification; In the i-th round of target operation, the language model is called, and instruction recognition is performed on the requirement information according to the candidate execution instructions of the previous i-1 round of target operation and the guidance prompt information to obtain the candidate execution instructions of the i-th round of target operation, and the executable check is performed on the candidate execution instructions of the i-th round of target operation to determine whether the candidate execution instructions of the i-th round of target operation pass the feasibility check, where i is an integer greater than 1; The candidate execution instruction that passes the executable check is used as the target execution instruction, and according to the target execution instruction, the control terminal executes the operation task corresponding to the demand information.

2. The instruction execution method according to claim 1, wherein the executable check comprises: Replace the instruction function function in the candidate execution instruction with the target empty function to obtain the execution instruction to be verified; Calling an instruction execution engine to simulate the execution of the to-be-verified execution instruction to obtain a simulation execution result; According to the simulation execution result, it is determined whether the candidate execution instruction passes the feasibility check.

3. The instruction execution method according to claim 2, wherein determining whether the candidate execution instruction passes the feasibility check according to the simulation execution result comprises: When the simulation execution result is successful execution of the to-be-verified execution instruction, determining that the candidate execution instruction passes the executable verification; Alternatively, when the simulation execution result is that the to-be-verified execution instruction is not successfully executed, it is determined that the candidate execution instruction has not passed the executable verification.

4. The instruction execution method according to any one of claims 1 to 3, wherein the step of executing the instruction according to the target and controlling the terminal to execute the operation task corresponding to the requirement information comprises: Acquire a corresponding instruction execution code according to the target execution instruction; The instruction execution code is run in a virtual environment to control the terminal to execute the operation task corresponding to the demand information.

5. The method according to claim 4, wherein the step of running the instruction execution code in a virtual environment and controlling the terminal to execute the operation task corresponding to the demand information comprises: Run the command execution process in a virtual environment; The instruction execution process is called to run the instruction execution code, and the terminal is controlled to execute the operation task corresponding to the demand information.

6. The method according to claim 4 or 5, wherein the terminal comprises a display module, and the display module is used to display an operation cursor; The controlling the terminal to execute the operation task corresponding to the demand information includes: Obtaining the initial coordinates of the operation cursor; Determine target coordinates, where the target coordinates are coordinates when executing the operation task corresponding to the demand information; Control the operation cursor to move from the initial coordinates to the target coordinates; The terminal is controlled to execute an operation task corresponding to the demand information at the target coordinates.

7. The method according to claim 6, wherein obtaining the initial coordinates of the operation cursor comprises: Calling an application programming interface to obtain the screen resolution of the display module; The initial coordinates of the operation cursor are acquired according to the screen resolution.

8. The instruction execution method according to any one of claims 4 to 7, further comprising: Creating an execution record table including the target execution instruction and the instruction execution code, wherein the execution record table also includes an execution status table entry and an execution result table entry; After the terminal is controlled to complete the operation task corresponding to the demand information, the content of the execution status table item in the execution record table is updated to completed, and the execution result is written into the execution result table item.

9. The method according to any one of claims 1 to 8, wherein calling the language model, performing instruction recognition on the demand information according to preset guidance prompt information, and obtaining candidate execution instructions for the first round of target operations comprises: Generate model call request information according to the demand information and the guidance prompt information; The model calling request information is sent to the calling interface of the language model, and the language model is called to perform instruction recognition on the requirement information according to the guiding prompt information to obtain the candidate execution instruction.

10. The method according to any one of claims 1 to 9, wherein receiving the demand information of the target object comprises: Receiving voice information input by the target object; Calling a pre-trained speech recognition model to perform speech recognition on the speech information to obtain text information; At least one of adding punctuation marks, correcting spelling errors, or correcting word order errors is performed on the text information to obtain the demand information.

11. The method according to claim 10, wherein calling a pre-trained speech recognition model to perform speech recognition on the speech information to obtain text information comprises: Performing at least one of noise removal and speech segmentation on the speech information to obtain preprocessed speech information; Calling a speech feature extraction model to extract speech features from the preprocessed speech information to obtain speech feature information; A pre-trained speech recognition model is called to perform speech recognition on the speech feature information to obtain the text information.

12. A language model-based instruction execution device, comprising: An information receiving unit, used for receiving demand information of a target object; An instruction processing unit, used for executing at least one round of target operation until a candidate execution instruction that passes the feasibility check is obtained; In the first round of target operations, the language model is called to perform instruction recognition on the requirement information according to preset guidance prompt information to obtain candidate execution instructions for the first round of target operations, and the executable verification is performed on the candidate execution instructions for the first round of target operations to determine whether the candidate execution instructions for the first round of target operations pass the feasibility verification; In the i-th round of target operation, the language model is called, and instruction recognition is performed on the requirement information according to the candidate execution instructions of the i-1-th round of target operation and the guidance prompt information to obtain the candidate execution instructions of the i-th round of target operation, and the executable check is performed on the candidate execution instructions of the i-th round of target operation to determine whether the candidate execution instructions of the i-th round of target operation pass the feasibility check, where i is an integer greater than 1; The instruction execution unit is used to use the candidate execution instruction that passes the executable check as the target execution instruction. According to the target execution instruction, the control terminal executes the operation task corresponding to the demand information.

13. A computer device comprising: at least one processor; at least one memory for storing at least one program; When at least one of the programs is executed by at least one of the processors, the language model-based instruction execution method according to any one of claims 1 to 11 is implemented.

14. A computer-readable storage medium storing a computer program executable by a processor, wherein the computer program executable by the processor is used to implement the instruction execution method based on the language model as described in any one of claims 1 to 11 when executed by the processor.

15. A computer program product, comprising a computer program or computer instructions, wherein the computer program or the computer instructions are stored in a computer-readable storage medium, and a processor of an instruction execution device based on a language model reads the computer program or the computer instructions from the computer-readable storage medium, and the processor executes the computer program or the computer instructions, so that the instruction execution device executes the instruction execution method based on the language model as described in any one of claims 1 to 11.

Images