Natural language instruction conversion method and device, equipment, medium and program product

Abstract

The invention provides a natural language instruction conversion method and device, equipment, a medium and a program product, and relates to the field of computer application, the method comprises the following steps: receiving an input text in a natural language form, and converting the input text into a first semantic vector; retrieving in a semantic vector database based on the first semantic vector to obtain at least one candidate semantic vector related to the first semantic vector, and obtaining at least one candidate performance function related to the at least one candidate semantic vector from a performance function resource pool, the performance functions in the performance function resource pool are associated with the semantic vectors in the semantic vector database; inputting the at least one candidate performance function into a large language model, and outputting a performance function call statement; and embedding the performance function call statement into a predefined machine executable script template to obtain a target operation script. And the universality and the flexibility of natural language instruction conversion are improved.

Description

Technical Field

[0001] This application relates to the field of computer applications, and in particular to a method, apparatus, device, medium, and program product for converting natural language instructions. Background Technology

[0002] In the field of machine control, although existing technical solutions have achieved intelligent control of intelligent agents using large language models, in practical applications, for intelligent machines with different functions, or when it is necessary to add or remove the smallest task unit executed by the machine, it is necessary to redesign and test the prompts for decomposing the task, which lacks generalization. Summary of the Invention

[0003] This application provides a natural language instruction conversion method, apparatus, device, and storage medium to solve the problem of high misjudgment rate in existing natural language instruction conversion methods.

[0004] To solve the above-mentioned technical problems, this application is implemented as follows:

[0005] In a first aspect, embodiments of this application provide a natural language instruction conversion method, the method comprising:

[0006] Receive input text in natural language form and convert the input text into a first semantic vector;

[0007] Based on the first semantic vector, a retrieval is performed in the semantic vector database to obtain at least one candidate semantic vector related to the first semantic vector, and at least one candidate function is obtained from the function resource pool that is associated with the at least one candidate semantic vector. The function functions in the function resource pool are associated with the semantic vectors in the semantic vector database.

[0008] Input the at least one of the candidate function into the large language model and output the function call statement;

[0009] The function call statement is embedded into a predefined machine-executable script template to obtain the target operation script.

[0010] Secondly, embodiments of this application also provide a natural language instruction conversion device. This natural language instruction conversion device includes:

[0011] A conversion module is used to receive input text in natural language form and convert the input text into a first semantic vector;

[0012] The acquisition module is used to retrieve at least one candidate semantic vector related to the first semantic vector from the semantic vector database, and to acquire at least one candidate function associated with the at least one candidate semantic vector from the function resource pool, wherein the function functions in the function resource pool are associated with the semantic vectors in the semantic vector database.

[0013] The output module is used to input the at least one candidate function into the large language model and output the function call statement;

[0014] The embedding module is used to embed the function call statement into a predefined machine-executable script template to obtain the target operation script.

[0015] Thirdly, embodiments of this application provide an electronic device, including a transceiver and a processor, wherein the processor is used for:

[0016] Receive input text in natural language form and convert the input text into a first semantic vector;

[0017] Based on the first semantic vector, a retrieval is performed in the semantic vector database to obtain at least one candidate semantic vector related to the first semantic vector, and at least one candidate function is obtained from the function resource pool that is associated with the at least one candidate semantic vector. The function functions in the function resource pool are associated with the semantic vectors in the semantic vector database.

[0018] Input the at least one of the candidate function into the large language model and output the function call statement;

[0019] The function call statement is embedded into a predefined machine-executable script template to obtain the target operation script.

[0020] Fourthly, embodiments of this application provide an electronic device, including: a processor, a memory, and a program stored in the memory and executable on the processor, wherein when the program is executed by the processor, it implements the steps of the natural language instruction conversion method as described in the first aspect above.

[0021] Fifthly, embodiments of this application provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the natural language instruction conversion method described in the first aspect above.

[0022] In a sixth aspect, embodiments of this application provide a computer program product, including computer instructions, which, when executed by a processor, implement the steps of the natural language instruction conversion method as described in the first aspect above.

[0023] This application's embodiment of the natural language instruction conversion method includes receiving input text in natural language form and converting the input text into a first semantic vector; retrieving at least one candidate semantic vector related to the first semantic vector from a semantic vector database, and obtaining at least one candidate function associated with the at least one candidate semantic vector from a function resource pool, wherein the function functions in the function resource pool are associated with the semantic vectors in the semantic vector database; inputting the at least one candidate function into a large language model and outputting a function call statement; embedding the function call statement into a predefined machine-executable script template to obtain a target operation script. This method, by retrieving associated candidate function functions from a semantic vector database based on the semantic vector of the natural language input text, generating a call statement through a large language model, and embedding it into a predefined script template, achieves direct conversion from natural language instructions to machine-executable operation scripts. This avoids, to some extent, the error propagation problem caused by complex steps in existing technologies, and improves the executability and accuracy of the generated script. Simultaneously, through flexible updates to the function resource pool, it can adapt to robots with different functions without redesigning prompt templates, improving the versatility and flexibility of natural language instruction conversion. Attached Figure Description

[0024] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1 This is one of the flowcharts of the natural language instruction conversion method provided in the embodiments of this application;

[0026] Figure 2 This is a flowchart illustrating the generation of an executable script provided in one embodiment of this application;

[0027] Figure 3 This is a second flowchart of a natural language instruction conversion method provided in an embodiment of this application;

[0028] Figure 4 This is a structural diagram of a natural language instruction conversion device provided in an embodiment of this application;

[0029] Figure 5 This is a structural diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0030] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0031] This application provides a natural language instruction conversion method. See also... Figure 1 , Figure 1 This is a flowchart of the natural language instruction conversion method provided in the embodiments of this application, such as... Figure 1 As shown, it includes the following steps:

[0032] Step 101: Receive input text in natural language form and convert the input text into a first semantic vector;

[0033] In this step, the input text in natural language can be understood as instructions or questions expressed by the user in everyday language, such as "open the document file on the computer" or "query this month's sales data." It covers multiple natural languages, such as Chinese and English, and has diverse expression forms, including colloquial and written styles. For example, natural language instructions can be transmitted through the following methods: Users can input natural language instructions into the system via keyboard. This is suitable for scenarios requiring manual input. Before starting an inspection task, operators can input natural language instructions through the management system interface; this process can be performed in real time and is suitable for temporary and special tasks. In addition, spoken instructions can also be received through a speech recognition system. When the inspection robot is in a dynamic environment, operators can directly issue instructions to the robot through the voice input system, suitable for scenarios where manual operation is inconvenient and a rapid response is required. After voice input, the system uses speech recognition technology to convert the instructions into text and transmit them to a large language model.

[0034] The aforementioned semantic vector can be understood as a numerical vector capable of representing the semantic information of text. Converting natural language text into semantic vectors allows computers to understand and process text semantics numerically. This application does not specifically limit the specific methods for converting natural language text into semantic vectors. For example, a task description in natural language text form can be input into a semantic vector model, such as the ByteDance General Model (BGEEmbedding) model or the Sentence-Bert model, to obtain a task description in semantic vector form. Taking the Sentence-Bert model as an example, the Siamese network model framework can be referenced, inputting different sentences into two bidirectional encoder representations from transformers (Bert) models with shared parameters. This can be understood as obtaining the sentence representation vector for each sentence within the same Bert model. Alternatively, word embedding-based methods, such as the Word2Vec word vector model and the Global Vectors for Word Representation (GloVe) model, can be used to first convert each word in the text into a word vector, and then combine the word vectors into a semantic vector of the text using certain strategies (such as average pooling, weighted summation, etc.). Furthermore, some pre-trained models based on the Transformer architecture, such as the Generative Pre-trained Transformer (GPT) and BERT, can also be fine-tuned to generate semantic vectors for the text. For example, text can be input into a finely tuned BERT model, and the output of a specific layer can be extracted as the semantic vector.

[0035] Step 102: Based on the first semantic vector, search the semantic vector database to obtain at least one candidate semantic vector related to the first semantic vector, and obtain at least one candidate function that is associated with the at least one candidate semantic vector from the function resource pool. The function functions in the function resource pool are associated with the semantic vectors in the semantic vector database.

[0036] The aforementioned semantic vector database is a pre-built database that stores a large number of semantic vectors and their corresponding information. These semantic vectors are typically generated from the documentation of various functional functions. For example, in an office automation scenario, the semantic vector database might store semantic vectors corresponding to various functional functions such as file operations and data processing.

[0037] The aforementioned retrieval of candidate semantic vectors can be achieved by calculating the similarity between the first semantic vector and each semantic vector in the semantic vector database. The similarity can be calculated using cosine similarity. For example, the semantic vectors in the vector database can be sorted in descending order of cosine similarity. Alternatively, similarity can be measured by calculating the linear distance between two semantic vectors in multidimensional space; the smaller the distance, the more similar the vectors.

[0038] The aforementioned function resource pool stores a collection of various functions that can be used to complete specific tasks, such as file reading functions and data calculation functions. The content of the function resource pool varies depending on the application scenario. In a data analysis scenario, it may include functions for data statistical analysis and data visualization; in an automation script writing scenario, it may include functions for file operations and system command execution.

[0039] The deployment environment for the aforementioned function resource pool and semantic vector database is not specifically limited in this application embodiment. They can be deployed on local servers, such as single-machine servers or server clusters; or cloud platform software, such as public clouds, private clouds, or hybrid clouds such as Alibaba Cloud, Tencent Cloud, and Amazon; or they can be deployed on edge computing devices such as industrial gateways and smart terminal devices.

[0040] Associating functional functions with semantic vectors can be understood as assigning a unique documentation to each functional function. This documentation includes the function's purpose, input and output parameters, and examples. For instance, the documentation for a file reading function would explain that the function reads file content from a specified path, with the file path as the input parameter and the file content as the output parameter. An example would be "Use this function to read the contents of the file 'C:\data\report.txt'". Each documentation is individually input into the semantic vector model to generate a unique semantic vector. This semantic vector is then associated with and stored in the semantic vector database. When a candidate semantic vector is retrieved, the associated candidate functional function can be obtained from the functional function resource pool.

[0041] Step 103: Input the at least one candidate function into the large language model and output the function call statement;

[0042] In this step, the aforementioned large language models can be understood as models with language understanding and generation capabilities, such as GPT-4, Wenxin Yiyan, and Tongyi Qianwen. These models are trained on a large amount of text data, enabling them to understand the semantics of input information and generate corresponding outputs based on the knowledge and patterns learned during training.

[0043] The function call statements described above can be understood as the large language model generating specific statements that can be used to call these functions based on the input candidate functions and its own understanding of natural language and programming knowledge. For example, in the Python environment, if the candidate function is "add_numbers", which calculates the sum of two numbers, the large language model might output a call statement like "result = add_numbers(3,5)", which explicitly specifies the function name and input parameters. The format and syntax of function call statements differ across programming languages, and the large language model will generate corresponding statements according to the specifications of the target programming language. In Java, for the same addition function, it can generate call statements such as "int result = addNumbers 3,5);".

[0044] Step 104: Embed the function call statement into a predefined machine-executable script template to obtain the target operation script.

[0045] In this step, you can refer to Figure 2 This approach leverages open-source large language models combined with Retrieval-Augmented Generation (RAG) technology to convert natural language input text into a pre-defined machine-executable script. RAG technology is a hybrid technique combining information retrieval and generative AI. Its core logic involves first converting the natural language task description into a semantic vector, retrieving multiple relevant candidate function names from a database, and then inputting this list of candidate functions along with the natural language task description into the large language model. This can be further enhanced by designed prompt templates, such as requiring sorting by the most efficient steps and outputting the call results in Python format, guiding the model to accurately filter and sort the functions. Based on the retrieved candidate functions and prompt templates, the large language model generates parameterized function call statements, embeds them into a predefined execution script template, and finally generates a complete, machine-executable script.

[0046] The aforementioned executable script template can be understood as a predefined running script template. This template may contain the basic framework structure required for task execution, such as task initialization, task execution, error handling, and task termination. Only the specific function call portion of the template is left blank, allowing for the subsequent filling in of specific function code and parameters. The pre-designed script framework with a specific structure and format can vary depending on different application scenarios and target execution environments; this application does not specifically limit this. For example, in an automation script scenario on a Linux system, the script template might begin with "#! / bin / bash" and contain some initialization settings and basic command execution structures. In a batch processing script template under a Windows system, it might begin with "@echo off". For example, in an automated file processing script, the predefined script template might be a framework containing the basic file operation flow, such as first setting the working directory, and then performing file reading, processing, and saving operations.

[0047] For preparing the aforementioned function documentation, a function resource pool can be pre-established, and detailed documentation can be written for each function, describing its purpose, input and output parameters, and examples. In some optional implementations, a standardized format can be used to write the documentation for subsequent automated retrieval and processing. Passing these function documentation documents into a semantic vector model (e.g., BGEEmbedding) yields a function vector database.

[0048] The above-mentioned embedding of function call statements into a predefined machine-executable script template allows the insertion of pre-generated parameterized function call statements into the predefined script template, ultimately generating a complete executable script. This embedding can be understood as inserting function call statements generated by a large language model into appropriate positions according to the structure and syntax requirements of the script template. For example, in a Python script template, there is a dedicated space for data processing function calls. Inserting the generated function call statement for calculating the average value of data, "average = calculate_average(data_list)", into this space will ultimately form a complete executable script, such as: "import numpyas np; data_list = [1,2,3,4,5];

[0049] average=calculate_average(data_list);print(average)”.

[0050] The predefined machines mentioned above can include industrial robots, collaborative robots such as Universal Robots and KUKA Robot Language (KRL), intelligent operation and maintenance equipment such as data center inspection robots, and general intelligent terminals such as smart speakers and service robots.

[0051] In this implementation, by retrieving associated candidate function statements from the semantic vector database based on the semantic vector of the natural language input text, generating calling statements through a large language model and embedding them into predefined script templates, the direct conversion of natural language instructions into machine-executable operation scripts is achieved. This avoids, to some extent, the error propagation problem caused by complex steps in existing technologies, and improves the executability and accuracy of the generated scripts. At the same time, through the flexible updating of the function resource pool, it can adapt to robots with different functions without redesigning prompt templates, thus improving the versatility and flexibility of natural language instruction conversion.

[0052] Optionally, after embedding the function call statement into a predefined execution script template to obtain the target operation script, the method further includes:

[0053] Execute the target operation script and record the execution result and the corresponding input text;

[0054] A preference optimization dataset is constructed based on multiple input texts and multiple execution results after multiple executions of the operation script;

[0055] Based on the aforementioned preference optimization dataset, the large language model is fine-tuned using the Direct Preference Optimization (DPO) technique.

[0056] In the natural language instruction conversion method of this application embodiment, the inspection robot's built-in script interpreter can execute corresponding instructions one by one according to the execution script to perform task operations. During field testing and use, the task execution results are monitored. The execution results may include recording success or failure status and recording task execution time. The data can be stored in the format of "task description; success or failure; execution time". This application embodiment does not limit the storage method of the records. For example, the data can be stored in the robot's local database (such as SQLite) or uploaded to a cloud server (such as Alibaba Cloud OSS) in real time for convenient subsequent data accumulation and retrieval.

[0057] After a period of use in real-world scenarios, the inspection robot will accumulate a large number of execution results. The aforementioned repeated execution of operation scripts can be understood as the robot repeatedly executing different or identical natural language instructions corresponding to target operation scripts in real-world scenarios, accumulating a sufficient number of execution records (e.g., hundreds to thousands). For example, a data center inspection robot might execute instructions such as "detect temperature" and "inspect equipment" 500 times within a month, generating 500 execution records. The aforementioned preference optimization dataset can be understood as a sample set used to train a large language model, with the aim of enabling the model to learn "better" script generation strategies—that is, successful and efficient strategies.

[0058] DPO is a method for directly optimizing model preferences without explicitly defining a reward function. It compares the outputs of different models and selects the result that best fits human preferences as the training objective. Optimization is mainly achieved by directly maximizing the objective function, using preferences to directly guide the optimization process, and it does not rely on a reinforcement learning framework.

[0059] The model takes a task description as input and outputs a function call statement. The optimization objective is to increase the probability of generating positive samples and decrease the probability of generating negative samples. After multiple rounds of preference alignment training, a large language model that generates positive samples as much as possible can be obtained; that is, the generated execution strategy is one that can be executed successfully and in the shortest possible time.

[0060] For example, given the input text "detect the temperature of area A in the computer room", the model before fine-tuning had a 60% probability of generating script B (taking 20 seconds) and a 40% probability of generating script A (taking 10 seconds). After fine-tuning via DPO, the model's probability of generating script A increased to 80%, the probability of generating script B decreased to 20%, and it no longer generated the failed script C. In another scenario, when the robot executes the instruction to "clean the conference room", the model after fine-tuning is more inclined to call the function combination of "planning the shortest path and efficient cleaning mode", which shortens the execution time from 30 seconds to 20 seconds and increases the success rate from 85% to 98%.

[0061] In this implementation, by executing the target operation script and recording the execution results and corresponding input text, a preference optimization dataset is constructed. Then, the Direct Preference Optimization (DPO) technique is used to fine-tune the large language model, which can achieve continuous optimization of the large language model generation strategy, improve the success rate and efficiency of robot task execution, and enable the robot to better adapt to the needs of actual scenarios.

[0062] Optionally, the construction of the preference optimization dataset based on multiple input texts and multiple execution results after multiple executions of the operation script includes:

[0063] Obtain multiple semantic vectors corresponding to multiple input texts, calculate the cosine similarity of the semantic vectors corresponding to different input texts, and determine that different input texts with the cosine similarity exceeding a preset threshold are semantically the same;

[0064] For multiple execution results corresponding to semantically identical input texts, select the operation script corresponding to the execution result with successful execution and the shortest execution time as the positive sample, and the operation scripts corresponding to the remaining execution results as the negative samples

[0065] Construct a preference optimization data set based on the input text, positive samples, and negative samples.

[0066] In the natural language instruction conversion method of the embodiments of the present application, each task description can be encoded using a semantic vector model and converted into a semantic vector representation. The embodiments of the present application do not limit the conversion method. Exemplarily, the input text can be directly passed into a pre-trained semantic vector model to obtain a vector; or the text can be pre-processed first, such as removing stop words ("of", "once"), and then input into the model to generate a vector to reduce noise interference. Then, calculate the cosine similarity between the task description in each task execution record and the task descriptions of all other records one by one. Set a similarity threshold (for example, 0.8). When the cosine similarity of two task descriptions exceeds this threshold, it is determined that they are semantically the same. The above preset threshold can be used as the critical value for determining semantic identity and can be adjusted according to the actual scenario. For example, if the threshold is set to 0.8, the two texts with the above cosine similarity of 0.92 are determined to be semantically the same, while those with 0.35 are determined to be different.

[0067] For execution records with the same task description semantics, select the execution strategy with successful execution and the shortest execution time as the only positive sample, and other strategies as negative samples. For the same task description, construct multiple positive and negative sample pairs to construct a preference optimization data set. Exemplarily, training samples in the form of triples can be formed.

[0068] For example: The positive sample A (the task description is successfully executed and the time is 10 seconds) can be paired with the following negative samples respectively:

[0069] Negative sample B (the task fails to execute and the time is 15 seconds);

[0070] Negative sample C (the task is successfully executed and the time is 20 seconds);

[0071] Two training data samples are formed as (task description, execution strategy A, execution strategy B) and (task description, execution strategy A, execution strategy C).

[0072] In this implementation, the constructed preference optimization dataset can accurately reflect the characteristics of the "better" execution strategy in the actual scenario, providing high-quality training data for the subsequent DPO fine-tuning of the large language model, and ensuring that the scripts generated by the model are more in line with the requirements of efficient and successful execution.

[0073] Optionally, the fine-tuning of the large language model based on the preference optimization dataset using direct preference optimization techniques includes:

[0074] Based on the preference optimization dataset, the first probability is higher than the second probability. The first probability is the probability that the large language model generates a function call statement corresponding to a positive sample based on the input text, and the second probability is the probability that the large language model generates a function call statement corresponding to a negative sample based on the input text.

[0075] In the natural language instruction conversion method of this application embodiment, the first probability can be understood as the probability that the large language model generates a function call statement corresponding to a positive sample after receiving a certain input text. The second probability can be understood as the probability that the large language model generates a function call statement corresponding to a negative sample after receiving the same input text. For example, the negative samples corresponding to the input text include "call a normal sensor function, path: area A, sampling frequency: 1 time / 5 seconds" (successful but inefficient) and "call a damaged sensor interface function" (execution failed). Then the second probability is the probability that the model generates these two types of call statements. The core logic of making the first probability higher than the second probability is that when fine-tuning the large language model through DPO technology, the triples (input text, positive sample, negative sample) in the preference optimization dataset are used as training samples to adjust the model parameters so that the "probability of generating a positive sample" is higher than the "probability of generating a negative sample". Ultimately, this allows the model to be more inclined to output efficient and successful function call statements in actual tasks. DPO technology does not require explicit definition of a reward function. It directly optimizes the output results of positive and negative samples by comparing the model. By maximizing the objective function, the probability (first probability) of generating the function call statement corresponding to the positive sample when receiving the task description is higher than the probability (second probability) of generating the call statement corresponding to the negative sample.

[0076] Specifically, during the fine-tuning process, the model takes the task description as input and positive and negative samples as optimization targets. Through multiple rounds of preference alignment training, the model parameters are gradually adjusted, ultimately making the model more "preferential" to generate positive samples (i.e., the first probability is higher than the second probability), ensuring that the output execution strategy is more likely to be successful and efficient.

[0077] For example, in a data center inspection scenario: the input text is "Inspect equipment in area B of the data center and upload abnormal data". The positive samples in the preference optimization dataset are "Call the infrared thermal imaging function (scanning range: all equipment in area B) + call the 5G transmission function (priority: high)" (execution successful, time taken 15 seconds). The negative samples include "Call the visible light imaging function (scanning range: some equipment in area B) + call the Wi-Fi transmission function" (successful but with a high false negative rate, time taken 25 seconds) and "Call the offline Bluetooth transmission function" (execution failed, time taken 5 seconds).

[0078] Before fine-tuning, the model had a first probability of generating a positive sample of 25%, and a second probability of generating two negative samples of 50% and 25%, respectively. After fine-tuning with DPO, the first probability increased to 60%, and the second probability decreased to 30% and 10%, respectively, which can achieve a higher probability of generating positive samples than negative samples.

[0079] In this implementation, by fine-tuning the large language model using direct preference optimization techniques based on the preference optimization dataset, the first probability of the model generating a function call statement corresponding to a positive sample is higher than the second probability of generating a statement corresponding to a negative sample. This enables the large language model to learn preferences for better execution strategies, i.e., more successfully and efficiently. This can improve the quality of the operation scripts generated by the model, thereby increasing the success rate and efficiency of robot task execution.

[0080] Optionally, the correlation between the functional function and the semantic vector is achieved in the following way:

[0081] Each function is configured with a unique corresponding documentation, which includes the function's purpose, input and output parameters, and examples.

[0082] Each of the aforementioned documentation is individually input into the semantic vector model to generate a unique semantic vector, and the semantic vector is associated with the corresponding functional function and stored in the semantic vector database.

[0083] In the natural language instruction conversion method of this application embodiment, the aforementioned documentation can be understood as a detailed textual description of the function, which can be used to clarify the core information of the function and ensure that the subsequent semantic vector model can accurately understand the meaning of the function. This application embodiment does not limit the format of the documentation, as long as it fully covers the functional purpose, input and output parameters, and examples. The aforementioned input parameters can refer to the variables that need to be passed in when calling the function. For example, the input parameters of the aforementioned temperature acquisition function may include "region number (string type, such as "A01")" and "sampling frequency (integer type, such as 1 time / second)". The aforementioned output parameters can be understood as the result returned after the function is executed. For example, the output parameters of the temperature acquisition function may be "temperature value (floating-point type, such as 25.3℃)" and "acquisition status (boolean type, true indicates success, false indicates failure)". For example: the function usage is demonstrated through a specific calling scenario, such as "calling the temperature acquisition function (region number: "A01", sampling frequency: 1), returning (25.3, true)".

[0084] For example, the documentation for the "device photo capture function" can be written as: "Function: Controls the robot's camera to capture and store an image of the appearance of a specified device; Input parameters: Device ID (string, such as "Server-001"), Shooting mode (enumeration type, normal / close-up); Output parameters: Image storage path (string, such as " / data / Server-001.jpg"), Shooting time (timestamp); Example: Calling the device photo capture function (Device ID: "Server-001", Shooting mode: normal), returns (" / data / Server-001.jpg", 1680000000)". Each documentation, after being processed by a semantic vector model, will yield a fixed-dimensional numerical vector. This vector uniquely corresponds to the functional function it describes and accurately reflects the semantic features of the function. For example, the semantic vectors of "temperature acquisition" and "photo capture" are significantly different, while the semantic vectors of "temperature acquisition" and "temperature and humidity detection" are highly similar.

[0085] The aforementioned associated storage can be understood as binding the generated semantic vector with the identification information of the corresponding functional function, such as the function ID and function name, and storing them together in the semantic vector database. For example, the semantic vector generated from the documentation for the "temperature acquisition function" is V001 (IDF001). The database stores "V001, F001, temperature acquisition function" to ensure that the corresponding functional function can be quickly located when searching through semantic vectors later.

[0086] For example, in the semantic vector database, the documentation for the "device photo-taking function" (ID: F002) can be processed by the BGEEmbedding model to generate a semantic vector V002, stored as "V002, F002, device photo-taking function"; the documentation for the "path planning function" (ID: F003) generates a semantic vector V003, stored as "V003, F003, path planning function". When a user inputs the natural language command "take a picture of the server", the generated semantic vector has the highest similarity to V002, and the system can retrieve the "device photo-taking function" corresponding to F002 as a candidate through database association.

[0087] In this implementation, by configuring a description document containing the function's purpose, input and output parameters, and examples for each function, and generating a unique semantic vector from its input semantic vector model and storing it in the semantic vector database, a precise binding between the function and the semantic vector is achieved. This provides a reliable foundation for semantic retrieval based on natural language instructions, ensuring that matching functions can be quickly located, improving the accuracy and efficiency of natural language to operation script conversion. At the same time, only the corresponding description document and vector need to be updated, which facilitates the expansion and maintenance of the function.

[0088] Optionally, retrieving at least one candidate semantic vector related to the first semantic vector from the semantic vector database based on the first semantic vector includes:

[0089] Calculate the cosine similarity between the first semantic vector and each semantic vector in the semantic vector database, and sort the semantic vectors in the vector database in descending order of the cosine similarity.

[0090] Select a preset number of semantic vectors that rank highly as candidate semantic vectors; or, set a first threshold and select semantic vectors whose cosine similarity exceeds the first threshold as candidate semantic vectors.

[0091] In the natural language instruction conversion method of this application embodiment, the sorting of semantic vectors in the vector database according to the cosine similarity from high to low can be understood as filtering and sorting functional functions. Cosine similarity can be understood as an index that measures the consistency of the directions of two vectors, with a value range of [-1, 1]. The closer the value is to 1, the more similar the two vectors are semantically. By calculating the cosine similarity between the first semantic vector (natural language instruction vector) and each vector (functional function description document vector) in the semantic vector database, the degree of semantic association between the two can be determined. The semantic vectors in the database can be arranged in order of cosine similarity from high to low, so that the functional function vectors that are closest to the semantics of the natural language instruction are ranked first.

[0092] For example, if a user inputs "detect the temperature of area A in the computer room", its first semantic vector is V1; there are three vectors in the semantic database: V2 (corresponding to "temperature detection function"), V3 (corresponding to "humidity detection function"), and V4 (corresponding to "equipment handling function"). The cosine similarity between V1 and V2 is calculated to be 0.9, with V3 it is 0.6, and with V4 it is 0.2. Therefore, the ranking result is V2 > V3 > V4.

[0093] Regarding the specific methods for selecting candidate semantic vectors, a preset number of semantic vectors that rank highly can be selected as candidate semantic vectors. The preset number is a fixed value set according to the actual scenario requirements, such as 3 or 5. Alternatively, a first threshold can be set for filtering. The first threshold can be used to determine whether the semantics are sufficiently similar. Semantic vectors with a cosine similarity exceeding the threshold can be selected as candidates.

[0094] In some alternative implementations, when inputting the natural language task description and the retrieved list of candidate functions, a prompt template can be designed to ensure that the large language model performs accurate filtering and sorting. For example, the prompt template for the large language model could be: "Based on the following task description, call the functions to be implemented from the list of candidate functions, sort them according to the most efficient implementation steps, and output the call results in Python format:"

[0095] Task Description: Fill in the natural language description;

[0096] Candidate function list: Enter a list of multiple candidate functions retrieved.

[0097] In this implementation, by calculating and sorting the cosine similarity between the first semantic vector and each vector in the semantic vector database, candidate semantic vectors are obtained by selecting a preset number of top-ranked vectors or by setting a threshold. This enables the accurate retrieval of function vectors related to natural language instructions from the database, providing a reliable candidate basis for the subsequent generation of executable operation scripts. This improves the accuracy and efficiency of the conversion from natural language to operation scripts. At the same time, the retrieval strategy can be flexibly adjusted according to the actual scenario, enhancing the adaptability of the method.

[0098] In some alternative implementations, the complete process can be referred to Figure 3First, natural language instructions are input. Task requirements issued by the user or system, such as "inspect the temperature of area A in the computer room" or "clean the corridor floor," are the core inputs driving the process. The robot executes script generation by combining semantic understanding of the natural language instructions, referencing function documentation to match available functions, and applying predefined templates to generate an executable task script. During task execution, the robot can call hardware or software functions based on the generated script to complete the task, such as activating sensors, controlling motors, and uploading data, and record the execution results, including success or failure, time consumption, and data output. Storing all task execution data can serve as "feedback material" for model optimization; the execution results are transformed into training data required for Direct Preference Optimization (DPO). The preference data is used to fine-tune the large model, allowing the model to "learn" better strategies, such as prioritizing script logic with high success rates and short execution times, improving the execution quality of future tasks, forming a closed loop of "execution, feedback, optimization to better execution." While a single task process may end, by "optimizing the closed loop," the model can be continuously iterated, making the robot's ability to perform tasks increasingly intelligent, better understanding user instructions, executing more efficiently, and making fewer mistakes.

[0099] See Figure 4 , Figure 4 This is a structural diagram of a natural language instruction conversion device provided in another embodiment of this application.

[0100] like Figure 4 As shown, the natural language instruction conversion device 400 includes:

[0101] The conversion module 401 is used to receive input text in natural language form and convert the input text into a first semantic vector;

[0102] The acquisition module 402 is used to retrieve at least one candidate semantic vector related to the first semantic vector from the semantic vector database, and to acquire at least one candidate function associated with the at least one candidate semantic vector from the function resource pool, wherein the function functions in the function resource pool are associated with the semantic vectors in the semantic vector database.

[0103] Output module 403 is used to input the at least one candidate function into the large language model and output a function call statement;

[0104] The embedding module 404 is used to embed the function call statement into a predefined machine-executable script template to obtain the target operation script.

[0105] Optionally, the natural language instruction conversion device 400 further includes:

[0106] A recording module is used to execute the target operation script and record the execution result and the input text corresponding to the execution result;

[0107] The building module is used to construct a preference optimization dataset based on multiple input texts and multiple execution results after multiple executions of the operation script;

[0108] The fine-tuning module is used to fine-tune the large language model based on the preference optimization dataset using the Direct Preference Optimization (DPO) technique.

[0109] Optionally, the building module can also be used for:

[0110] Obtain multiple semantic vectors corresponding to multiple input texts, calculate the cosine similarity of the semantic vectors corresponding to different input texts, and determine different input texts whose cosine similarity exceeds a preset threshold as having the same semantics;

[0111] For multiple execution results corresponding to multiple input texts with the same semantics, the operation script corresponding to the execution result with the shortest execution time is selected as the positive sample, and the operation scripts corresponding to the other execution results are selected as the negative samples.

[0112] A preference optimization dataset is constructed based on the input text, positive samples, and negative samples.

[0113] Optionally, the fine-tuning module can also be used for:

[0114] Based on the preference optimization dataset, the first probability is higher than the second probability. The first probability is the probability that the large language model generates a function call statement corresponding to a positive sample based on the input text, and the second probability is the probability that the large language model generates a function call statement corresponding to a negative sample based on the input text.

[0115] Optionally, the correlation between the functional function and the semantic vector is achieved in the following way:

[0116] Each function is configured with a unique corresponding documentation, which includes the function's purpose, input and output parameters, and examples.

[0117] Each of the aforementioned documentation is individually input into the semantic vector model to generate a unique semantic vector, and the semantic vector is associated with the corresponding functional function and stored in the semantic vector database.

[0118] Optionally, the acquisition module 402 can also be used for:

[0119] Calculate the cosine similarity between the first semantic vector and each semantic vector in the semantic vector database, and sort the semantic vectors in the vector database in descending order of the cosine similarity.

[0120] Select a preset number of semantic vectors that rank highly as candidate semantic vectors; or, set a first threshold and select semantic vectors whose cosine similarity exceeds the first threshold as candidate semantic vectors.

[0121] For details, see Figure 5 As shown in the figure, this application embodiment also provides an electronic device, including a bus 501, a transceiver 502, an antenna 503, a bus interface 504, a processor 505, and a memory 506.

[0122] Processor 505, used for:

[0123] Receive input text in natural language form and convert the input text into a first semantic vector;

[0124] Based on the first semantic vector, a retrieval is performed in the semantic vector database to obtain at least one candidate semantic vector related to the first semantic vector, and at least one candidate function is obtained from the function resource pool that is associated with the at least one candidate semantic vector. The function functions in the function resource pool are associated with the semantic vectors in the semantic vector database.

[0125] Input the at least one of the candidate function into the large language model and output the function call statement;

[0126] The function call statement is embedded into a predefined machine-executable script template to obtain the target operation script.

[0127] exist Figure 5 In this document, a bus architecture (represented by bus 501) is used. Bus 501 can include any number of interconnected buses and bridges, linking various circuits including one or more processors represented by processor 505 and memory represented by memory 506. Bus 501 can also link various other circuits such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and therefore will not be described further herein. Bus interface 504 provides an interface between bus 501 and transceiver 502. Transceiver 502 can be a single element or multiple elements, such as multiple receivers and transmitters, providing a unit for communicating with various other devices over a transmission medium. Data processed by processor 505 is transmitted over a wireless medium via antenna 503, which further receives data and transmits it to processor 505.

[0128] Processor 505 manages bus 501 and general processing, and also provides various functions, including timing, peripheral interfaces, voltage regulation, power management, and other control functions. Memory 506 can be used to store data used by processor 505 during operation.

[0129] Optionally, the processor 505 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or a complex programmable logic device (CPLD).

[0130] Optionally, the processor 505 can also be used for:

[0131] Execute the target operation script and record the execution result and the corresponding input text;

[0132] A preference optimization dataset is constructed based on multiple input texts and multiple execution results after multiple executions of the operation script;

[0133] Based on the aforementioned preference optimization dataset, the large language model is fine-tuned using the Direct Preference Optimization (DPO) technique.

[0134] Optionally, the processor 505 can also be used for:

[0135] Obtain multiple semantic vectors corresponding to multiple input texts, calculate the cosine similarity of the semantic vectors corresponding to different input texts, and determine different input texts whose cosine similarity exceeds a preset threshold as having the same semantics;

[0136] For multiple execution results corresponding to multiple input texts with the same semantics, the operation script corresponding to the execution result with the shortest execution time is selected as the positive sample, and the operation scripts corresponding to the other execution results are selected as the negative samples.

[0137] A preference optimization dataset is constructed based on the input text, positive samples, and negative samples.

[0138] Optionally, a unique documentation is configured for each function, including the function's purpose, input and output parameters, and examples.

[0139] Each of the aforementioned documentation is individually input into the semantic vector model to generate a unique semantic vector, and the semantic vector is associated with the corresponding functional function and stored in the semantic vector database.

[0140] Optionally, the processor 505 can also be used for:

[0141] Calculate the cosine similarity between the first semantic vector and each semantic vector in the semantic vector database, and sort the semantic vectors in the vector database in descending order of the cosine similarity.

[0142] Select a preset number of semantic vectors that rank highly as candidate semantic vectors; or, set a first threshold and select semantic vectors whose cosine similarity exceeds the first threshold as candidate semantic vectors.

[0143] It should be noted that the electronic device provided in this application embodiment is a device capable of executing the above-described natural language instruction conversion method. Therefore, all implementation methods in the above-described natural language instruction conversion method embodiments are applicable to this electronic device and can achieve the same or similar beneficial effects. To avoid repetition, this embodiment will not elaborate further.

[0144] This invention also provides an electronic device, including: a processor, a memory, and a program stored in the memory and executable on the processor. When the program is executed by the processor, it implements the various processes of the above-described natural language instruction conversion method embodiments and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0145] This application also provides a computer-readable storage medium storing a computer program. When executed by a processor, this computer program implements the various processes of the above-described natural language instruction conversion method embodiments and achieves the same technical effects. To avoid repetition, it will not be described again here. The computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc.

[0146] This application also provides a computer program product, including computer instructions. When executed by a processor, the computer instructions implement the various processes of the above-described natural language instruction conversion method embodiments and achieve the same technical effects. To avoid repetition, these will not be described again here.

[0147] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0148] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0149] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A natural language instruction conversion method, characterized in that, The method includes: Receive input text in natural language form and convert the input text into a first semantic vector; Based on the first semantic vector, a retrieval is performed in the semantic vector database to obtain at least one candidate semantic vector related to the first semantic vector, and at least one candidate function is obtained from the function resource pool that is associated with the at least one candidate semantic vector. The function functions in the function resource pool are associated with the semantic vectors in the semantic vector database. Input the at least one of the candidate function into the large language model and output the function call statement; The function call statement is embedded into a predefined machine-executable script template to obtain the target operation script.

2. The method according to claim 1, characterized in that, After embedding the function call statement into a predefined execution script template to obtain the target operation script, the process further includes: Execute the target operation script and record the execution result and the corresponding input text; A preference optimization dataset is constructed based on multiple input texts and multiple execution results after multiple executions of the operation script; Based on the aforementioned preference optimization dataset, the large language model is fine-tuned using the Direct Preference Optimization (DPO) technique.

3. The method according to claim 2, characterized in that, The preference optimization dataset constructed based on multiple input texts and multiple execution results after multiple executions of the operation script includes: Obtain multiple semantic vectors corresponding to multiple input texts, calculate the cosine similarity of the semantic vectors corresponding to different input texts, and determine different input texts whose cosine similarity exceeds a preset threshold as having the same semantics; For multiple execution results corresponding to multiple input texts with the same semantics, the operation script corresponding to the execution result with the shortest execution time is selected as the positive sample, and the operation scripts corresponding to the other execution results are selected as the negative samples. A preference optimization dataset is constructed based on the input text, positive samples, and negative samples.

4. The method according to claim 2, characterized in that, The fine-tuning of the large language model based on the preference optimization dataset using direct preference optimization techniques includes: Based on the preference optimization dataset, the first probability is higher than the second probability. The first probability is the probability that the large language model generates a function call statement corresponding to a positive sample based on the input text, and the second probability is the probability that the large language model generates a function call statement corresponding to a negative sample based on the input text.

5. The method according to claim 1, characterized in that, The correlation between the functional function and the semantic vector is achieved in the following way: Each function is configured with a unique corresponding documentation, which includes the function's purpose, input and output parameters, and examples. Each of the aforementioned documentation is individually input into the semantic vector model to generate a unique semantic vector, and the semantic vector is associated with the corresponding functional function and stored in the semantic vector database.

6. The method according to claim 1, characterized in that, The step of retrieving at least one candidate semantic vector related to the first semantic vector from the semantic vector database based on the first semantic vector includes: Calculate the cosine similarity between the first semantic vector and each semantic vector in the semantic vector database, and sort the semantic vectors in the vector database in descending order of the cosine similarity. Select a preset number of semantic vectors that rank highly as candidate semantic vectors; or, set a first threshold and select semantic vectors whose cosine similarity exceeds the first threshold as candidate semantic vectors.

7. A natural language instruction conversion device, characterized in that, include: A conversion module is used to receive input text in natural language form and convert the input text into a first semantic vector; The acquisition module is used to retrieve at least one candidate semantic vector related to the first semantic vector from the semantic vector database, and to acquire at least one candidate function associated with the at least one candidate semantic vector from the function resource pool, wherein the function functions in the function resource pool are associated with the semantic vectors in the semantic vector database. The output module is used to input the at least one candidate function into the large language model and output the function call statement; The embedding module is used to embed the function call statement into a predefined machine-executable script template to obtain the target operation script.

8. An electronic device, characterized in that, include: A processor, a memory, and a program stored in the memory and executable on the processor, wherein the program, when executed by the processor, implements the steps of the natural language instruction conversion method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the natural language instruction conversion method as described in any one of claims 1 to 6.

10. A computer program product, characterized in that, It includes computer instructions that, when executed by a processor, implement the steps of the natural language instruction conversion method as described in any one of claims 1 to 6.

Images