Man-machine interaction system and method based on large language model

Abstract

The invention provides a man-machine interaction system and method based on a large language model, and relates to the technical field of man-machine interaction, the system comprises an operation instruction generation module and controlled equipment, and software middleware and man-machine interaction software run on the controlled equipment; the operation instruction generation module processes the user request information by using a built-in large language model, and generates a character operation instruction in a format which can be analyzed by the software middleware; and the software middleware is used for mapping the character operation instruction and forming a control action equivalent to manual operation on the man-machine interaction software based on a control mechanism provided by an operation system. According to the method, decoupling of business and operation is achieved through the software middleware and the large language model, the large language model does not need to learn interfaces and functions special for the man-machine interaction software, and the control process of the man-machine interaction software is made to be higher in efficiency, higher in intelligence and lower in cost.

Description

Technical Field

[0001] The present invention relates to the technical field of human-computer interaction, and in particular to a human-computer interaction system and method based on a large language model. Background Art

[0002] With the rapid development of artificial intelligence (AI) technology, the intelligence of large language models has achieved significant breakthroughs. Large language models now have the ability to understand information at or above the level of an average person and possess the capabilities of a junior assistant to assist in problem solving. Furthermore, they are particularly adept at understanding and learning fixed knowledge-based systems. Therefore, it is possible to train AI large language models to replace humans in controlling equipment. A common industry practice is for device control software developers to define a set of control interfaces or function libraries for controlling the controlled equipment. These libraries are dedicated to the controlled equipment. Professional equipment operators then write descriptions for these interfaces and manually train the AI ​​large language model to understand the functions and specific usage of these libraries, as well as how to complete human-generated commands for the controlled equipment. The AI ​​large language model then uses these libraries to generate executable code to control the controlled equipment. Ultimately, the trained AI large language model generates correct and executable code and deploys it to the controlled equipment. When the device control task is executed, the executable code performs the operations according to the user's commands.

[0003] However, this approach has the following drawbacks: 1. Because function libraries are designed specifically for the controlled device and match the underlying implementation functions of the device's control mechanism, each device requires a dedicated function library for the large language model to learn from, adding significant additional development effort. 2. Human operators are required to define the purpose and specific usage of the function libraries related to the operational instructions and input them into the large language model in text that is easily understood by the large language model, increasing the workload for them to first learn and understand these specialized libraries. 3. During training, the AI ​​large language model outputs executable code. Verifying its correctness requires a human operator with a professional understanding of coding to detect errors in the AI ​​large language model's output. Without professional coders, the AI ​​large language model's output must be verified on a simulation platform, which is both costly and time-consuming. 4. Complex device-specific function libraries and functions require a high level of understanding of the AI ​​large language model. 5. If the business logic of the controlled device changes, the device-specific function library, tied to that business, typically needs to be redeveloped, and the training files for the large model also need to be rewritten, representing a significant workload.

[0004] Therefore, there is an urgent need to provide a human-computer interaction system and method based on a large language model to overcome the shortcomings of the current traditional method of using large language models, such as excessive resource investment, excessive training difficulty, and difficulty in checking correctness. Summary of the Invention

[0005] In view of this, it is necessary to provide a human-computer interaction system and method based on a large language model to solve technical problems existing in the existing technology, such as excessive resource investment, excessive training difficulty, and difficulty in checking correctness.

[0006] In a first aspect, in order to solve the above technical problems, the present invention provides a human-computer interaction system based on a large language model, comprising: an operation instruction generation module with a built-in large language model and a controlled device, wherein software middleware and human-computer interaction software are running on the controlled device; The operation instruction generation module is used to process the user request information based on the built-in large language model and generate text operation instructions in a format that can be parsed by the software middleware; The software middleware is used to map the text operation instructions to generate control actions that perform equivalent manual operations on the human-computer interaction software.

[0007] In one possible implementation, the controlled device also includes an operating system, which provides a control mechanism for triggering the human-computer interaction software to respond. The control mechanism includes an interface control interface, an operation library, a function, a message or event triggering mechanism, and the control action can call the control mechanism to simulate the action of manually operating the user interface.

[0008] In one possible implementation, the human-computer interaction software includes a user interface; The user interface is used to respond to manual operations and display the results of the operations.

[0009] In a possible implementation, the operation instruction generation module includes a model training unit and a model use unit; The model training unit is used to train the initial large language model based on the relevant usage documents of the human-computer interaction software to obtain the trained large language model; The model using unit is used to process the user request information based on the trained large language model to generate text operation instructions in a format that can be parsed by software middleware.

[0010] In a possible implementation, the user interface includes an interface page and interface controls on the page, and the related usage document includes functions of the interface controls and operation descriptions of the interface controls.

[0011] In one possible implementation, the system further includes an output observation module, which is used to record the text operation instructions, and compare the text operation instructions with theoretical operation instructions or feed back the text operation instructions to a human operator to determine whether the text operation instructions are accurate.

[0012] In a second aspect, the present invention further provides a human-computer interaction method based on a large language model, applicable to the human-computer interaction system based on a large language model described in any of the possible implementations above, the method comprising: Process user request information based on a large language model and generate text operation instructions in a format that can be parsed by the software middleware; The text operation instructions are mapped based on the software middleware to generate control actions that perform equivalent manual operations on the human-computer interaction software.

[0013] The beneficial effects of the present invention are as follows: the human-computer interaction system based on a large language model provided by the present invention uses the large language model to output text-formatted operation instructions. The decoupling of business and operation can be achieved through the operation mechanism of the human-computer interaction software provided by the software middleware running on the controlled device and the operating system of the controlled device. In other words, the large language model only needs to understand human operator instructions and user needs by learning the relevant usage documentation of the human-computer interaction system and generate text operation instructions. There is no need to specially compile a function library for the interface and function of the human-computer interaction software on the controlled device, nor is there a need for human operators to additionally compile text input for the large language model based on the function library that is easy for the artificial intelligence model to understand, thereby greatly reducing the development workload of the human-computer interaction system based on the large language model. In addition, the mapping of text operation instructions to control actions equivalent to manual operations of the human-computer interaction software can be achieved only through the software middleware, making development and debugging simpler. Moreover, since the large language model outputs text operation instructions, which conforms to the reading habits of human operators, the accuracy of the text operation instructions output by the large language model can be intuitively judged, thereby improving the control accuracy of the control instructions.

[0014] Furthermore, the text operation instructions are understandable to ordinary operators. Compared with the existing solution that requires personnel with professional programming skills to read the operation instructions, the requirements for personnel are reduced, and the applicability of the human-computer interaction system based on the large language model is further improved.

[0015] In summary, the present invention is simpler, more efficient and less costly than the traditional method of using a device-specific control interface or function library to form an executable program for a large language model. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0017] Figure 1 A schematic diagram of the structure of an embodiment of the human-computer interaction system based on a large language model provided by the present invention; Figure 2 A flowchart of an embodiment of the human-computer interaction method based on a large language model provided by the present invention. DETAILED DESCRIPTION

[0018] The following will provide a clear and complete description of the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0019] It should be understood that the schematic drawings are not drawn to scale. The flowcharts used in the present invention illustrate operations implemented according to some embodiments of the present invention. It should be understood that the operations of the flowcharts can be implemented out of sequence, and steps that have no logical contextual relationship can be reversed in order or implemented simultaneously. In addition, those skilled in the art, guided by the content of the present invention, can add one or more other operations to the flowcharts or remove one or more operations from the flowcharts. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in the form of software, or in one or more hardware modules or integrated circuits, or in different networks and / or processor systems and / or microcontroller systems.

[0020] References herein to "embodiments" mean that a particular feature, structure, or characteristic described in connection with the embodiments may be included in at least one embodiment of the present invention. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it constitute a separate or alternative embodiment that is mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0021] The present invention provides a human-computer interaction system and method based on a large language model, which are described below.

[0022] Figure 1A schematic diagram of the structure of an embodiment of the human-computer interaction system based on a large language model provided by the present invention is shown as follows: Figure 1 As shown, the human-computer interaction system 10 based on the large language model includes: an operation instruction generation module 100 with a built-in large language model and a controlled device 200, on which a software middleware 210 and a human-computer interaction software 220 are running; The operation instruction generation module 100 is used to process the user request information based on the built-in large language model and generate text operation instructions in a format that can be parsed by the software middleware; The software middleware 210 is used to map text operation instructions to generate control operations that perform equivalent manual operations on the human-computer interaction software 220.

[0023] Among them, the large language model can be any one or more of large models such as GPT, Deepseek, Qwen, LLaMA, etc.

[0024] It should be understood that the user request information is an operation instruction or functional requirement issued manually, and is used to instruct the controlled device 200 to complete a specified function or task, for example: "Complete login".

[0025] It should be noted that the user request information can be a text instruction directly input or an instruction in another mode, such as a voice instruction or an instruction issued by clicking an integrated button. When the user request information is an instruction in another mode, the human-computer interaction system 10 based on the large language model needs to be compatible with an external instruction conversion function module to convert the instruction in the other mode into a text instruction.

[0026] By setting user request information in multiple modes, the usability and applicability of the human-computer interaction system 10 based on the large language model can be improved.

[0027] In a specific embodiment of the present invention, the text operation instruction may be "click." After conversion by the software middleware 210, the generated control instruction is: convert it to a click action on the user interface of the human-computer interaction software 220. Similarly, the text operation instruction may be "input." After conversion by the software middleware 210, the generated control instruction is: convert it to an input action on the input box of the user interface of the human-computer interaction software 220.

[0028] In a specific embodiment of the present invention, the format of the text operation instruction is a format that can be recognized by the software middleware 210, and the format of the text operation instruction includes but is not limited to XML format, HTML format or JSON format.

[0029] Compared to the prior art, the human-computer interaction system 10 based on a large language model provided by the embodiment of the present invention can achieve decoupling of business and operations by setting up a software middleware 210 running on the controlled device 200. In other words, the large language model only needs to generate text operation instructions in a format that can be parsed by the software middleware. There is no need to specifically compile a function library for the interface and functions of the human-computer interaction software, nor is there a need for human operators to compile training text input for the functions implemented by the function library and the use of the function library for the large language model based on the function library. This greatly reduces the development workload of the human-computer interaction system based on the large language model. The text operation instructions output by the large model can be mapped to the control actions of the human-computer interaction software that are equivalent to manual operations only through the software middleware 210, making development and debugging simpler. In addition, since the large language model outputs text operation instructions, which conform to the reading habits of human operators, the accuracy of the text operation instructions can be intuitively judged, thereby improving the control accuracy of the control instructions.

[0030] Furthermore, the text operation instructions are understandable to ordinary operators. Compared with the existing solution that requires personnel with professional programming skills to read the operation instructions, the requirements for personnel are reduced, and the applicability of the human-computer interaction system based on the large language model is further improved.

[0031] In summary, the embodiments of the present invention are simpler, more efficient and less costly than the conventional method of using a device-specific control interface or function library to form an executable program for a large language model.

[0032] In order to realize the response of the human-computer interaction software to the control instruction, in some embodiments of the present invention, such as Figure 1 As shown, the controlled device 200 also includes an operating system 230, which provides a control mechanism that triggers the response of the human-computer interaction software 220. The control mechanism includes but is not limited to an interface control interface, an operation library, a function, a message or an event triggering mechanism. The control instructions can call these control mechanisms to simulate the actions of manually operating the user interface.

[0033] The software middleware 210 and the human-computer interaction software 220 are both application software running on the operating system 230 .

[0034] It should be noted that: by calling the control mechanism provided by the operating system, the operating system and human-computer interaction software can be triggered to respond to various basic actions equivalent to external input, such as clicking, inputting, moving, dragging and other external input actions, thereby calling the control mechanism to produce action responses equivalent to manual operations such as mouse, keyboard and hand touch.

[0035] By providing an operating system 230, the embodiment of the present invention can further simplify the development complexity of the software middleware 220. Specifically, the software middleware 220 only needs to adapt to different operating systems 230, not to adapt to every type of human-computer interaction software. Compared to the vast number of device types, there are only a few types of operating systems. The software middleware only needs to be developed once for a limited number of operating systems, and the development content only needs to focus on basic actions such as movement, clicking, and input, making development and debugging simpler. This further improves the convenience of the human-computer interaction system 10 based on the large language model.

[0036] In some embodiments of the present invention, the human-computer interaction software 220 includes a user interface; The user interface is used to respond to manual operations on the user interface and display the results of the operations.

[0037] It should be noted that: in addition to the user interface, the human-computer interaction software may also include business logic processing, which is used to parse control instructions and generate logical processing results corresponding to the control instructions.

[0038] It should also be noted that: in some scenarios, there is no need to display the results of the operation, and it is only necessary to provide feedback on the operation results through the execution status of the controlled device, etc. Therefore, in some special embodiments, the human-computer interaction software 220 may not include a user interface, but only include a business logic processing layer.

[0039] Since the accuracy of the text operation instructions output by the large language model depends on the training results of the large language model, in some embodiments of the present invention, such as Figure 2 As shown, the operation instruction generation module 100 includes a model training unit 110 and a model use unit 120; The model training unit 110 is used to train the initial large language model based on the relevant usage documents of the human-computer interaction software to obtain a trained large language model; The model using unit 120 is used to process the user request information based on the trained large language model and generate text operation instructions.

[0040] It should be noted that: after the training of the model training unit 110 is completed, the model training unit 110 does not participate in the subsequent use process of the model usage unit 120. In other words, the model training unit 110 and the model usage unit 120 do not work at the same time. The model usage unit 120 can only be used after the training of the model training unit 110 is completed.

[0041] Specifically, when the large language model needs to interactively control a new controlled device 200 , the model training unit 110 needs to work. After the training is completed, the model using unit 120 can perform human-computer interactive control on the new controlled device 200 .

[0042] In a specific embodiment of the present invention, the relevant usage documents include but are not limited to at least one of the software operation-related files provided to human operators, such as the user manual, function documents and operation documents of the human-computer interaction software.

[0043] Since the large language model of the embodiment of the present invention only needs to output text operation instructions, which conforms to human operating language, the human-computer interaction system 10 based on the large language model of the embodiment of the present invention can use operation documents to train the large language model, give full play to the role of operation documents, and further improve the accuracy and reliability of the large language model obtained through training.

[0044] To ensure the accuracy of control of the human-computer interaction software 220, in a specific embodiment of the present invention, the user interface includes interface pages and interface controls on the interface pages, and the relevant usage documents include the functions of the interface controls and the content of the operation description of the interface controls.

[0045] In a specific embodiment of the present invention, the relevant usage document may also include position parameters of the interface controls, which can accurately obtain the interface control positions and specific operation information, thereby ensuring the accuracy and reliability of the human-computer interaction control of the human-computer interaction software 220.

[0046] In a specific embodiment of the present invention, the trained AI language model may output a series of user login function instructions in the following json format text: { "Time": "t0", "Page": "main interface", "Control": "Login button", "Position": { "X axis": 1300, "Y axis": 150 }, "Action": "click", "Content": "null" }, { "Time": "t1", "Page": "user login", "Control": "name input box", "Position": { "X axis": 850, "Y axis": 400 }, "Action": "input", "Content": "user" }, { "Time": "t2", "Page": "user login", "Control": "password input box", "Position": { "X axis": 830, "Y axis": 400 }, "Action": "input", "Content": "ctrl0!" }, { "Time": "t3", "Page": "user login", "Control": "OK button", "Position": { "X axis": 1000, "Y axis": 700 }, "Action": "click", "Content": "null" } After the above instructions are mapped by the software middleware, the operating system's operating mechanism is called, which is equivalent to a human operator performing the following steps: 1. Click the login button on the main interface 2. On the pop-up user login page, select the username input box and enter the username 3. On the user login page, select the password input box and enter the login password 4. On the user login page, click the OK button to log in.

[0047] Since the accurate output of the control instruction depends on the accuracy of the text operation instruction, in some embodiments of the present invention, such as Figure 1As shown, the human-computer interaction system 10 based on the large language model also includes an output observation module 300, which is used to display and record the text operation instructions output by the large model, and feed back the text operation instructions to the human operator to confirm the correctness or compare them with the theoretical operation instructions to determine whether the text operation instructions are accurate.

[0048] The theoretical operation instructions are the correct operation instruction flow recorded in the operation documents of the human-computer interaction software.

[0049] After obtaining the text operation instructions, the embodiment of the present invention records them and compares them with the theoretical operation instructions or feeds the text operation instructions back to the human operator for inspection, thereby ensuring the accuracy of the generated text operation instructions and further improving the accuracy of equivalent manual operations on the human-computer interaction software.

[0050] In summary, the human-computer interaction system based on a large language model proposed in the embodiments of the present invention uses software middleware to convert textual operation instructions output by the large language model into control actions. Applying the large language model to the human-computer interaction process of a device can assist or even replace human operators in performing user interface operations on the device. Compared to traditional manual operation, the large language model learns much faster than humans. It can learn a million-word operating manual in minutes, while humans would need days or weeks. Furthermore, the large language model can learn online 24 / 7, significantly reducing the learning time required by humans. Furthermore, once the large language model is learned and trained, it can be directly deployed as software at a very low cost, eliminating the time and cost of training human operators. Furthermore, the knowledge and content learned by the large language model is not lost or forgotten. Finally, the large language model running on a computer can maintain high-standard operation throughout the day. Even in harsh environments and under special circumstances, this approach can maintain more stable and efficient operations than human operators. In addition, compared with the traditional method of using a device-specific control interface or function library for a large language model to understand, call and form an executable program, the embodiment of the present invention is simpler, more efficient and lower cost.

[0051] Correspondingly, the embodiment of the present invention further provides a human-computer interaction method based on a large language model, which is applicable to the human-computer interaction system based on a large language model in any of the above embodiments, such as Figure 2 As shown in Figure 2, the human-computer interaction method based on the large language model includes: S201: Processing user request information based on a large language model to generate text operation instructions in a format that can be parsed by the software middleware; S202: Map the text operation instructions based on the software middleware to generate control actions that perform equivalent manual operations on the human-computer interaction software.

[0052] The specific implementation process and details of each step of the human-computer interaction method based on a large language model provided in the above embodiment can be found in the technical solution described in the above embodiment of the human-computer interaction system based on a large language model, and will not be repeated here.

[0053] The above is a detailed introduction to a human-computer interaction system and method based on a large language model provided by the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for technical personnel in this field, based on the ideas of the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A human-computer interaction system based on a large language model, characterized in that: include: An operation instruction generation module with a built-in large language model and a controlled device, wherein the controlled device runs software middleware and human-computer interaction software; The operation instruction generation module is used to process the user request information based on the built-in large language model and generate text operation instructions in a format that can be parsed by the software middleware; The software middleware is used to map the text operation instructions to generate control actions that perform equivalent manual operations on the human-computer interaction software.

2. The human-computer interaction system based on a large language model according to claim 1, characterized in that: The controlled device also includes an operating system, which provides a control mechanism that triggers the human-computer interaction software to respond. The control mechanism includes an interface control interface, an operation library, a function, a message or event triggering mechanism. The control action can call the control mechanism to simulate the action of manually operating the user interface.

3. The human-computer interaction system based on a large language model according to claim 2, characterized in that: The human-computer interaction software includes a user interface; The user interface is used to respond to manual operations and display the results of the operations.

4. The human-computer interaction system based on a large language model according to claim 3, characterized in that: The operation instruction generation module includes a model training unit and a model use unit; The model training unit is used to train the initial large language model based on the relevant usage documents of the human-computer interaction software to obtain the trained large language model; The model using unit is used to process the user request information based on the trained large language model to generate text operation instructions in a format that can be parsed by software middleware.

5. The human-computer interaction system based on a large language model according to claim 4, characterized in that: The user interface includes interface pages and interface controls on the pages, and the related usage documents include functions of the interface controls and operation descriptions of the interface controls.

6. The human-computer interaction system based on a large language model according to claim 1, characterized in that: The system further includes an output observation module, which is used to record and feedback the text operation instructions, and compare the text operation instructions with theoretical operation instructions or feedback the text operation instructions to a human operator to determine whether the text operation instructions are accurate.

7. A human-computer interaction method based on a large language model, characterized in that: The human-computer interaction system based on a large language model according to any one of claims 1 to 6, wherein the method comprises: Process user request information based on a large language model and generate text operation instructions in a format that can be parsed by the software middleware; The text operation instructions are mapped based on the software middleware to generate control actions that perform equivalent manual operations on the human-computer interaction software.

Images