Interaction control method and system based on large language model assistance

Abstract

The invention belongs to the technical field of interaction control, and discloses an interaction control method and system based on large language model assistance, and the method comprises the steps: obtaining image information of an interaction interface of an operation object; performing identification and text conversion on the image information of the interactive interface to obtain natural language description of the interactive interface; constructing an interactive operation command set of the interactive interface based on the natural language description; acquiring a natural language of the user, and performing intention recognition on the natural language of the user based on the large language model to obtain an operation intention of the user; and matching an interactive operation command from the interactive operation command set of the interactive interface according to the operation intention of the user, and controlling the operation tool to execute interactive operation on the operation object by using the interactive operation command. According to the invention, manual operation is simplified, convenience of interactive operation is improved, and popularization and application to client groups with difficult operation are facilitated.

Description

Technical Field

[0001] The present invention belongs to the technical field of interactive control, and in particular relates to an interactive control method and system based on the assistance of a large language model. Background Art

[0002] Human-Machine Interaction (HMI) refers to the interaction and communication between humans and computer systems or machines, involving multiple aspects such as user interface, user experience, and the way humans and machines interact.

[0003] With the continuous breakthroughs in computer science, artificial intelligence and sensor technology, the field of human-computer interaction control has evolved from simple command line interfaces to complex multi-module state interactions. In this context, human-computer interaction control technology is increasingly becoming an indispensable part of people's daily life and work. It not only changes the way we interact with machines, but also greatly improves operational efficiency and user experience.

[0004] At present, most human-computer interaction controls still require user operation, such as inputting control instructions through the mouse and keyboard, and some users often cannot complete the interactive operation smoothly when they have difficulty in operation. With the development of natural language, computer vision and other technologies, large language models can handle a variety of natural language tasks, such as text classification, question answering, dialogue, etc., and computer vision technology can complete image recognition tasks.

[0005] However, in the prior art, natural language recognition technology and visual technology have not been applied to human-computer interaction control to reduce the operational difficulties of users. Summary of the invention

[0006] The purpose of the present invention is to provide an interactive control method and system based on the assistance of a large language model, so as to solve the problem that the prior art fails to apply natural language recognition technology and visual technology to human-computer interactive control.

[0007] In order to achieve the above object, the present invention adopts the following technical solutions: In a first aspect, the present invention provides an interactive control method based on the assistance of a large language model, the method comprising: Obtain image information of the interactive interface of the operation object; Recognize and convert the image information of the interactive interface into text to obtain a natural language description of the interactive interface; An interactive operation command set for building an interactive interface based on natural language description; Obtain the user's natural language, identify the user's intention based on the large language model, and obtain the user's operation intention; An interactive operation command is matched from an interactive operation command set of the interactive interface according to the user's operation intention, and the interactive operation command is used to control the operation tool to perform the interactive operation on the operation object.

[0008] Preferably, the image information of the interactive interface is recognized and converted into text to obtain a natural language description of the interactive interface, including: Based on the machine vision target detection algorithm, feature extraction is performed on the image information of the interactive interface to obtain regional features and grid features, and a first position code corresponding to the regional features and a second position code corresponding to the grid features are extracted; An image description model is constructed based on the self-attention mechanism, and the regional features and the first position code and the grid features and the second position code are used as inputs of the image description model. The image description model outputs a text description corresponding to the image information of the interactive interface, and the text description corresponding to the image information of the interactive interface is used as the natural language description of the interactive interface.

[0009] Preferably, the image description model comprises: two encoding units and one decoding unit; One of the encoding units is used to take the region feature and the first position code as input, encode the region feature and the first position code based on the self-attention mechanism, and output the region feature code; Another encoding unit is used for taking the grid feature and the second position code as input, encoding the grid feature and the second position code based on the self-attention mechanism, and outputting the grid feature code; The decoding unit is used to take the region feature code and the grid feature code as input, perform fusion decoding on the region feature code and the grid feature code, and output a text description corresponding to the image information of the interactive interface.

[0010] Preferably, the first encoding unit includes a first attention module, a first feedforward network module, a second attention module and a second feedforward network module connected in sequence; the second encoding unit includes a third attention module, a third feedforward network module, a fourth attention module and a fourth feedforward network module connected in sequence.

[0011] Preferably, the method further comprises: encoding the region feature and the first position code and the grid feature and the second position code respectively based on two coding units to obtain the region feature code and the grid feature code, including: Constructing a triple matrix of a first attention module based on the region feature and the first position encoding, and constructing a triple matrix of a third attention module based on the grid feature and the second position encoding; Calculating the attention output of the first attention module based on the triplet matrix of the first attention module, taking the attention output of the first attention module as the input of the first feedforward network module, and the first feedforward network outputs a first feature output; Calculating the attention output of the third attention module based on the triplet matrix of the third attention module, using the attention output of the third attention module as the input of the third feedforward network module, and the third feedforward network outputs a second feature output; Integrate the second feature output into the first feature output, and construct a triplet matrix of the second attention module based on the first feature output integrated with the second feature output; Calculating the attention output of the second attention module based on the triplet matrix of the second attention module, using the attention output of the second attention module as the input of the second feedforward network, and the second feedforward network outputs the regional feature code; Integrate the first feature output into the second feature output, and construct a triplet matrix of a fourth attention module based on the second feature output integrated into the first feature output; The attention output of the fourth attention module is calculated based on the triplet matrix of the fourth attention module, and the attention output of the fourth attention module is used as the input of the fourth feedforward network, and the fourth feedforward network outputs the grid feature code.

[0012] Preferably, the triple matrix of the first attention module, the triple matrix of the second attention module, the triple matrix of the third attention module and the triple matrix of the fourth attention module all include: a query matrix, a key matrix and a value matrix; the first attention module, the second attention module, the third attention module and the fourth attention module all use the same calculation method when calculating their respective attention outputs, and the calculation method includes: Input the query matrix, key matrix and value matrix into the attention function to get the output matrix; Calculate the similarity between the output matrix and the query matrix to obtain a similarity matrix; Performing a linear transformation on the similarity matrix to obtain a first linear matrix; Performing a nonlinear transformation on the first linear matrix to obtain a nonlinear matrix; Perform a linear transformation on the output rectangle to obtain a second linear matrix; Multiply the nonlinear matrix with the second linear matrix to obtain the attention output matrix.

[0013] Preferably, the interactive operation command set for constructing the interactive interface based on the natural language description includes: Collect the elements in the interactive interface and the interactive commands corresponding to each element; Define the structure of interactive commands; Extract text features from natural language descriptions; The text features in the natural language description are mapped to the structure of the interactive command to obtain the interactive operation command set of the interactive interface.

[0014] Preferably, the method also includes: using text features in the natural language description as sample input and the interactive operation command set of the interactive interface as labels to train the neural network model to obtain a trained neural network model, and using the trained neural network model as an operation instruction prediction model.

[0015] Preferably, matching the interactive operation command from the interactive operation command set of the interactive interface according to the user's operation intention includes: Extracting intention features of user's operation intention; The intention features are input into the operation instruction prediction model to obtain the interactive operation commands.

[0016] In a second aspect, the present invention provides an interactive control system assisted by a large language model, the system is used to implement the above-mentioned interactive control method assisted by a large language model, and the system includes: A data acquisition module is used to acquire image information of the interactive interface of the operation object; The text conversion module is used to recognize and convert the image information of the interactive interface into text to obtain a natural language description of the interactive interface; The instruction building module is used to build an interactive operation command set of the interactive interface based on natural language description; The intention recognition module is used to obtain the user's natural language, perform intention recognition on the user's natural language based on the large language model, and obtain the user's operation intention; The instruction generation module is used to match the interactive operation command from the interactive operation command set of the interactive interface according to the user's operation intention, and control the operation tool to perform the interactive operation on the operation object with the interactive operation command.

[0017] Beneficial effects: The present invention can obtain image information of the operation object, convert the image information of the operation object into a natural language description, then establish a correspondence between the natural language description of the operation object and the interactive operation instruction, and then collect the user's natural language (for example: voice data), use a large language model to identify the user's intention, determine the user's operation intention, and then determine the corresponding interactive operation command according to the operation intention. The elements in the interactive interface can be controlled through the interactive operation instructions, which simplifies manual operation, improves the convenience of interactive operation, and is conducive to promoting application to customer groups who have difficulty in operation. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The accompanying drawings are used to provide a further understanding of the embodiments of the present invention and constitute a part of the specification. Together with the following specific implementations, they are used to explain the embodiments of the present invention, but do not constitute a limitation on the embodiments of the present invention. In the accompanying drawings: Figure 1 is a flow chart of an interactive control method based on large language model assistance provided by an embodiment of the present invention; Figure 2 It is a block diagram of an interactive control system based on the assistance of a large language model provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the present invention will be briefly introduced below in combination with the drawings and the description of the embodiments or the prior art. Obviously, the following description of the structure of the drawings is only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. It should be noted that the description of these embodiments is used to help understand the present invention, but does not constitute a limitation of the present invention.

[0020] Embodiment 1 Figure 1 is a flow chart of an interactive control method based on large language model assistance provided by an embodiment of the present invention, such as Figure 1 As shown, this embodiment provides an interactive control method based on the assistance of a large language model, and the method includes: Step S10: Acquire image information of the interactive interface of the operation object.

[0021] In this embodiment, the operation object can be a smart device such as a PC, and the interactive interface is the monitor of the PC; the operation object can also be an instrument and equipment operation panel (such as an elevator panel, etc.), and the image information is collected through a camera. For example, the display picture on the PC monitor and the operation panel picture of the instrument and equipment can be collected through the camera.

[0022] Step S20: Recognize and convert the image information of the interactive interface into text to obtain a natural language description of the interactive interface.

[0023] In this embodiment, after the camera captures image information such as display images and operation panel images, it is necessary to convert the elements in the corresponding interactive interface in the image information, such as buttons, icons, text boxes and other elements, into natural language descriptions. By subsequently establishing mapping rules between natural language descriptions and operating instructions, automatic control of the interactive interface can be achieved through natural language.

[0024] Therefore, in order to improve the accuracy and efficiency of converting image data into natural language data, specifically, the image information of the interactive interface is recognized and converted into text to obtain a natural language description of the interactive interface, including: Step S201: extracting features of the image information of the interactive interface based on a machine vision target detection algorithm to obtain region features and grid features, and extracting first position codes corresponding to the region features and second position codes corresponding to the grid features.

[0025] In this embodiment, the machine vision target detection algorithm can be Faster RCNN (Fast Regional Convolutional Neural Network) and YOLO target detection algorithm. In this embodiment, the regional features are preferably extracted by Faster RCNN, and the grid features are preferably extracted by YOLO target detection algorithm, wherein the regional features have a relatively high semantic level, and the grid features can cover the entire image, can provide background information, and also include the details of the target.

[0026] Step S202: construct an image description model based on the self-attention mechanism, and use the regional features and the first position code and the grid features and the second position code as inputs of the image description model. The image description model outputs a text description corresponding to the image information of the interactive interface, and uses the text description corresponding to the image information of the interactive interface as the natural language description of the interactive interface.

[0027] In this embodiment, the image description model includes: two encoding units and one decoding unit; One of the encoding units is used to take the region feature and the first position code as input, encode the region feature and the first position code based on the self-attention mechanism, and output the region feature code; Another encoding unit is used for taking the grid feature and the second position code as input, encoding the grid feature and the second position code based on the self-attention mechanism, and outputting the grid feature code; The decoding unit is used to take the region feature code and the grid feature code as input, perform fusion decoding on the region feature code and the grid feature code, and output a text description corresponding to the image information of the interactive interface.

[0028] In this embodiment, the self-attention mechanism can calculate the weights of the input data through mutual interaction to achieve the effect of focusing on key data. At the same time, it can also integrate regional features and grid features to achieve complementary advantages of the two features.

[0029] There are still the following problems in the application of the self-attention mechanism: the self-attention mechanism involves the superposition of multiple self-attention layers, and the distribution of input data is bound to affect the high-level data, causing the high-level input data distribution to change dramatically, affecting the effect of feature fusion and the update of model parameters; and the self-attention calculation represents the first-order interaction between the query and the key, without utilizing higher-order interaction information, so the obtained feature representation ability is not high, which limits the multi-module state reasoning ability in the image description task, and ultimately reduces the quality of the model's description text generation.

[0030] In order to solve the above problems, the first encoding unit includes a first attention module, a first feedforward network module, a second attention module and a second feedforward network module connected in sequence; the second encoding unit includes a third attention module, a third feedforward network module, a fourth attention module and a fourth feedforward network module connected in sequence.

[0031] As a further optimization of this embodiment, the method further includes: encoding the region feature and the first position code and the grid feature and the second position code respectively based on two coding units to obtain the region feature code and the grid feature code, including: Step a10: construct a triplet matrix of the first attention module based on the region feature and the first position encoding, and construct a triplet matrix of the third attention module based on the grid feature and the second position encoding.

[0032] In this embodiment, the first position code is fused with the region feature to obtain a fused region feature; similarly, the second position code is fused with the grid feature to obtain a fused grid feature.

[0033] All three groups of matrices in this embodiment include: a query matrix Q, a key matrix K and a value matrix V, and the general calculation formula for constructing a triple matrix is: ; In the formula, X is the input matrix, which is the fused regional features and the fused grid features. is the first weight matrix, is the second weight matrix, is the third weight matrix, and the first weight matrix, the second weight matrix and the third weight matrix can all be obtained through learning.

[0034] Step a20: Calculate the attention output of the first attention module based on the triplet matrix of the first attention module, and use the attention output of the first attention module as the input of the first feedforward network module, and the first feedforward network outputs the first feature output.

[0035] Step a30: Calculate the attention output of the third attention module based on the triplet matrix of the third attention module, and use the attention output of the third attention module as the input of the third feedforward network module, and the third feedforward network outputs the second feature output.

[0036] In this embodiment, the first attention module and the third attention module mainly process the correlation between elements in the input sequence, and the output is a weighted representation; after the attention output of the first attention module is input into the first feedforward network module, and the attention output of the third attention module is input into the third feedforward network module; the first feedforward network module and the third feedforward network module adopt a feedforward neural network, and the feedforward neural network increases the nonlinear ability of the model by introducing a nonlinear activation function (such as ReLU or GeLU), so that the model can capture more complex features and patterns.

[0037] Step a40: Integrate the second feature output into the first feature output, and construct a triplet matrix of the second attention module based on the first feature output integrated into the second feature output.

[0038] Step a50: Calculate the attention output of the second attention module based on the triplet matrix of the second attention module, and use the attention output of the second attention module as the input of the second feedforward network, and the second feedforward network outputs the regional feature code.

[0039] In this embodiment, the first feature output is integrated into the second feature output, that is, the grid feature is integrated into the regional feature, so that the regional feature can integrate the advantages of the grid feature.

[0040] Step a60: Integrate the first feature output into the second feature output, and construct a triplet matrix of the fourth attention module based on the second feature output integrated into the first feature output.

[0041] Step a70: Calculate the attention output of the fourth attention module based on the triplet matrix of the fourth attention module, and use the attention output of the fourth attention module as the input of the fourth feedforward network, and the fourth feedforward network outputs the grid feature code.

[0042] In this embodiment, the second feature output is integrated into the first feature output, that is, the regional feature is integrated into the grid feature, so that the grid feature can integrate the advantages of the regional feature.

[0043] In this embodiment, in the process of encoding grid features and regional features for the two encoding units, the regional features are integrated into the encoding process of grid features, and the grid features are integrated into the encoding process of regional features at the same time; the two types of features respectively bring out the advantages of each other to achieve the purpose of complementary advantages; at the same time, the grid feature encoding and the regional feature encoding are combined as the input of the decoding unit, which can make the grid features and the regional features deeply integrated. Since the interaction between features can be utilized to enhance the representation ability between features, the quality of the natural language description output by the decoding unit can be improved.

[0044] As a further optimization of this embodiment, the triple matrix of the first attention module, the triple matrix of the second attention module, the triple matrix of the third attention module and the triple matrix of the fourth attention module all include: a query matrix, a key matrix and a value matrix; since each query Q of the attention module outputs a weighted average, regardless of whether Q and K or V are related, when there is no correlation between them, the attention module will still generate a weighted average vector, which will mislead the results and thus affect the quality of the generated description text.

[0045] Therefore, the first attention module, the second attention module, the third attention module and the fourth attention module all use the same calculation method when calculating their respective attention outputs, and the calculation method includes: Step b10: Input the query matrix, key matrix and value matrix into the attention function to obtain the output matrix; Step b20: Calculate the similarity between the output matrix and the query matrix to obtain a similarity matrix; Step b30: performing a linear transformation on the similarity matrix to obtain a first linear matrix; Among them, the calculation expression of the first linear matrix is: ; In the formula, The first linear matrix, is the first linear transformation weight, is the cosine similarity function, and Y is the output matrix.

[0046] Step b40: performing a nonlinear transformation on the first linear matrix to obtain a nonlinear matrix; Step b50: performing a linear transformation on the output rectangle to obtain a second linear matrix; Step b60: Multiply the nonlinear matrix by the second linear matrix to obtain the attention output matrix.

[0047] The calculation expression of the attention output matrix is: ; In the formula, is the attention output matrix, is the second linear transformation weight, is a nonlinear transformation function.

[0048] In the process of each attention output, the cosine similarity function can be used to calculate the cosine value between the output matrix and the query matrix. The cosine value can represent the similarity. When the cosine value is close to 1, it means that the output matrix is ​​more similar to the query matrix. When the cosine value is close to 0, it means that there is no correlation between the output matrix and the query matrix. After the first linear matrix undergoes a nonlinear change (using the ReLU or GeLU function for nonlinear change), if the cosine value is close to 0, there is no correlation between the two features, and the corresponding weights can be cleared to better enhance the reasoning ability and further improve the quality of image-generated text.

[0049] Step S30: constructing an interactive operation command set of the interactive interface based on the natural language description.

[0050] Specifically, the interactive operation command set for constructing the interactive interface based on natural language description includes: Step S301: Collecting elements in the interactive interface and the interactive commands corresponding to each element; Step S302: define the structure of the interactive command; Step S303: extracting text features in the natural language description; Step S304: Map the text features in the natural language description to the structure of the interactive command to obtain an interactive operation command set of the interactive interface.

[0051] In this embodiment, a data set containing a large number of interactive interface elements and their corresponding interactive commands is collected or created to ensure that the natural language description of each interactive interface element and the corresponding interactive command are accurately labeled, extract features such as color, shape, position, etc. from the natural language description, and define the structure of the interactive command, such as verbs (click, slide), nouns (buttons, sliders), etc.

[0052] Define a set of grammar rules to map text features to the structure of interactive commands. For example, a "red round button" might be mapped to a "click" action and a "red round" description, as well as ensuring that the semantics of the natural language description matches the semantics of the interactive command, for example, "open" is associated with a "click" action.

[0053] In this embodiment, after obtaining the interactive operation command set of the interactive interface, the text features in the natural language description are used as sample input, and the interactive operation command set of the interactive interface is used as a label to train the neural network model to obtain a trained neural network model, and the trained neural network model is used as an operation instruction prediction model; therefore, the user's actual operation intention can be input into the operation instruction prediction model, and the operation instruction prediction model can predict the corresponding interactive operation instruction according to the user's operation intention.

[0054] Step S40: Acquire the user's natural language, perform intent recognition on the user's natural language based on the large language model, and obtain the user's operation intention.

[0055] In this embodiment, the natural language used may be the user's voice, and the large language model obtains the user's operation intention by recognizing the user's speech.

[0056] Step S50: matching an interactive operation command from the interactive operation command set of the interactive interface according to the user's operation intention, and controlling the operation tool to perform an interactive operation on the operation object with the interactive operation command.

[0057] In this embodiment, matching the interactive operation command from the interactive operation command set of the interactive interface according to the user's operation intention includes: Step S501: extracting intention features of the user's operation intention; Step S502: Input the intention features into the operation instruction prediction model to obtain the interactive operation command.

[0058] In this embodiment, the operating tool can be a mechanical arm and a command output interface, the command output interface can be a USB interface, and the interactive operation command can be used as an input signal of a PC (connected to the PC by the command output interface) to control the PC to perform corresponding operations. Secondly, the interactive operation command of this embodiment can also be used as an input signal of other control devices, such as a mechanical arm, and the interactive control command is used as a control signal of the mechanical arm, and the operating tool such as the mechanical arm is used to replace the user to complete the operation of the operation panel of the instrument.

[0059] The present invention can obtain image information of the operation object, convert the image information of the operation object into a natural language description, then establish a correspondence between the natural language description of the operation object and the interactive operation instruction, and then collect the user's natural language (for example: voice data), use a large language model to identify the user's intention, determine the user's operation intention, and then determine the corresponding interactive operation command according to the operation intention. The elements in the interactive interface can be controlled through the interactive operation instructions, which simplifies manual operation, improves the convenience of interactive operation, and is conducive to promoting application to customer groups who have difficulty in operation.

[0060] Embodiment 2 Figure 2 is a block diagram of an interactive control system based on a large language model assisted by an embodiment of the present invention, such as Figure 2 As shown, this embodiment provides an interactive control system assisted by a large language model, the system is used to implement the interactive control method assisted by a large language model of embodiment 1, and the system includes: A data acquisition module is used to acquire image information of the interactive interface of the operation object; The text conversion module is used to recognize and convert the image information of the interactive interface into text to obtain a natural language description of the interactive interface; The instruction building module is used to build an interactive operation command set of the interactive interface based on natural language description; The intention recognition module is used to obtain the user's natural language, perform intention recognition on the user's natural language based on the large language model, and obtain the user's operation intention; The instruction generation module is used to match the interactive operation command from the interactive operation command set of the interactive interface according to the user's operation intention, and control the operation tool to perform the interactive operation on the operation object with the interactive operation command.

[0061] The present invention can obtain image information of the operation object, convert the image information of the operation object into a natural language description, then establish a correspondence between the natural language description of the operation object and the interactive operation instruction, and then collect the user's natural language (for example: voice data), use a large language model to identify the user's intention, determine the user's operation intention, and then determine the corresponding interactive operation command according to the operation intention. The elements in the interactive interface can be controlled through the interactive operation instructions, which simplifies manual operation, improves the convenience of interactive operation, and is conducive to promoting application to customer groups who have difficulty in operation.

[0062] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0063] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0064] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.

Claims

1. An interactive control method based on a large language model, characterized in that: The method comprises: Obtain image information of the interactive interface of the operation object; Recognize and convert the image information of the interactive interface into text to obtain a natural language description of the interactive interface; An interactive operation command set for building an interactive interface based on natural language description; Obtain the user's natural language, identify the user's intention based on the large language model, and obtain the user's operation intention; An interactive operation command is matched from an interactive operation command set of the interactive interface according to the user's operation intention, and the interactive operation command is used to control the operation tool to perform the interactive operation on the operation object.

2. The interactive control method based on the assistance of a large language model according to claim 1, characterized in that: Recognize and convert the image information of the interactive interface into text to obtain a natural language description of the interactive interface, including: Based on the machine vision target detection algorithm, feature extraction is performed on the image information of the interactive interface to obtain regional features and grid features, and a first position code corresponding to the regional features and a second position code corresponding to the grid features are extracted; An image description model is constructed based on the self-attention mechanism, and the regional features and the first position code and the grid features and the second position code are used as inputs of the image description model. The image description model outputs a text description corresponding to the image information of the interactive interface, and the text description corresponding to the image information of the interactive interface is used as the natural language description of the interactive interface.

3. The interactive control method based on the assistance of a large language model according to claim 2, characterized in that: The image description model includes: two encoding units and a decoding unit; One of the encoding units is used to take the region feature and the first position code as input, encode the region feature and the first position code based on the self-attention mechanism, and output the region feature code; Another encoding unit is used for taking the grid feature and the second position code as input, encoding the grid feature and the second position code based on the self-attention mechanism, and outputting the grid feature code; The decoding unit is used to take the region feature code and the grid feature code as input, perform fusion decoding on the region feature code and the grid feature code, and output a text description corresponding to the image information of the interactive interface.

4. The interactive control method based on the assistance of a large language model according to claim 3 is characterized in that: The first encoding unit includes a first attention module, a first feedforward network module, a second attention module and a second feedforward network module connected in sequence; the second encoding unit includes a third attention module, a third feedforward network module, a fourth attention module and a fourth feedforward network module connected in sequence.

5. The interactive control method based on the assistance of a large language model according to claim 4 is characterized in that: The method further includes: encoding the region feature and the first position code and the grid feature and the second position code respectively based on two coding units to obtain the region feature code and the grid feature code, including: Constructing a triple matrix of a first attention module based on the region feature and the first position encoding, and constructing a triple matrix of a third attention module based on the grid feature and the second position encoding; Calculating the attention output of the first attention module based on the triplet matrix of the first attention module, taking the attention output of the first attention module as the input of the first feedforward network module, and the first feedforward network outputs a first feature output; Calculating the attention output of the third attention module based on the triplet matrix of the third attention module, using the attention output of the third attention module as the input of the third feedforward network module, and the third feedforward network outputs a second feature output; Integrate the second feature output into the first feature output, and construct a triplet matrix of the second attention module based on the first feature output integrated with the second feature output; Calculating the attention output of the second attention module based on the triplet matrix of the second attention module, using the attention output of the second attention module as the input of the second feedforward network, and the second feedforward network outputs the regional feature code; Integrate the first feature output into the second feature output, and construct a triplet matrix of a fourth attention module based on the second feature output integrated into the first feature output; The attention output of the fourth attention module is calculated based on the triplet matrix of the fourth attention module, and the attention output of the fourth attention module is used as the input of the fourth feedforward network, and the fourth feedforward network outputs the grid feature code.

6. The interactive control method based on the assistance of a large language model according to claim 5, characterized in that: The triple matrix of the first attention module, the triple matrix of the second attention module, the triple matrix of the third attention module and the triple matrix of the fourth attention module all include: a query matrix, a key matrix and a value matrix; the first attention module, the second attention module, the third attention module and the fourth attention module all use the same calculation method when calculating their respective attention outputs, and the calculation method includes: Input the query matrix, key matrix and value matrix into the attention function to get the output matrix; Calculate the similarity between the output matrix and the query matrix to obtain a similarity matrix; Performing a linear transformation on the similarity matrix to obtain a first linear matrix; Performing a nonlinear transformation on the first linear matrix to obtain a nonlinear matrix; Perform a linear transformation on the output rectangle to obtain a second linear matrix; Multiply the nonlinear matrix with the second linear matrix to obtain the attention output matrix.

7. The interactive control method based on the assistance of a large language model according to claim 1, characterized in that: The interactive operation command set for building the interactive interface based on natural language description includes: Collect the elements in the interactive interface and the interactive commands corresponding to each element; Define the structure of interactive commands; Extract text features from natural language descriptions; The text features in the natural language description are mapped to the structure of the interactive command to obtain the interactive operation command set of the interactive interface.

8. The interactive control method based on the assistance of a large language model according to claim 7, characterized in that: The method also includes: using text features in natural language descriptions as sample inputs and interactive operation command sets of an interactive interface as labels to train a neural network model to obtain a trained neural network model, and using the trained neural network model as an operation instruction prediction model.

9. The interactive control method based on the assistance of a large language model according to claim 8, characterized in that: Matching interactive operation commands from the interactive operation command set of the interactive interface according to the user's operation intention includes: Extracting intention features of user's operation intention; The intention features are input into the operation instruction prediction model to obtain the interactive operation commands.

10. An interactive control system assisted by a large language model, the system being used to implement the interactive control method assisted by a large language model as claimed in any one of claims 1 to 9, characterized in that: The system comprises: A data acquisition module is used to acquire image information of the interactive interface of the operation object; The text conversion module is used to recognize and convert the image information of the interactive interface into text to obtain a natural language description of the interactive interface; Instruction construction module, used to construct interactive operation command sets of interactive interfaces based on natural language descriptions; The intention recognition module is used to obtain the user's natural language, perform intention recognition on the user's natural language based on the large language model, and obtain the user's operation intention; The instruction generation module is used to match the interactive operation command from the interactive operation command set of the interactive interface according to the user's operation intention, and control the operation tool to perform the interactive operation on the operation object with the interactive operation command.

Images