Application program operation instruction processing system and method

Abstract

The invention discloses an application program operation instruction processing system and method, and relates to the technical field of application program operation instruction processing.The system comprises a client side and a server side, the client side is used for receiving natural language information such as characters and voice input by a user; the server side is used for querying the mapping relation table according to the natural language information, determining that the natural language information contains a natural language instruction and a function button identifier corresponding to the natural language instruction, executing corresponding operation according to the function button identifier, and sending the information for executing the corresponding operation to the client side; and executing and displaying by the client. According to the application, a user inputs natural language information such as characters and voices through the client and can directly query the mapping relation table through the server, so that the corresponding function button identifier is determined and the corresponding operation is immediately executed, the operation difficulty of an application program is reduced, the operation is simpler and more convenient, and the user experience is improved. And the simplicity and the operability of operating the application program are improved.

Description

Technical Field

[0001] The present application relates to the technical field of application program operation instruction processing, and in particular to an application program operation instruction processing system and method. Background Art

[0002] At present, applications such as APPs, mini-programs, and operations on the client or computer side generally click on the corresponding function button, or touch, slide, and click the screen to open the corresponding interface and perform the corresponding operation. For those who want to directly open or operate a function in a sub-interface under the main interface, multiple clicks are required to achieve the above operation. Although this operation is relatively conventional, for users to perform certain operations, especially to enter a certain function operation under a sub-interface, they still need to click multiple times through multiple interfaces and sub-interfaces to open it, especially when users operate unfamiliar new applications for the first time, and when middle-aged and elderly users who are not familiar with electronic products operate, it is still difficult and not very convenient. In addition, for new functions of new applications, users also need to explore and gradually learn to use the new functions in the new applications. There are generally problems of difficulty, inconvenience, and difficulty in operation, especially for the rapidly developing applications.

[0003] Based on this, how to improve the simplicity and ease of operating application programs and reduce the difficulty of operating application programs has become a technical problem that needs to be solved urgently in this field. Summary of the invention

[0004] The purpose of this application is to provide an application operation instruction processing system and method, which can reduce the operating difficulty of the application and improve the simplicity and ease of operation of the application.

[0005] To achieve the above objectives, this application provides the following solutions:

[0006] In a first aspect, the present application provides an application operation instruction processing system, the application operation instruction processing system comprising a client and a server, the client and the server are connected;

[0007] The client is used to receive natural language information input by a user and send the natural language information to the server; wherein the natural language information includes text information or voice information;

[0008] The server is provided with a mapping relationship table, wherein the mapping relationship between the natural language instruction and the corresponding function button identifier is preset in the mapping relationship table;

[0009] The server is used to query the mapping relationship table according to the natural language information, determine whether the natural language information contains the natural language instruction and the function button identifier corresponding to the natural language instruction, and perform the corresponding operation according to the function button identifier; and send the corresponding operation information to the client, which is executed and displayed by the client.

[0010] Optionally, the application operation instruction processing system further includes a large language model end, the large language model end is connected to the server end, and is connected to the client end through the server end;

[0011] The large language model end is used to obtain the natural language information input by the user in the client, and use the large language model to perform in-depth analysis on the natural language information to determine the user's true intention, generate a processing result, and send the processing result to the server end; the processing result is used by the server end to determine the function button identifier;

[0012] The server end is also used to determine the function button identifier according to the processing result generated by the large language model when the natural language instruction and its corresponding function button identifier cannot be found in the mapping relationship table.

[0013] Optionally, the client includes a user interface module, a network communication module and an instruction execution display module;

[0014] The user interface module is used to display a user interface to the user, interact with the user through the user interface, and receive the natural language information input by the user;

[0015] The network communication module is used to perform network communication and data interaction with the server side;

[0016] The instruction execution display module is used to display the specific operation process when executing the natural language instruction in real time.

[0017] Optionally, the client further includes an input processing module and an execution step recording module;

[0018] The input processing module is used to perform format conversion processing on the natural language information, converting the format of the natural language information into a data format suitable for reception by the server;

[0019] The execution step recording module includes an execution step recording unit and an execution step sending unit;

[0020] The execution step recording unit is used to record the specific operation steps when executing the natural language instruction;

[0021] The execution step sending unit is used to send the recorded specific operation steps when executing the natural language instruction to the instruction execution display module, so that the instruction execution display module can display the specific operation steps when executing the natural language instruction in real time.

[0022] Optionally, the server side includes an animation special effects module;

[0023] The animation special effects module is used to generate and play animation special effects; the animation special effects include animation special effects of pressing a button when executing the natural language instruction.

[0024] Optionally, the server side further includes an instruction preprocessing module, which includes a keyword extraction unit, a semantic analysis unit and a regular expression unit;

[0025] The keyword extraction unit is used to extract keywords from the natural language information;

[0026] The semantic analysis unit is used to perform semantic analysis on the natural language information according to the keywords to obtain a semantic analysis result;

[0027] The regular expression unit is used to perform format matching on the natural language information according to the semantic analysis result to obtain instructions in a specific format; the instructions in the specific format refer to instructions in a preset format that characterize key information and semantic information of the natural language information, and the preset format refers to a format that is the same as the format of each natural language instruction in the mapping relationship table.

[0028] Optionally, the server side further includes a mapping management module, and the mapping management module includes a mapping establishment unit, a mapping query unit and a mapping storage unit;

[0029] The mapping establishment unit is used to establish a mapping relationship between the natural language instruction and the function button identifier, and generate the mapping relationship table;

[0030] The mapping storage unit is used to store the mapping relationship table;

[0031] The mapping query unit is used to query a mapping relationship table according to the natural language information, and determine whether the natural language information contains the natural language instruction and a function button identifier corresponding to the natural language instruction.

[0032] Optionally, the server side also includes an API calling module, which is used to initiate an API calling request to the large language model side, and the API calling request is to request the large language model side to perform in-depth analysis of the natural language information and generate the processing result, and feed back the processing result to the server side.

[0033] Optionally, the natural language instruction includes an operation instruction, and the operation instruction includes a click instruction, an open instruction, a close instruction, a touch slide instruction, an up slide instruction, a down slide instruction, a left slide instruction, a right slide forward instruction, a return instruction, a back instruction and an exit instruction;

[0034] The natural language instruction also includes a client interface button name and / or a function button name following the operation instruction.

[0035] In a second aspect, the present application provides a method for processing an application operation instruction, wherein the execution subject of the method for processing an application operation instruction is a client, and the method for processing an application operation instruction includes:

[0036] Obtain the natural language information entered by the user on the client;

[0037] The natural language information is sent to a server; the server queries a mapping relationship table according to the natural language information, wherein the mapping relationship table is preset with a mapping relationship between a natural language instruction and the corresponding function button identifier, determines that the natural language information contains the natural language instruction and the function button identifier corresponding to the natural language instruction, and performs a corresponding operation according to the function button identifier; and sends the corresponding operation information to a client, which is executed and displayed by the client.

[0038] Optionally, the natural language information is sent to a server; the server queries a mapping relationship table according to the natural language information, wherein the mapping relationship table is preset with a mapping relationship between a natural language instruction and a corresponding function button identifier, and after determining that the natural language information contains the natural language instruction and the function button identifier corresponding to the natural language instruction, the application operation instruction processing method further includes:

[0039] When the server side queries the mapping relationship table according to the natural language information and fails to find that the natural language information contains the natural language instruction and the function button identifier corresponding to the natural language instruction, the natural language information is sent to the large language model side, and the large language model analyzes and summarizes and returns the result to the server side; the server side then queries the mapping relationship table according to the return result of the large language model to determine whether the return result of the large language model contains the natural language instruction and the function button identifier corresponding to the natural language instruction; if so, the corresponding operation is executed, and the corresponding operation information is sent to the client, which is executed and displayed by the client; if not, the return result of the large language model is directly sent to the client, and the client displays it to the user;

[0040] Or, the natural language information is sent to the server and to the large language model at the same time; when the server queries the mapping relationship table according to the natural language information, it is found that the natural language information does not contain the natural language instruction and the function button identifier corresponding to the natural language instruction, and the large language model analyzes and summarizes and returns the result to the server, and then the server queries the mapping relationship table according to the return result of the large language model to determine whether the return result of the large language model contains the natural language instruction and the function button identifier corresponding to the natural language instruction; if so, the corresponding operation is executed, and the corresponding operation information is sent to the client, which is executed and displayed by the client; if not, the return result of the large language model is directly sent to the client, and the client displays it to the user.

[0041] Optionally, obtaining natural language information input by the user on the client includes:

[0042] Displaying a user interface to the user, and acquiring the natural language information input by the user through the user interface;

[0043] The natural language information includes text information or voice information.

[0044] Optionally, the user interface includes an intelligent assistant interface;

[0045] The step of displaying a user interface to the user and obtaining the natural language information input by the user through the user interface specifically includes:

[0046] The smart assistant interface is displayed to the user, the smart assistant interface interacts with the user, and the natural language information input by the user is received.

[0047] Optionally, the smart assistant interface can be opened in at least one of the following ways:

[0048] The first method is to set a smart assistant button on the application home page interface and / or the application non-home page interface, and the user clicks the smart assistant button to open the smart assistant interface;

[0049] The second method is to set the smart assistant interface as the application home page interface, so that the user directly enters the smart assistant interface when opening the application.

[0050] Optionally, the smart assistant interface also includes a display area for each interface and each function button of the application, and a display frame is set according to each interface and the function buttons contained in each interface, and each interface and its function buttons are displayed in the display area and the display frame according to the interface hierarchy structure and the function hierarchy structure.

[0051] Optionally, when the user directly clicks on the display area of ​​each interface and each function button and the function buttons and interface buttons in the display frame in the assistant interface, the corresponding operation is directly performed.

[0052] Optionally, when the user enters the natural language information of "turn on a certain function button" in the smart assistant interface, the mapping relationship table is queried based on the natural language information to determine whether the natural language information contains the natural language instruction and the function button identifier corresponding to the natural language instruction, and the corresponding operation is performed based on the function button identifier.

[0053] Optionally, the intelligent assistant interface further includes a command execution display area;

[0054] When the user inputs natural language information of "open a certain function button" in the intelligent assistant interface, after the steps of querying the mapping relationship table according to the natural language information, determining that the natural language information contains the natural language instruction and the function button identifier corresponding to the natural language instruction, and performing the corresponding operation according to the function button identifier, the application operation instruction processing method further includes:

[0055] The display area and the display frame of each product interface and each function button are folded upward to the minimum size, and the user dialogue bar is folded downward to the minimum size, and the remaining area after the display area, the display frame and the user dialogue bar are folded is used as the instruction execution display area, and the instruction execution display area is used to display the instruction execution process of the natural language instruction;

[0056] Or, directly jump to the corresponding operation interface executed according to the function button identifier, and overlay the smart assistant interface or the simplified interface of the smart assistant on the corresponding operation interface, and display the corresponding operation interface executed according to the function button identifier after the jump through the smart assistant interface or the simplified interface of the smart assistant.

[0057] Or, directly jump to the operation interface corresponding to the function button identifier, and display the smart assistant interface button floating on the corresponding operation interface, and click the button to open the smart assistant interface.

[0058] Optionally, after the step of performing a corresponding operation according to the function button identifier, the application operation instruction processing method further includes:

[0059] Display the animation effect of the function button being pressed, and display or jump to the interface after the function button executes the operation.

[0060] Optionally, the interface after the function button performs the operation is provided with a smart assistant button or a smart assistant return button;

[0061] After the steps of displaying the animation effect of the function button being pressed and displaying or jumping to the interface after the function button performs the operation, the application operation instruction processing method further includes:

[0062] When the user clicks the smart assistant button or the smart assistant return button, the interface after the operation is performed on the function button returns to the smart assistant interface.

[0063] In a third aspect, the present application provides a method for processing application operation instructions, wherein the execution subject of the method for processing application operation instructions is a server side, and the method for processing application operation instructions includes:

[0064] Receiving natural language information sent by a client; the natural language information includes text information or voice information;

[0065] According to the natural language information, a mapping relationship table is searched, wherein the mapping relationship between the natural language instruction and the corresponding function button identifier is preset in the mapping relationship table, determining that the natural language information contains the natural language instruction and the function button identifier corresponding to the natural language instruction, and performing a corresponding operation according to the function button identifier;

[0066] The operation information corresponding to the execution is sent to the client, which executes and displays it.

[0067] Optionally, the application operation instruction processing method further includes:

[0068] When the server side queries the mapping relationship table according to the natural language information and fails to find that the natural language information contains the natural language instruction and the function button identifier corresponding to the natural language instruction, the natural language information is sent to the large language model side, and the large language model analyzes and summarizes and returns the result to the server side; the server side then queries the mapping relationship table according to the return result of the large language model to determine whether the return result of the large language model contains the function button identifier corresponding to the natural language instruction; if so, the corresponding operation is executed, and the corresponding operation information is sent to the client side, which is executed and displayed by the client side; if not, the return result of the large language model is directly sent to the client side, and the client side displays it to the user;

[0069] Or, the natural language information is sent to the server and to the large language model at the same time; when the server queries the mapping relationship table according to the natural language information, it is not found that the natural language information contains the natural language instruction and the function button identifier corresponding to the natural language instruction, and the large language model analyzes and summarizes and returns the result to the server, and then the server queries the mapping relationship table according to the return result of the large language model to determine whether the return result of the large language model contains the function button identifier corresponding to the natural language instruction; if so, the corresponding operation is executed, and the corresponding operation information is sent to the client, which is executed and displayed by the client; if not, the return result of the large language model is directly sent to the client, and the client displays it to the user.

[0070] According to the specific embodiments provided in this application, this application discloses the following technical effects:

[0071] The present application provides an application operation instruction processing system and method, by setting a mapping relationship table containing the mapping relationship between each natural language instruction and the corresponding function button identifier on the server side, a one-to-one binding mapping between the natural language instruction and the function button identifier is realized, based on which, after the user inputs natural language information such as text information and / or voice information through the client, the mapping relationship table can be queried through the server side and it can be determined that the natural language information contains the natural language instruction and the function button identifier corresponding to the natural language instruction, and then the corresponding operation is performed according to the function button identifier. Therefore, in the present application, the user only needs to input natural language information such as text and voice to the server side through the client, and the function button identifier corresponding to the natural language instruction of the natural language information can be immediately retrieved through the server side and the corresponding operation is performed, which not only enables ordinary users to obtain a more convenient operation experience, but also is suitable for new users and middle-aged and elderly users who are operating unfamiliar new applications for the first time. Users do not need to perform complex operations, and it is simpler and more convenient to use, which effectively reduces the difficulty of operating the application and improves the simplicity and ease of operation of the application. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0073] Figure 1 A schematic diagram of the structure of an application program operation instruction processing system provided in one embodiment of the present application.

[0074] Figure 2 A schematic diagram of the structure of another application program operation instruction processing system provided in one embodiment of the present application.

[0075] Figure 3 A working principle diagram of an application operation instruction processing system provided in one embodiment of the present application.

[0076] Figure 4 A working principle diagram of another application operation instruction processing system provided in one embodiment of the present application.

[0077] Figure 5 A working principle diagram of another application operation instruction processing system provided in one embodiment of the present application.

[0078] Figure 6 A flowchart of a method for processing application operation instructions provided in one embodiment of the present application.

[0079] Figure 7 A flowchart of another method for processing application program operation instructions provided in one embodiment of the present application. DETAILED DESCRIPTION

[0080] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0081] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0082] In an exemplary embodiment, a system for processing application operation instructions is provided, which mainly includes a client and a server, and the client and the server are connected. The client is used to receive natural language information input by a user and send the natural language information to the server. The natural language information includes text information or voice information. The natural language information input by the user can be regarded as a natural language instruction issued, and the natural language information contains natural language instructions, that is, what operation the user wants to achieve, such as opening a certain interface, jumping to a certain interface, opening a certain function button, exiting and other natural language instructions, and the natural language instructions also include text instructions and / or voice instructions. The server is provided with a mapping relationship table, and the mapping relationship table is preset with a mapping relationship between the natural language instruction and the corresponding function button identifier. The server is used to query the mapping relationship table according to the natural language information, and the mapping relationship table is preset with a mapping relationship between the natural language instruction and the corresponding function button identifier, determine that the natural language information contains the natural language instruction and the function button identifier corresponding to the natural language instruction, and perform the corresponding operation according to the function button identifier.

[0083] Based on the above-mentioned application operation instruction processing system, a mapping relationship table containing the mapping relationship between each natural language instruction and the corresponding function button identifier is set on the server side to realize a one-to-one binding mapping between the natural language instruction and the function button identifier. The user only needs to input natural language information such as text and voice to the server side through the client, and the function button identifier corresponding to the natural language instruction of the natural language information can be immediately retrieved through the server side and the corresponding operation can be performed. It is suitable for new users who are operating unfamiliar new applications for the first time and middle-aged and elderly users. Users do not need to perform complicated operations, and it is simpler and more convenient to use, which effectively reduces the difficulty of operating the application and improves the simplicity and ease of operation of the application.

[0084] In an exemplary embodiment, a system for processing application program operation instructions is provided, such as Figure 1 As shown, the application operation instruction processing system mainly includes a client, a server and a large language model, wherein the client is connected to the server, the large language model is connected to the server, and is connected to the client through the server.

[0085] In this embodiment, the client is used to receive natural language information input by a user and send the natural language information to the server; wherein the natural language information includes text information or voice information.

[0086] In this embodiment, the server side is provided with a mapping relationship table, in which a mapping relationship between a natural language instruction and a corresponding function button identifier is preset.

[0087] In this embodiment, the server is used to query the mapping relationship table according to the natural language information, determine whether the natural language information contains the natural language instruction and the function button identifier corresponding to the natural language instruction, and perform the corresponding operation according to the function button identifier; and send the corresponding operation information to the client, which is executed and displayed by the client.

[0088] In this embodiment, the large language model end is used to obtain the natural language information input by the user in the client, and use the large language model to perform in-depth analysis on the natural language information to determine the user's true intention, generate a processing result, and send the processing result to the server end; the processing result is used by the server end to determine the function button identifier.

[0089] In this embodiment, the server is also used to determine the function button identifier according to the processing result generated by the large language model when the natural language instruction and its corresponding function button identifier cannot be found in the mapping relationship table.

[0090] The large language model provides powerful natural language understanding and generation capabilities, helping the server to more accurately understand the user's natural language instructions. In addition, the large language model also provides a standard API interface, accepting natural language information and its natural language instructions as input and returning processing results.

[0091] In this embodiment, the client includes a user interface module, a network communication module and an instruction execution display module. The user interface module is used to display a user interface to the user, interact with the user through the user interface, and receive the natural language information input by the user; the network communication module is used to perform network communication and data interaction with the server; the instruction execution display module is used to display the specific operation process when executing the natural language instruction in real time.

[0092] In this embodiment, the client further includes an input processing module and an execution step recording module.

[0093] The input processing module is used to perform format conversion processing on the natural language information, and convert the format of the natural language information into a data format suitable for reception by the server.

[0094] The execution step recording module includes an execution step recording unit and an execution step sending unit connected in sequence; the execution step recording unit is used to record the specific operation steps when executing the natural language instruction; the execution step sending unit is used to send the recorded specific operation steps when executing the natural language instruction to the instruction execution display module, so that the instruction execution display module can display the specific operation steps when executing the natural language instruction in real time.

[0095] In this embodiment, the server side includes an animation special effect module; the animation special effect module is used to generate and play animation special effects; the animation special effect includes an animation special effect of pressing a button when executing the natural language instruction.

[0096] In this embodiment, the server side also includes an instruction preprocessing module, and the instruction preprocessing module includes a keyword extraction unit, a semantic analysis unit, and a regular expression unit. The keyword extraction unit is used to extract keywords from the natural language information; the semantic analysis unit is used to perform semantic analysis on the natural language information according to the keywords to obtain semantic analysis results; the regular expression unit is used to perform format matching on the natural language information according to the semantic analysis results to obtain instructions in a specific format; the instructions in the specific format refer to instructions in a preset format that characterizes the key information and semantic information of the natural language information, and the preset format refers to a format that is the same as the format of each of the natural language instructions in the mapping relationship table.

[0097] In this embodiment, the server side also includes a mapping management module, which includes a mapping establishment unit, a mapping query unit and a mapping storage unit. The mapping establishment unit is used to establish a mapping relationship between the natural language instruction and the function button identifier, and generate the mapping relationship table; the mapping storage unit is used to store the mapping relationship table; the mapping query unit is used to query the mapping relationship table according to the natural language information, and determine whether the natural language information contains the natural language instruction and the function button identifier corresponding to the natural language instruction.

[0098] In this embodiment, the server side also includes an API calling module, which is used to initiate an API calling request to the large language model side. The API calling request is to request the large language model side to perform in-depth analysis of the natural language information and generate the processing result, and feed back the processing result to the server side.

[0099] In this embodiment, the natural language instruction includes an operation instruction, and the operation instruction includes a click instruction, an open instruction, a close instruction, a touch slide instruction, an up slide instruction, a down slide instruction, a left slide instruction, a right slide forward instruction, a return instruction, a back instruction, and an exit instruction, etc. The natural language instruction also includes a client interface button name and / or a function button name following the operation instruction.

[0100] In this embodiment, the function button identifiers include a function button identifier for a click instruction, a function button identifier for an open instruction, a function button identifier for a close instruction, a function button identifier for a touch slide instruction, a function button identifier for an up slide instruction, a function button identifier for a down slide instruction, a function button identifier for a left slide instruction, a function button identifier for a right slide forward instruction, a function button identifier for a return instruction, a function button identifier for a back instruction, and a function button identifier for an exit instruction, etc.

[0101] As an optional implementation, Figure 2 As shown, another application program operation instruction processing system is also provided. Figure 1 The functional modules of the application operation instruction processing system in the system are the same, and the only difference is that an animation special effect module is provided in the client of the system. The animation special effect module is used to generate and play animation special effects, and the animation special effects include animation special effects of pressing a function button when executing the natural language instruction.

[0102] As an optional implementation, Figure 3As shown, after the client obtains the natural language information input by the user, it first sends the natural language information to the server. The instruction preprocessing module of the server performs natural language preprocessing on the natural language information, and queries the mapping relationship table according to the natural language preprocessing result, and determines that the natural language information contains the natural language instruction and the function button identifier corresponding to the natural language instruction, and performs the corresponding operation according to the function button identifier. While the instruction preprocessing module is performing natural language preprocessing, the server side will also request to call the large language model in the large language model side through the API call module, and forward the natural language information to the large language model side through the API interface. The large language model side then uses the large language model to conduct in-depth analysis of the natural language information, generate processing results, and feed the processing results back to the instruction preprocessing module on the server side. If the server side completes the query of the mapping relationship table and finds the natural language instructions of the natural language information and their corresponding function button identifiers based on the natural language preprocessing results, it will not further analyze and perform subsequent table lookup based on the processing results generated by the large language model side; otherwise, the instruction preprocessing module will analyze and query the mapping relationship table based on the processing results, complete the table lookup based on the processing results and find the function button identifiers corresponding to the natural language instructions of the natural language information, and perform corresponding operations based on the function button identifiers.

[0103] As an optional implementation, Figure 4 As shown, after obtaining the natural language information input by the user, the client first sends the natural language information to the server, and the instruction preprocessing module of the server performs natural language preprocessing on the natural language information, and queries the mapping relationship table according to the natural language preprocessing result, determines that the natural language information contains the natural language instruction and the function button identifier corresponding to the natural language instruction, and performs the corresponding operation according to the function button identifier. If the server does not query the function button identifier corresponding to the natural language instruction of the natural language information according to the natural language preprocessing result of the instruction preprocessing module, the server will interact with the large language model end at this time, and the server requests to call the large language model in the large language model end through the API calling module, and forwards the natural language instruction to the large language model end through the API interface, and the large language model end uses the large language model to perform in-depth analysis of the natural language information, generates a processing result, and feeds the processing result back to the server, and the server further analyzes and queries the mapping relationship table according to the processing result, completes the table lookup work based on the processing result, and finds the function button identifier corresponding to the natural language instruction of the natural language information, and performs the corresponding operation according to the function button identifier.

[0104] As an optional implementation, Figure 5As shown, after the client obtains the natural language information input by the user, it will first send the natural language information to the server. The server requests to call the large language model in the large language model end through the API call module, and forwards the natural language instruction to the large language model end through the API interface. Then, the large language model end uses the large language model to perform in-depth analysis of the natural language information, generate processing results, and forward the processing results to the server. Then, the server queries the mapping relationship table according to the processing results, and determines that the natural language information contains the natural language instruction and the function button identifier corresponding to the natural language instruction, and performs the corresponding operation according to the function button identifier.

[0105] In this embodiment, when the server side queries the mapping relationship table according to the natural language information and fails to find that the natural language information contains the natural language instruction and the function button identifier corresponding to the natural language instruction, the natural language information is sent to the large language model side, and the large language model analyzes and summarizes the information and returns the result to the server side; the server side then queries the mapping relationship table according to the return result of the large language model to determine whether the natural language information contains the natural language instruction and the function button identifier corresponding to the natural language instruction; if so, the corresponding operation is performed, and the corresponding operation information is sent to the client, which is executed and displayed by the client; if not, the return result of the large language model is directly sent to the client, which is displayed to the user by the client.

[0106] Alternatively, the natural language information is sent to the server and to the large language model at the same time; when the server queries the mapping relationship table according to the natural language information, and fails to find that the natural language information contains the natural language instruction and the function button identifier corresponding to the natural language instruction, the large language model analyzes and summarizes the information and returns the result to the server, and then the server queries the mapping relationship table according to the return result of the large language model to determine whether the natural language information contains the natural language instruction and the function button identifier corresponding to the natural language instruction; if yes, the corresponding operation is executed, and the corresponding operation information is sent to the client, which is executed and displayed by the client; if no, the return result of the large language model is directly sent to the client, which is displayed to the user by the client.

[0107] As an optional implementation, the application operation instruction processing system proposed in this embodiment can be used for client applications and can also be used for computer applications.

[0108] The specific structure of the application operation instruction processing system and the specific functions of each functional module are described in detail below.

[0109] (1) Client: It includes a user interface module, an input processing module, a command execution and display module, and an animation special effect module connected in sequence, as well as a network communication module connected to each of the above modules.

[0110] User interface module: displays the user interface to the user, interacts with the user through the user interface, and receives natural language information input by the user, including text information, voice information and other information.

[0111] The user interface module displays the user interface and interacts with the user through the user interface. A further embodiment also includes: the displayed user interface includes displaying an intelligent assistant interface, and interacts with the user through the intelligent assistant interface, receiving natural language information input by the user, including text information, voice information, etc.

[0112] The smart assistant interface can be opened in at least one of the following ways:

[0113] The first method is to set a smart assistant button on the application home page interface and / or the application non-home page interface, and the user clicks the smart assistant button to open the smart assistant interface;

[0114] The second method is to set the smart assistant interface as the application home page interface, so that the user directly enters the smart assistant interface when opening the application.

[0115] In this embodiment, the smart assistant interface also includes a display area for each interface and each function button of the application. A display frame is set according to each interface and the function buttons contained in each interface, and each interface and its function button are displayed in the display area and display frame according to the interface hierarchy structure and the function hierarchy structure. For example, the interface name or button of the home page main interface (if the application has four main interfaces, the interface button arranged in the lower left corner and the corresponding opened interface are usually used as the home page main interface or the first main interface) and the various function buttons of the interface are arranged in the first display frame of the display area, and the name or button of the second main interface and the various function buttons of the interface are arranged in the second display frame of the display area, and so on. This display method allows users to directly see the various function buttons of each interface of the application, and can understand the various interfaces and functions of the application in one interface, which is more convenient for users to operate and use.

[0116] In this embodiment, when the user enters the natural language information of "open a certain function button" in the smart assistant interface, the mapping relationship table is queried based on the natural language information to determine whether the natural language information contains the natural language instruction and the function button identifier corresponding to the natural language instruction, and the corresponding operation is performed based on the function button identifier.

[0117] In this embodiment, the intelligent assistant interface also includes a command execution display area;

[0118] When the user inputs natural language information of "open a certain function button" in the intelligent assistant interface, after the steps of querying the mapping relationship table according to the natural language information, determining that the natural language information contains the natural language instruction and the function button identifier corresponding to the natural language instruction, and performing the corresponding operation according to the function button identifier, the application operation instruction processing method further includes:

[0119] The display area and display frame of each product interface and each function button are folded upward to the minimum size, and the user dialog box is folded downward to the minimum size. The remaining area after the display area, the display frame and the user dialog box are folded is used as the instruction execution display area, and the instruction execution display area is used to display the instruction execution process of the natural language instruction.

[0120] Or, directly jump to the corresponding operation interface executed according to the function button identifier, and overlay the intelligent assistant interface or the simplified interface of the intelligent assistant on the corresponding operation interface, and display the corresponding operation interface executed according to the function button identifier after the jump through the intelligent assistant interface or the simplified interface of the intelligent assistant. Among them, the overlay display of the intelligent assistant interface or the simplified interface of the intelligent assistant on the corresponding operation interface specifically includes: overlaying or inlaying the message input boxes at the top and bottom of the intelligent assistant interface on the corresponding operation interface after the jump, and the remaining large area is used to display the corresponding operation interface executed according to the function button identifier after the jump.

[0121] Or, directly jump to the operation interface corresponding to the function button identifier, and display the smart assistant interface button floating on the corresponding operation interface, and click the button to open the smart assistant interface.

[0122] Input processing module: processes natural language information such as text information or voice information input by the user, mainly converting the format into a data format suitable for sending to the server.

[0123] Network communication module: responsible for network communication and data exchange with the server.

[0124] Instruction execution display module: reserve display space in the user interface to display the specific operation process when executing natural language instructions in real time, including the initial state, intermediate state and final result. By displaying the specific operation process and operation steps when executing natural language instructions in real time, it is convenient for users to perform fool-proof operation and for initial users and middle-aged and elderly users to learn the operation process, thereby enhancing the user experience.

[0125] The above instruction execution display module also includes the following implementation modes:

[0126] The function button display area and display frame of each interface of the product are displayed on the smart assistant interface. When the user enters the natural language information of "open a certain function button" in the smart assistant interface, the mapping relationship table is queried according to the natural language information, and it is determined that the natural language information contains the natural language instruction and the function button identifier corresponding to the natural language instruction, and the corresponding operation is performed according to the function button identifier, and the animation effect of the function button being pressed is displayed, and then the interface after the function button performs the operation is displayed or jumped to.

[0127] The display or jump to the interface after the function button is operated is specifically: displaying the interface after the function button is operated on the smart assistant interface, or jumping to the interface after the function button is operated. After jumping to the interface after the function button is operated, a smart assistant button or a smart assistant return button is set on the interface after the function button is operated.

[0128] In this embodiment, the smart assistant button or the smart assistant return button is used to return to the smart assistant interface from the interface after the operation is performed on the function button when the user clicks the smart assistant button or the smart assistant return button.

[0129] An implementation method of the above-mentioned instruction execution display module also includes setting an instruction execution display area in the smart assistant interface. When the user inputs natural language information of "open a certain function button" in the smart assistant interface, the mapping relationship table is queried according to the natural language information to determine that the natural language information contains the natural language instruction and the function button identifier corresponding to the natural language instruction, and the corresponding operation is performed according to the function button identifier. The function button display area and display frame of each interface of the product are folded upward to the minimum size, and the user dialog bar is folded downward to the minimum size. The remaining area after the display area, the display frame and the user dialog bar are folded is used as the instruction execution display area, and the instruction execution display area is used to display the instruction execution process of the natural language instruction.

[0130] Animation special effects module: Generates and plays animation special effects such as button pressing, thereby simulating the scene of button pressing operation, which helps users to master the button positions and button pressing order used in each operation step, and enhances the user interaction experience. For specific embodiments, refer to the implementation method of the above instruction execution display module.

[0131] (2) Server side: It includes a network communication module and an instruction preprocessing module, a mapping management module, an execution step recording module, an API calling module, a function execution module, and a data storage module, etc., which are connected to the network communication module.

[0132] Network communication module: Similar to the network communication module in the client, a network communication module is also provided in the server side, which is responsible for network communication and data interaction between the server side and the client side and between the server side and the large language model side. It mainly receives natural language information from the client side, and sends the operation process and execution results of the natural language instructions of the natural language information back to the client; and receives processing results from the large language model side.

[0133] The instruction preprocessing module includes: a keyword extraction unit, a semantic analysis unit and a regular expression unit. Among them, the keyword extraction unit is used to extract keywords from natural language information. The semantic analysis unit is used to perform semantic analysis on the natural language information according to the keywords to obtain semantic analysis results. The regular expression unit is used to perform format matching on the natural language information according to the semantic analysis results to obtain instructions in a specific format. The instructions in the specific format refer to instructions in a preset format that characterizes the key information and semantic information of the natural language information, and the preset format refers to a format that is the same as the format of each of the natural language instructions in the mapping relationship table.

[0134] The mapping management module includes: a mapping establishment unit, a mapping storage unit and a mapping query unit connected in sequence. The mapping establishment unit is used to establish a mapping relationship between a natural language instruction and a function button identifier, and generate a mapping relationship table. The mapping storage unit is used to store the mapping relationship table. The mapping query unit is used to query the mapping relationship table according to the natural language instruction in the natural language information, and determine whether the natural language information contains the natural language instruction and the function button identifier corresponding to the natural language instruction.

[0135] The execution step recording module includes: an execution step recording unit and an execution step sending unit. Among them, the execution step recording unit is used to record the specific operation steps when executing natural language instructions, including intermediate states and final results. The execution step sending unit is used to send the recorded specific operation steps when executing natural language instructions to the instruction execution display module in the client, so that the client can display the specific operation steps when executing natural language instructions in real time through the instruction execution display module, thereby facilitating users to perform fool-proof operations and new users and middle-aged and elderly users to learn the specific operation steps when executing natural language instructions in a real-time visual manner.

[0136] API call module: Initiate an API call request to the large language model end based on the parsing results, so that the large language model end can obtain deeper processing results. Among them, the parsing result is mainly the result of whether the natural language instructions contained in the natural language information input by the user and the function button identifier corresponding to the natural language instruction are in the mapping relationship table when the mapping query unit queries the mapping relationship table. The processing result is the result generated by the large language model end based on the large language model technology, which conducts an in-depth analysis of the natural language instructions, and is used to help the server end make further accurate decisions about the function button identifier.

[0137] Function execution module: Calls the corresponding function module according to the function button identifier and performs the corresponding operation, thereby realizing the corresponding function of the application according to the user's needs.

[0138] Data storage module: stores information such as the mapping relationship between preset natural language instructions and function button identifiers.

[0139] Database: Deployed on the server side, you can choose to coexist with the application, or deploy it on an independent database server, or use a cloud database for storage. Through the data storage module, you can store mapping relationship tables and mapping relationship information in the database to facilitate data backup and data update to prevent data loss.

[0140] (3) Large language model. This includes the following:

[0141] API interface: Provides a standard API interface, accepts natural language information as input, and returns processing results.

[0142] Large language model: Based on large language model technology, it has natural language processing capabilities and provides powerful natural language understanding and generation capabilities. It helps the server side understand the natural language information input by users more accurately and generates processing results that represent the user's true intentions, which is used to assist the server side in determining function button identifiers.

[0143] In order to support the mapping relationship between natural language instructions and function button identifiers, this embodiment designs a database table, as shown in Table 1.

[0144] Table 1 Database table

[0145]

[0146] As an optional implementation, application operation execution instructions can also be set for the application to implement a mapping relationship of "natural language instructions-application operation execution instructions". The application operation execution instructions are no longer in the form of function button identifiers, but in the form of operation execution instructions for application buttons or pages. The application operation execution instructions and the natural language information input by the user are preprocessed. The preprocessing here refers to establishing a one-to-one mapping relationship between the operation execution instructions of the application button or page and the natural language instructions contained in the natural language information input by the user, wherein the operation execution instructions of the application button or page refer to the operation flow of the application button or page corresponding to the natural language instruction. Corresponding to the operation flow is the function button identifier, and the application button or page is operated according to the function button identifier to complete the operation flow.

[0147] In this embodiment, the interaction between the client and the large language model is implemented in the form of a dialogue interface. After the user enters natural language information in the dialogue interface of the client, it can be acquired by the large language model. After the user enters natural language information in the dialogue interface of the client, in response to the natural language information, the server queries whether there is an application operation execution instruction corresponding to it, and the large language model uses the large language model to parse the natural language information; if the natural language information contains a natural language instruction and a function button identifier corresponding to the natural language instruction, the corresponding application operation execution instruction is executed, and the execution result is displayed on the client or on the dialogue interface between the user and the large language model and / or the interface corresponding to the application operation execution instruction is opened.

[0148] When the execution result is displayed on the client or on the dialogue interface between the user and the large language model end and / or on the interface corresponding to the opening application operation execution instruction, it also includes the operation execution process display, and the operation execution process display is specifically: when the user inputs natural language information as the operation instruction of the sub-interface in the dialogue interface between the client and the large language model end; in response to the natural language information, the large language model end parses the natural language information, and at the same time the server end queries whether there is a corresponding sub-interface application operation execution instruction; if it is determined that the natural language information contains a natural language instruction and a function button identifier corresponding to the natural language instruction, the corresponding sub-interface application operation execution instruction is executed, and the execution process and execution result are displayed on the client or on the dialogue interface between the user and the large language model end and / or on the interface corresponding to the opening application operation execution instruction.

[0149] In this embodiment, the interface includes a main interface and a sub-interface, and the execution process is: first open and display the main interface, and then open and display the sub-interface.

[0150] The specific workflow of the application operation instruction processing system proposed in this embodiment is as follows:

[0151] (1) User input

[0152] The user inputs natural language information on the client by inputting text or voice.

[0153] The user interface module of the client displays the application interface and receives natural language information input by the user.

[0154] (2) Client processing

[0155] The natural language information input by the user is converted into structured data and sent to the server through the network communication module.

[0156] The input processing module processes the text or voice input by the user, converts it into a data format, and sends it to the server through the network communication module to exchange data with the server.

[0157] (3) Server-side reception and preprocessing

[0158] Receive data such as natural language information from the client through the network communication module.

[0159] The user's natural language information is parsed through the instruction preprocessing module, which specifically includes the following processes:

[0160] Keywords are extracted from the natural language information input by the user by the keyword extraction unit.

[0161] The semantic analysis unit performs semantic analysis on keywords to understand the true intention of the user's natural language information.

[0162] The regular expression unit performs format matching on the natural language instructions according to the semantic analysis result to obtain instructions in a specific format, and the instructions in the specific format are sent to the mapping query unit.

[0163] (4) Mapping query

[0164] The mapping relationship table is queried through the mapping query unit to find the corresponding function button identifier.

[0165] Before the mapping query unit queries the mapping relationship table, it also includes a process of generating and storing the mapping relationship table. Specifically, the mapping relationship between the natural language instruction and the function button identifier is established by the mapping establishment unit, and a mapping relationship table is generated, and then the mapping relationship table including the mapping relationship between the preset natural language instruction and the function button identifier (characterizing the product function such as the button) is stored in the database through the data storage module. This process is completed in the initialization stage of the system or the manual configuration stage of the system administrator.

[0166] (5) Recording of execution steps

[0167] The specific operation steps when executing natural language instructions are recorded through the execution step recording unit, including intermediate states and final results.

[0168] (6) Calling the large language model in the large language model terminal

[0169] If the mapping query unit does not find the corresponding function button identifier in the mapping relationship table, it needs to further understand the natural language information input by the user, or the user inputs more complex natural language information, so that the semantic analysis unit on the server side cannot understand the true intention of the user's natural language information at the first time, then the server side will initiate an API call request to the large language model side to call the large language model in the large language model side for in-depth analysis, thereby more deeply understanding the true intention of the user's natural language information based on the large language model technology.

[0170] Therefore, there is also a step of initiating an API call request to the large language model end through the API call module to obtain a deeper processing result, so that the server end can further determine the function button identifier according to the processing result.

[0171] This embodiment includes the following three implementation modes:

[0172] 1) The natural language information input by the user through the client is sent to the server. The server first parses the natural language information input by the user. If the mapping relationship can be determined by querying the mapping relationship table in the first time, and the natural language instruction contained in the natural language information and the function button identifier corresponding to the natural language instruction can be found, the corresponding function module is called according to the function button identifier to perform the corresponding operation; if the corresponding function button identifier cannot be found in the first time, the natural language information input by the user is sent to the large language model, and the large language model is used to parse and obtain the processing result, which includes a standard instruction. The standard instruction is returned to the server to query the mapping relationship in the mapping relationship table to determine the function button identifier corresponding to the standard instruction, which is actually the natural language instruction contained in the natural language information input by the user and the function button identifier corresponding to the natural language instruction, and the corresponding function module is called according to the function button identifier to perform the corresponding operation.

[0173] 2) After the natural language information input by the user through the client is sent to the server, the server sends it to the large language model. The server first parses the natural language information input by the user. If the mapping relationship table can be queried to determine the mapping relationship and the corresponding function button identifier can be found in the first time, the corresponding function module is called according to the function button identifier to perform the corresponding operation. If the corresponding function button identifier cannot be found in the first time, the large language model in the large language model is used to parse and obtain the processing result. The processing result includes a standard instruction. The standard instruction is returned to the server to query the mapping relationship in the mapping relationship table to determine the function button identifier corresponding to the standard instruction, which is actually the natural language instruction contained in the natural language information input by the user and the function button identifier corresponding to the natural language instruction, and the corresponding function module is called according to the function button identifier to perform the corresponding operation.

[0174] 3) The natural language information input by the user through the client is sent to the server, and the server sends it to the large language model at the same time. The server and the large language model respectively parse the natural language information input by the user, and query the mapping relationship in the mapping relationship table. If one of them preferentially finds the natural language instructions contained in the natural language information and the function button identifier corresponding to the natural language instruction, the corresponding function module is called according to the function button identifier to perform the corresponding operation.

[0175] This embodiment combines the mapping technology with the large language model technology, and uses the mapping relationship table and the large language model to determine whether the natural language information contains the natural language instruction and the function button identifier corresponding to the natural language instruction. The mapping relationship table is used as the main method and the large language model is used as the auxiliary method, so that the real intention of the user's natural language information and its natural language instruction can be accurately and reliably understood. Specifically, a mapping relationship table between the natural language instruction and the function button identifier is first established, and the natural language information contains the natural language instruction and the function button identifier corresponding to the natural language instruction is determined by querying the mapping relationship table; if the mapping relationship table does not have a function button identifier corresponding to the natural language instruction of the natural language information input by the user, then the large language model end can be used to perform in-depth analysis based on the natural language information input by the user to obtain a processing result, which is used to characterize the real intention of the user's natural language information, thereby completing the operation process that is consistent with the natural language instruction of the user's natural language information. Since the user only needs to input one natural language information throughout the process, the subsequent process does not require user participation, and it is convenient, fast and uncomplicated to use.

[0176] (7) Instruction Execution

[0177] The function execution module calls the corresponding function module according to the function button identifier and performs the corresponding operation, thereby realizing the corresponding function of the application program according to the user's needs.

[0178] The specific operation steps recorded when executing the natural language instruction are sent to the client through the execution step sending unit.

[0179] (8) The client displays the execution process

[0180] The instruction execution display module in the client reserves display space in the user interface for real-time display of specific operation steps when executing natural language instructions, so as to facilitate user operation and learning for new users and middle-aged and elderly users.

[0181] The animation effects module in the client generates and plays animation effects such as button pressing, which helps users master the button positions and the order of pressing buttons used in each operation step, enhances the user interaction experience, and improves learning effects.

[0182] (9) Feedback Client

[0183] Network communication is performed through the network communication module in the server side and the network communication module in the client side, so that the server side feeds back the execution result to the client side through the network communication module, and the client side receives the execution result from the server side through the network communication module and updates the user interface.

[0184] In order to make the technical solution of this embodiment clearer, the specific usage and implementation process of the system of this embodiment are described in detail below in the form of examples.

[0185] For an application APP or mini program, it includes AI painting, AI video and corresponding functions and buttons, among which AI painting includes two sub-functions and buttons of Vincent picture and picture-generated picture, and AI video includes two sub-functions and buttons of Vincent video and picture-generated video. In the "My" interface, there is also a collection function button. If according to the operation process of the prior art, the user first clicks the "My" button to open the "My" interface; the second step is to click the collection button in the "My" interface to open the collection interface; the third step is to click the Vincent video button in the collection interface to open the Vincent video work interface collected by the user for viewing. The user needs to perform the above three steps and click the button three times to achieve the above operations.

[0186] In this embodiment, the user only needs to input the natural language information and natural language instructions of "open my favorite Vincent videos" or the simpler "favorite Vincent videos" in the dialogue interface with the large language model end by text or voice. In this way, there is no need to click each function button one by one, and the Vincent video interface in the AI ​​video in the "My" interface can be directly opened to browse the works in just one step, which is convenient for users to operate fool-proof, and is extremely convenient for new users who use the application for the first time and middle-aged and elderly users. It is simple, fast and easy to operate. At the same time, the process of opening each interface can also be displayed to the user on the client in sequence. For users who use the application for the first time and middle-aged and elderly users, they can understand and master the process and sequence of learning how to operate to open each interface, and provide users with a tool for learning to use application software, so that they can skillfully operate the application software manually without entering operation instructions under certain conditions of use.

[0187] In an exemplary embodiment, a method for processing an application program operation instruction is provided. The method for processing an application program operation instruction is a method corresponding to the aforementioned application program operation instruction processing system. Figure 6 As shown, the execution subject of the application operation instruction processing method is the client, and the application operation instruction processing method includes the following steps:

[0188] Step S101: Acquire natural language information input by the user on the client.

[0189] Step S102, sending the natural language information to the server; the server queries a mapping relationship table according to the natural language information, wherein the mapping relationship table is preset with a mapping relationship between the natural language instruction and the corresponding function button identifier, determines that the natural language information contains the natural language instruction and the function button identifier corresponding to the natural language instruction, and performs a corresponding operation according to the function button identifier; sends the corresponding operation information to the client, which is executed and displayed by the client.

[0190] In an exemplary embodiment, another application program operation instruction processing method is provided. The application program operation instruction processing method is a method corresponding to the aforementioned application program operation instruction processing system, such as Figure 7 As shown, the execution subject of the application operation instruction processing method is the server side, and the application operation instruction processing method includes the following steps:

[0191] Step S201: receiving natural language information sent by a client; the natural language information includes text information or voice information.

[0192] Step S202: query a mapping relationship table according to the natural language information, wherein the mapping relationship table has a preset mapping relationship between the natural language instruction and the corresponding function button identifier, determine that the natural language information contains the natural language instruction and the function button identifier corresponding to the natural language instruction, and perform a corresponding operation according to the function button identifier.

[0193] Step S203: Send the operation information corresponding to the execution to the client, which executes and displays it.

[0194] The present embodiment proposes a system and method for processing application operation instructions. The user only needs to use text or voice to input a natural language information in the dialogue interface between the client and the large language model end. The natural language instruction of the natural language information represents the instruction of any function button or operation of any interface in the application. The corresponding function of the corresponding interface can be directly opened and the corresponding operation can be performed. The existing technology requires multiple steps to complete the operation, while the present embodiment only needs one step (i.e., the step of inputting natural language information) to achieve it. This is extremely convenient, simple and fast for users who use the application for the first time and middle-aged and elderly users. At the same time, the present application will also display the specific operation steps of opening each interface to the user in sequence, so that users who use the application for the first time and middle-aged and elderly users can learn the operation process and sequence of opening each interface, and provide users with a tool for learning to use application software, so that they can manually operate the application software proficiently without inputting operation instructions under certain usage conditions in the future, effectively reducing the difficulty of operating the application and improving the simplicity and ease of operation of the application.

[0195] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0196] The application operation instruction processing system and method described in this application refers to the use of AI technology to implement operations and instruction execution on products such as applications, not just applications. The application described in this application includes software products and systems, specifically including small programs or APPs running on mobile terminals, and also including computer clients running on computer terminals. Therefore, the application operation instruction processing system and method described in this application includes common small programs or APPs and computer client products, and also includes application products that contain AI functions such as large language models, AI paintings, Vincent images, image-generated images, AI videos, Vincent videos, image-generated videos, AI voice, Vincent voice, AI music, Vincent music and other products and functions.

[0197] This specification uses specific examples to illustrate the principles and implementation methods of this application. The above examples are only used to help understand the method and core ideas of this application. At the same time, for those skilled in the art, according to the ideas of this application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. An application operation instruction processing system, characterized in that: The application operation instruction processing system comprises a client and a server, wherein the client and the server are connected; The client is used to receive natural language information input by a user and send the natural language information to the server; wherein the natural language information includes text information or voice information; The server is provided with a mapping relationship table, wherein the mapping relationship between the natural language instruction and the corresponding function button identifier is preset in the mapping relationship table; The server is used to query the mapping relationship table according to the natural language information, determine whether the natural language information contains the natural language instruction and the function button identifier corresponding to the natural language instruction, and perform the corresponding operation according to the function button identifier; and send the corresponding operation information to the client, which is executed and displayed by the client.

2. The application program operation instruction processing system according to claim 1, characterized in that: The application operation instruction processing system further includes a large language model end, the large language model end is connected to the server end, and is connected to the client end through the server end; The large language model end is used to obtain the natural language information input by the user in the client, and use the large language model to perform in-depth analysis on the natural language information to determine the user's true intention, generate a processing result, and send the processing result to the server end; The processing result is used by the server to determine the function button identifier; The server end is also used to determine the function button identifier according to the processing result generated by the large language model when the natural language instruction and its corresponding function button identifier cannot be found in the mapping relationship table.

3. The application program operation instruction processing system according to claim 1, characterized in that: The client includes a user interface module, a network communication module and an instruction execution display module; The user interface module is used to display a user interface to the user, interact with the user through the user interface, and receive the natural language information input by the user; The network communication module is used to perform network communication and data interaction with the server side; The instruction execution display module is used to display the specific operation process when executing the natural language instruction in real time.

4. The application program operation instruction processing system according to claim 3, characterized in that: The client also includes an input processing module and an execution step recording module; The input processing module is used to perform format conversion processing on the natural language information, converting the format of the natural language information into a data format suitable for reception by the server; The execution step recording module includes an execution step recording unit and an execution step sending unit; The execution step recording unit is used to record the specific operation steps when executing the natural language instruction; The execution step sending unit is used to send the recorded specific operation steps when executing the natural language instruction to the instruction execution display module, so that the instruction execution display module can display the specific operation steps when executing the natural language instruction in real time.

5. The application program operation instruction processing system according to claim 4, characterized in that: The server side includes an animation special effects module; The animation special effects module is used to generate and play animation special effects; the animation special effects include animation special effects of pressing a button when executing the natural language instruction.

6. The application program operation instruction processing system according to claim 1, characterized in that: The server side also includes an instruction preprocessing module, which includes a keyword extraction unit, a semantic analysis unit and a regular expression unit; The keyword extraction unit is used to extract keywords from the natural language information; The semantic analysis unit is used to perform semantic analysis on the natural language information according to the keywords to obtain a semantic analysis result; The regular expression unit is used to perform format matching on the natural language information according to the semantic analysis result to obtain instructions in a specific format; the instructions in the specific format refer to instructions in a preset format that characterize key information and semantic information of the natural language information, and the preset format refers to a format that is the same as the format of each natural language instruction in the mapping relationship table.

7. The application program operation instruction processing system according to claim 1, characterized in that: The server side also includes a mapping management module, which includes a mapping establishment unit, a mapping query unit and a mapping storage unit; The mapping establishment unit is used to establish a mapping relationship between the natural language instruction and the function button identifier, and generate the mapping relationship table; The mapping storage unit is used to store the mapping relationship table; The mapping query unit is used to query a mapping relationship table according to the natural language information, and determine whether the natural language information contains the natural language instruction and a function button identifier corresponding to the natural language instruction.

8. The application program operation instruction processing system according to claim 2, characterized in that: The server side also includes an API calling module, which is used to initiate an API calling request to the large language model side. The API calling request is to request the large language model side to perform in-depth analysis of the natural language information and generate the processing result, and feed back the processing result to the server side.

9. The application program operation instruction processing system according to claim 1, characterized in that: The natural language instructions include operation instructions, and the operation instructions include click instructions, open instructions, close instructions, touch slide instructions, up slide instructions, down slide instructions, left slide instructions, right slide forward instructions, return instructions, back instructions and exit instructions; The natural language instruction also includes a client interface button name and / or a function button name following the operation instruction.

10. A method for processing application operation instructions, characterized in that: The execution subject of the application operation instruction processing method is the client, and the application operation instruction processing method includes: Obtain the natural language information entered by the user on the client; The natural language information is sent to a server; the server queries a mapping relationship table according to the natural language information, wherein the mapping relationship table is preset with a mapping relationship between natural language instructions and corresponding function button identifiers, determines that the natural language information contains the natural language instruction and the function button identifier corresponding to the natural language instruction, and performs a corresponding operation according to the function button identifier; and sends the corresponding operation information to a client, which is executed and displayed by the client.

11. The method for processing application program operation instructions according to claim 10, characterized in that: The natural language information is sent to a server; the server queries a mapping relationship table according to the natural language information, wherein the mapping relationship between the natural language instruction and the corresponding function button identifier is preset in the mapping relationship table, and after determining that the natural language information contains the natural language instruction and the function button identifier corresponding to the natural language instruction, the application operation instruction processing method further includes: When the server side queries the mapping relationship table according to the natural language information and fails to find that the natural language information contains the natural language instruction and the function button identifier corresponding to the natural language instruction, the natural language information is sent to the large language model side, and the large language model analyzes and summarizes and returns the result to the server side; the server side then queries the mapping relationship table according to the return result of the large language model to determine whether the return result of the large language model contains the natural language instruction and the function button identifier corresponding to the natural language instruction; if so, the corresponding operation is executed, and the corresponding operation information is sent to the client, which is executed and displayed by the client; if not, the return result of the large language model is directly sent to the client, and the client displays it to the user; Or, the natural language information is sent to the server and to the large language model at the same time; when the server queries the mapping relationship table according to the natural language information, it is found that the natural language information does not contain the natural language instruction and the function button identifier corresponding to the natural language instruction, and the large language model analyzes and summarizes and returns the result to the server, and then the server queries the mapping relationship table according to the return result of the large language model to determine whether the return result of the large language model contains the natural language instruction and the function button identifier corresponding to the natural language instruction; if so, the corresponding operation is executed, and the corresponding operation information is sent to the client, which is executed and displayed by the client; if not, the return result of the large language model is directly sent to the client, and the client displays it to the user.

12. The method for processing application operation instructions according to claim 10 or 11, characterized in that: Obtain the natural language information entered by the user on the client, including: Displaying a user interface to the user, and obtaining the natural language information input by the user through the user interface; The natural language information includes text information or voice information.

13. The method for processing application program operation instructions according to claim 12, characterized in that: The user interface includes an intelligent assistant interface; The step of displaying a user interface to the user and obtaining the natural language information input by the user through the user interface specifically includes: The smart assistant interface is displayed to the user, the smart assistant interface interacts with the user, and the natural language information input by the user is received.

14. The method for processing application program operation instructions according to claim 13, characterized in that: The smart assistant interface can be opened in at least one of the following ways: The first method is to set a smart assistant button on the application home page interface and / or the application non-home page interface, and the user clicks the smart assistant button to open the smart assistant interface; The second method is to set the smart assistant interface as the application home page interface, so that the user directly enters the smart assistant interface when opening the application.

15. The method for processing application program operation instructions according to claim 14, characterized in that: The smart assistant interface also includes a display area for each interface and each function button of the application. A display frame is set according to each interface and the function buttons contained in each interface. Each interface and its function buttons are displayed in the display area and the display frame according to the interface hierarchy structure and the function hierarchy structure.

16. The method for processing application program operation instructions according to claim 15, characterized in that: When the user directly clicks on the display area of ​​each interface and each function button and the function buttons and interface buttons in the display frame in the energy assistant interface, the corresponding operation is directly executed.

17. The method for processing application program operation instructions according to claim 16, characterized in that: When the user inputs the natural language information of "open a certain function button" in the smart assistant interface, the mapping relationship table is queried according to the natural language information to determine whether the natural language information contains the natural language instruction and the function button identifier corresponding to the natural language instruction, and the corresponding operation is performed according to the function button identifier.

18. The method for processing application program operation instructions according to claim 17, characterized in that: The smart assistant interface also includes a command execution display area; When the user inputs natural language information of "open a certain function button" in the smart assistant interface, after the steps of querying the mapping relationship table according to the natural language information, determining that the natural language information contains the natural language instruction and the function button identifier corresponding to the natural language instruction, and performing the corresponding operation according to the function button identifier, the application operation instruction processing method further includes: The display area and the display frame of each product interface and each function button are folded upward to the minimum size, and the user dialogue bar is folded downward to the minimum size, and the remaining area after the display area, the display frame and the user dialogue bar are folded is used as the instruction execution display area, and the instruction execution display area is used to display the instruction execution process of the natural language instruction; Or, directly jump to the corresponding operation interface executed according to the function button identifier, and overlay the smart assistant interface or the simplified interface of the smart assistant on the corresponding operation interface, and display the corresponding operation interface executed according to the function button identifier after the jump through the smart assistant interface or the simplified interface of the smart assistant. Or, directly jump to the operation interface corresponding to the function button identifier, and display the smart assistant interface button floating on the corresponding operation interface, and click the button to open the smart assistant interface.

19. The method for processing application program operation instructions according to claim 17, characterized in that: After the step of executing the corresponding operation according to the function button identifier, the application operation instruction processing method further includes: Display the animation effect of the function button being pressed, and display or jump to the interface after the function button executes the operation.

20. The method for processing application program operation instructions according to claim 19, characterized in that: The interface after the function button performs the operation is provided with a smart assistant button or a smart assistant return button; After the steps of displaying the animation effect of the function button being pressed and displaying or jumping to the interface after the function button performs the operation, the application operation instruction processing method further includes: When the user clicks the smart assistant button or the smart assistant return button, the interface after the operation is performed on the function button returns to the smart assistant interface.

21. A method for processing an application program operation instruction, characterized in that: The execution subject of the application operation instruction processing method is the server side, and the application operation instruction processing method includes: Receiving natural language information sent by a client; the natural language information includes text information or voice information; According to the natural language information, a mapping relationship table is searched, wherein the mapping relationship between the natural language instruction and the corresponding function button identifier is preset in the mapping relationship table, determining that the natural language information contains the natural language instruction and the function button identifier corresponding to the natural language instruction, and performing a corresponding operation according to the function button identifier; The operation information corresponding to the execution is sent to the client, which executes and displays it.

22. The method for processing application program operation instructions according to claim 21, characterized in that: The application program operation instruction processing method further includes: When the server side queries the mapping relationship table according to the natural language information and fails to find that the natural language information contains the natural language instruction and the function button identifier corresponding to the natural language instruction, the natural language information is sent to the large language model side, and the large language model analyzes and summarizes and returns the result to the server side; the server side then queries the mapping relationship table according to the return result of the large language model to determine whether the return result of the large language model contains the function button identifier corresponding to the natural language instruction; if so, the corresponding operation is executed, and the corresponding operation information is sent to the client side, which is executed and displayed by the client side; if not, the return result of the large language model is directly sent to the client side, and the client side displays it to the user; Or, the natural language information is sent to the server and to the large language model at the same time; when the server queries the mapping relationship table according to the natural language information, it is not found that the natural language information contains the natural language instruction and the function button identifier corresponding to the natural language instruction, and the large language model analyzes and summarizes and returns the result to the server, and then the server queries the mapping relationship table according to the return result of the large language model to determine whether the return result of the large language model contains the function button identifier corresponding to the natural language instruction; if so, the corresponding operation is executed, and the corresponding operation information is sent to the client, which is executed and displayed by the client; if not, the return result of the large language model is directly sent to the client, and the client displays it to the user.

Images