Terminal control method and device, terminal and storage medium

Abstract

The embodiment of the invention discloses a terminal control method and device, a terminal and a storage medium, and relates to the field of artificial intelligence. Comprising the steps that a natural language instruction is acquired, and the natural language instruction is natural language content indicating a terminal to achieve a target function; an instruction analysis request containing the natural language instruction is sent to a server, the server is used for generating an interface operation instruction sequence corresponding to the natural language instruction through a generative large language model, the interface operation instruction sequence comprises interface operation instructions executed in sequence, and the interface operation instructions are instructions corresponding to interface operations supported by the terminal; receiving an interface operation instruction sequence sent by the server; and executing the interface operation in the user interface based on the interface operation instruction sequence to realize the target function. According to the method, the interface operation instruction sequence is generated through the generative large language model, and the effect of deeply understanding user instructions is achieved; the interface operation is executed based on the interface operation instruction sequence, and the terminal can be automatically controlled.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of artificial intelligence, and in particular to a terminal control method, device, terminal and storage medium. Background Art

[0002] With the development of terminal technology, voice assistants provide more convenience for people to control terminals.

[0003] In the related art, the user can issue a voice command to the terminal, and the voice assistant extracts the key words from the voice command and matches them with the words pre-set in the database to return the answer corresponding to the key words, or based on the key words, implement the control operation of the terminal in a pre-defined manner. For example, when the user issues the command "create a 5-minute countdown", the voice assistant matches the key word "countdown" with the countdown setting content pre-stored in the database to determine that the application corresponding to "countdown" is "clock", and controls the terminal to automatically open the clock and create a 5-minute countdown. Summary of the invention

[0004] The embodiment of the present application provides a terminal control method, device, terminal and storage medium. The technical solution is as follows:

[0005] On the one hand, an embodiment of the present application provides a terminal control method, the method comprising:

[0006] Acquire a natural language instruction, where the natural language instruction is natural language content that instructs the terminal to implement a target function;

[0007] Sending an instruction parsing request including the natural language instruction to a server, wherein the server is used to generate an interface operation instruction sequence corresponding to the natural language instruction through a generative large language model, wherein the interface operation instruction sequence includes interface operation instructions to be executed in sequence, and the interface operation instructions are instructions corresponding to the interface operations supported by the terminal;

[0008] Receiving the interface operation instruction sequence sent by the server;

[0009] An interface operation is performed in a user interface based on the interface operation instruction sequence to achieve the target function.

[0010] On the other hand, an embodiment of the present application provides a terminal control device, the device comprising:

[0011] An acquisition module, used to acquire a natural language instruction, wherein the natural language instruction is a natural language content instructing the terminal to implement a target function;

[0012] A sending module, used to send an instruction parsing request containing the natural language instruction to a server, wherein the server is used to generate an interface operation instruction sequence corresponding to the natural language instruction through a generative large language model, wherein the interface operation instruction sequence includes interface operation instructions to be executed in sequence, and the interface operation instructions are instructions corresponding to the interface operations supported by the terminal;

[0013] A receiving module, used for receiving the interface operation instruction sequence sent by the server;

[0014] An execution module is used to execute interface operations in a user interface based on the interface operation instruction sequence to achieve the target function.

[0015] On the other hand, an embodiment of the present application provides a server, which includes a processor and a memory, wherein at least one program code is stored in the memory, and the at least one program code is loaded and executed by the processor to implement the method described in any of the above embodiments.

[0016] On the other hand, an embodiment of the present application provides a terminal, which includes a processor and a memory, wherein the memory stores at least one instruction, and the at least one instruction is loaded and executed by the processor to implement the method described in the above aspect.

[0017] On the other hand, an embodiment of the present application provides a computer-readable storage medium, in which at least one instruction is stored, and the instruction is loaded and executed by a processor to implement the method described in the above aspect.

[0018] On the other hand, an embodiment of the present application provides a computer program product, which includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. A processor of a terminal reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the terminal executes the method provided in various optional implementations of the above aspects.

[0019] In the embodiment of the present application, the interface operation instruction sequence is generated by the generative large language model, which can deeply understand the natural language instructions input by the user and convert them into multiple interface operation instructions executed in sequence, wherein the interface operation instructions are instructions corresponding to the interface operations supported by the terminal, so the terminal can perform interface operations based on the interface operation instruction sequence, thereby realizing automatic control of the terminal. The method of the embodiment of the present application can deeply understand the user's intention, greatly expand the scenarios for realizing automatic control of the terminal, and realize a variety of different functions. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0021] Figure 1 is a schematic diagram of an implementation environment provided by an exemplary embodiment of the present application;

[0022] Figure 2 is a flow chart of a terminal control method provided by an exemplary embodiment of the present application;

[0023] Figure 3 is a schematic diagram of an execution interface operation provided by an exemplary embodiment of the present application;

[0024] Figure 4 is a schematic diagram of determining a target control that matches a target control function provided by an exemplary embodiment of the present application;

[0025] Figure 5 is a flow chart of determining an interface element function corresponding to an interface element provided by an exemplary embodiment of the present application;

[0026] Figure 6 is a schematic diagram of determining a target control in combination with a human eye gaze area provided by an exemplary embodiment of the present application;

[0027] Figure 7 is a schematic diagram of a control terminal in a multi-device collaborative mode provided by an exemplary embodiment of the present application;

[0028] Figure 8 is a structural block diagram of a terminal control device provided by an exemplary embodiment of the present application;

[0029] Fig. 9 It is a structural block diagram of a terminal provided by an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0030] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0031] To achieve automatic control of the terminal, current terminal devices such as smartphones and smart computers are integrated with voice assistants. When users issue operation instructions to the terminal device, the voice assistant will give answers or perform quick operations. For example, if a user issues an instruction to create a 5-minute countdown, the terminal will call the ready-made interface in the system to create the countdown.

[0032] However, traditional voice assistants have many disadvantages. First, traditional voice assistants have limited ability to understand user instructions. When users use different ways to give instructions, traditional voice assistants cannot understand the user's intentions well. Secondly, traditional voice assistants can only perform a limited number of control operations. Usually, they need to pre-create interfaces or pre-record methods to achieve automatic control of the terminal, and other control methods that are not within the predefined range cannot be achieved.

[0033] The present application provides a terminal control method, which generates an interface operation instruction sequence corresponding to a natural language instruction through a generative large language model, and performs interface operations in a user interface based on the interface operation instruction sequence to realize the target function indicated by the natural language instruction, thereby realizing automatic control of the terminal.

[0034] Figure 1 It is a schematic diagram of an implementation environment provided by an exemplary embodiment of the present application.

[0035] The implementation environment includes a terminal 110 , a server 120 , and a generative large language model 130 .

[0036] The terminal 110 includes mobile terminals such as smart phones, tablet computers, laptop computers, and vehicle-mounted terminals, and may also be terminals such as desktop computers and projection computers. This specification uses the example of the terminal 110 being a smart phone for schematic description, but does not constitute a limitation to this.

[0037] In some embodiments, the terminal 110 receives a natural language instruction from the user, and executes the interface operation corresponding to the natural language instruction in the user interface to implement the target function indicated by the natural language instruction. For example, the user inputs a natural voice instruction "change the alarm at 6 o'clock tomorrow morning to 6:30" to the terminal 110 through voice, and the terminal 110 clicks the control corresponding to the clock application in the user interface through a simulated operation, and clicks the alarm task at 6 o'clock tomorrow morning to change the time to 6:30.

[0038] The server 120 is a server that provides services for the terminal 110. In some embodiments, the server 120 receives the natural language instruction sent by the terminal 110, processes the natural language instruction, obtains the interface operation instruction sequence, and returns it to the terminal 110. For more information about the interface operation instruction sequence, see Figure 2 And the related description will not be repeated here.

[0039] Server 120 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), as well as big data and artificial intelligence platforms.

[0040] In order to process the natural language instructions to obtain an interface operation instruction sequence, in some embodiments, the server 120 can pass the natural language instructions to the generative large language model 130 through the large language model interface, and generate an interface operation instruction sequence corresponding to the natural language instructions through the generative large language model 130.

[0041] The generative large language model 130 refers to a machine learning model pre-trained on a large-scale text corpus, for example, a generative pre-trained transformer (GPT) model, including GPT-1, GPT-2, GPT-3, GPT-3.5, GPT-4, etc. and their subsequent iterative models.

[0042] The GPT model is a large language model pre-trained by OpenAI based on a huge training sample and parameter system. Taking GPT-3 as an example, its model training layer depth reaches 96 layers, the dimension of the hidden layer reaches 12288, and the number of parameters is as high as 175 billion, which can generate more complex, coherent and creative texts.

[0043] Due to the powerful language understanding and text generation capabilities of the generative large language model 130, the generative large language model 130 can generate an interface operation instruction sequence corresponding to the natural language instruction, and return it to the server 120, and the server 120 returns it to the terminal 110, so that the terminal 110 executes the interface operation indicated by the interface operation instruction sequence in the user interface to realize the target function corresponding to the natural language instruction.

[0044] It should be noted that the above embodiment only describes the general architecture of the implementation environment. The system may also include more or fewer components, or combine certain components, and this embodiment does not limit this.

[0045] See also Figure 2 , Figure 2 1 is a flow chart of a terminal control method provided by an exemplary embodiment of the present application. The method comprises the following steps:

[0046] Step 201 , obtaining a natural language instruction, where the natural language instruction is a natural language content that instructs a terminal to implement a target function.

[0047] In one possible scenario, the natural language instruction may be natural language content that directly instructs the terminal to implement the target function, such as "open the weather application"; in another possible scenario, the natural language instruction may be natural language content that indirectly instructs the terminal to implement the target function, such as "check today's weather conditions."

[0048] Optionally, the natural language instruction may be input by the user into the terminal via voice, or input into the terminal via text.

[0049] The target function refers to the terminal function to be implemented as indicated by the natural voice command. For example, the target functions corresponding to the natural voice commands "check today's weather conditions" or "open the weather application" are both to run the weather application.

[0050] In some embodiments, the natural language instruction is natural language content instructing the terminal to implement the target function. For example, "open the weather application" instructs the terminal to open the weather application in the user interface. In other embodiments, the natural language instruction is natural language content instructing other terminals in a multi-device collaborative mode with the terminal to implement the target function. For example, if a user sends "turn down the air conditioner by 2 degrees" to a smart phone, the smart air conditioner in a multi-device collaborative mode with the smart phone will implement the corresponding target function.

[0051] Step 202, sending an instruction parsing request containing natural language instructions to the server, the server is used to generate an interface operation instruction sequence corresponding to the natural language instructions through a generative large language model, the interface operation instruction sequence includes interface operation instructions executed in sequence, and the interface operation instructions are instructions corresponding to the interface operations supported by the terminal.

[0052] The instruction parsing request is a request for processing natural language instructions to generate an interface operation instruction sequence.

[0053] For the same target function, different types or models of terminals may have different corresponding interface operation methods. For example, to download an application, for a smartphone of model A, you need to open the "Store" operation control, search for the application in the search bar, select the application and download it; for a tablet of model B, you need to open the "App Market" operation control, search for the application in the search bar, select the application and verify the payment information, and download the application.

[0054] Therefore, in some embodiments, in addition to natural language instructions, the instruction parsing request may also include terminal identification information (such as terminal model information, system version number corresponding to the terminal, etc.), so that the server returns an interface operation instruction sequence adapted to the terminal.

[0055] The interface operation instruction sequence includes interface operation instructions that are executed in sequence. For example, the interface operation instruction sequence may be <interface operation instruction 1; interface operation instruction 2; interface operation instruction 3>, which means that interface operation instruction 1 is executed first, then interface operation instruction 2, and then interface operation instruction 3.

[0056] Among them, the interface operation instructions are instructions corresponding to the interface operations supported by the terminal.

[0057] For example, the interface operation supported by the smartphone of model A includes searching for applications in the search bar, and the interface operation instruction may be an instruction corresponding to searching for applications in the search bar.

[0058] Regarding the type of interface operation instructions, in some embodiments, the interface operation instructions can be natural language type instructions, program control type instructions, or other types of instruction forms, and the embodiments of the present application are not limited to this.

[0059] In some embodiments, the interface operation instruction sequence can be generated by a generative large language model based on natural language instructions. In some embodiments, the interface operation instruction sequence can be generated by a generative large language model based on natural language instructions and terminal identification information.

[0060] Exemplarily, based on the natural language content of "shorten the 5-minute countdown by 3 minutes", the generated interface operation instruction sequence can be <open the clock; select a 5-minute countdown; click to edit the countdown duration; change the countdown duration to 2 minutes; click the Save button>.

[0061] Step 203: Receive the interface operation instruction sequence sent by the server.

[0062] In some embodiments, the server receives the interface operation instruction sequence generated by the generative large language model through the large language model interface, and returns the interface operation instruction sequence to the terminal.

[0063] Step 204: Performing interface operations in the user interface based on the interface operation instruction sequence to achieve the target function.

[0064] In some embodiments, the terminal can extract the target control function corresponding to the interface operation instruction from the interface operation instruction sequence, and determine the target control and target operation that match the target control function, and perform the target operation on the target control by calling the automated testing framework to achieve the target function.

[0065] Exemplarily, the target control function corresponding to the interface operation instruction <open clock> is to run the clock application, the target control matching the target control function is the clock application, and the target operation matching the target control function is a click operation.

[0066] In some embodiments, the terminal may implement a target function by calling an automated testing framework and performing interface operations in a user interface through simulated operations.

[0067] Optional automated testing frameworks include Selenium, Appium, and UIAutomator.

[0068] In some embodiments, the terminal may also perform interface operations in the user interface through a predefined operation path to achieve a target function.

[0069] In summary, by generating an interface operation instruction sequence through a generative large language model, the natural language instructions input by the user can be deeply understood and converted into multiple interface operation instructions executed in sequence, wherein the interface operation instructions are instructions corresponding to the interface operations supported by the terminal, so the terminal can perform interface operations based on the interface operation instruction sequence, thereby realizing automatic control of the terminal. The method of the embodiment of the present application can deeply understand the user's intention, greatly expand the scenario of automatic control of the terminal, and realize a variety of different functions.

[0070] Since the interface operation instruction sequence may include multiple interface operation instructions to be executed in sequence, when the terminal executes the interface operation, it will sequentially execute the interface operations corresponding to the multiple interface operation instructions.

[0071] See also Figure 3 , Figure 3 It is a schematic diagram of an execution interface operation provided by an exemplary embodiment of the present application.

[0072] like Figure 3 As shown, the interface operation instruction sequence includes n interface operation instructions executed in sequence, namely the 1st interface operation instruction, the 2nd interface operation instruction, ..., the ith interface operation instruction, the (i+1)th interface operation instruction, ..., the nth interface operation instruction.

[0073] Each interface operation instruction corresponds to a sub-step of executing the interface operation.

[0074] Taking the ith interface operation instruction as an example, the corresponding sub-steps of executing the interface operation include:

[0075] Step 311, obtaining the i-th interface operation instruction in the interface operation instruction sequence, where i is a positive integer.

[0076] Step 312, determining the target control function corresponding to the ith interface operation instruction, where the target control function is the control function corresponding to the control that executes the ith interface operation instruction.

[0077] For example, if the ith interface operation instruction is <Open Clock>, then the control that executes the ith interface operation instruction is the clock control, and its corresponding target control function is to run the clock application. For another example, if the ith interface operation instruction is <Click Save Button>, then the control that executes the ith interface operation instruction is the Save Button, and its corresponding target control function is to confirm and save the current changes.

[0078] In some embodiments, the target control function can be extracted from the i-th interface operation instruction through a target extraction model.

[0079] Optionally, the target extraction model is integrated in the terminal or in the server.

[0080] In some embodiments, the terminal may input the ith interface operation instruction into a target extraction model to extract a target control function corresponding to the ith interface operation instruction.

[0081] In other embodiments, the terminal may also receive a target control function sent by the server, where the target control function is extracted by the server through a target extraction model based on the i-th interface operation instruction.

[0082] In some embodiments, the target extraction model can be trained based on training samples with training labels.

[0083] Step 313: determine a target control and a target operation that match the function of the target control. The target control is a control to be operated in the user interface, and the target operation refers to an operation performed on the target control.

[0084] The target control that matches the target control function refers to the operation control that can realize the target control function in the user interface of the terminal. For example, if the target control function is to run the clock application, the target control is the clock control; for another example, if the target control function is to confirm and save the current modification, the target control is the save button.

[0085] The target operation that matches the function of the target control refers to the control operation on the target control that can realize the function of the target control; for example, if the function of the target control is to run a clock application and the target control is a clock control, then the target operation is a click operation; for another example, if the function of the target control is an unlock function and the target control is a slide to unlock button, then the target operation is a slide operation.

[0086] In some embodiments, an interface element function corresponding to an interface element in a user interface may be determined, and when the interface element function matches the target control function, the interface element may be determined as a target control.

[0087] In some embodiments, a target operation may be determined based on a target control function and a target control.

[0088] For the relevant content of determining the target control and the target operation, please refer to Figure X and its related description, which will not be repeated here.

[0089] Step 314, executing the target operation on the target control by calling the automated testing framework.

[0090] Optional automated testing frameworks include Selenium, Appium, and UIAutomator.

[0091] Exemplarily, the target control function is to run a clock application, the target control is a clock control, and the target operation is a click operation. The terminal can call the Selenium automated testing framework to perform a click operation at the location of the clock control in the user interface to run the clock application.

[0092] Similarly, the sub-steps of executing the interface operation corresponding to the (i+1)th interface operation instruction are similar to those of the (i)th interface operation instruction.

[0093] Step 321, obtaining the (i+1)th interface operation instruction in the interface operation instruction sequence.

[0094] …

[0095] Until the sub-step of executing the interface operation corresponding to the nth interface operation instruction is completed.

[0096] At this point, the terminal has completed the sub-steps corresponding to all the interface operation instructions in the interface operation instruction sequence in order, and therefore, the target function can be achieved.

[0097] To determine the target control that matches the function of the target control, the terminal may identify an interface element from the user interface and determine whether the interface element matches the function of the target control in a variety of ways.

[0098] In some embodiments, the terminal determines an interface element function corresponding to an interface element in the user interface, and when the interface element function matches the target control function, determines the interface element as the target control.

[0099] Among them, the interface element function refers to the function that the interface element can achieve.

[0100] Exemplarily, if the interface element is a trash can icon, the corresponding interface element function is the delete function; or, if the interface element is a clock control, the corresponding interface element function is to run a clock application; or, if the interface element is a save button, the corresponding interface element function is to save information.

[0101] In a possible scenario, the same interface element may have multiple interface element functions. For example, for the same auxiliary control on a tablet computer, the corresponding interface element function when a double-click operation is performed is a screenshot function, and the corresponding interface element function when a long-press operation is performed is a screen recording function.

[0102] Regarding the method of determining the interface element function corresponding to the interface element, it can be achieved through various methods such as image recognition, optical character recognition, and accessibility services.

[0103] See also Figure 4 , Figure 4 It is a schematic diagram of determining a target control that matches the function of the target control provided by an exemplary embodiment of the present application.

[0104] like Figure 4 As shown, in the user interface of terminal 410 , interface elements corresponding to three documents are displayed, for example, a document icon 420 corresponding to document 2.

[0105] When the terminal receives the natural language instruction "delete document 2" input by the user, the terminal sends a command parsing request containing the natural language instruction to the server. The server passes the natural language instruction to the generative large language model through the large language model interface, so that the generative large language model generates an interface operation instruction sequence corresponding to the natural language instruction. For example, the interface operation instruction sequence is <select document 2; click the delete control; click to confirm deletion>, the server sends the interface operation instruction sequence to the terminal, and the terminal performs the interface operation in the user interface 410 based on the interface operation instruction sequence.

[0106] For the interface operation instruction <Select Document 2>, the terminal determines that the target control function is Select Document 2 through the target extraction model.

[0107] In some embodiments, terminal 410 may determine the accessibility tag corresponding to the interface element through an accessibility service. For example, terminal 410 may determine the accessibility tag of document icon 420 as <Document 2> through an accessibility service.

[0108] In some embodiments, the terminal may determine the interface element function corresponding to the interface element based on the accessibility tag. For example, the terminal may determine the interface element function corresponding to the document icon 420 for <document 2> based on the accessibility tag, including: selecting document 2, opening document 2, editing document 2, moving document 2, etc.

[0109] When the interface element function “Select Document 2” corresponding to the document icon 420 matches the target control function “Select Document 2”, the document icon 420 is determined as the target control.

[0110] In some embodiments, the terminal may also determine the target operation based on the target control function and the target control. For example, when the target control function is "select document 2" and the target control is the document icon 420, the terminal may determine that the target operation is a long press operation based on the pre-set operation correspondence between the selection operation and the control.

[0111] Therefore, when the target control is the document icon 420 and the target operation is a long press operation, the terminal calls the automated testing framework and performs a long press operation at the location of the document icon 420 to implement the function of selecting document 2.

[0112] In some embodiments, after the sub-step of executing the interface operation corresponding to the i-th interface operation instruction is completed, the interface update status of the user interface can be determined; when the interface update status represents a user interface update, the i+1-th interface operation instruction in the interface operation instruction sequence is obtained.

[0113] Optionally, the interface update status can be determined based on the difference between the user interface image before the sub-step of executing the interface operation and the user interface image after the sub-step of executing the interface operation.

[0114] Optionally, when the interface update status represents a user interface update, the terminal may further play a progress prompt voice to prompt the user that the sub-step of the interface operation corresponding to the (i+1)th interface operation instruction has been executed.

[0115] For example, before the terminal performs a long press operation on the location where the document icon 420 is located, a screenshot of the user interface can be taken to obtain a first screen; after the terminal performs a long press operation on the location where the document icon 420 is located, a screenshot of the user interface can be taken to obtain a second screen; wherein the second screen displays a delete control 430. When there is a difference between the first screen and the second screen, it is determined that the interface update situation indicates that the user interface is updated, and thus the interface operation instruction <click to delete the control> is obtained.

[0116] For the interface operation instruction <click to delete control>, the terminal determines that the function of the target control is to delete document 2 through the target extraction model.

[0117] In some embodiments, terminal 410 may perform image recognition on the interface element, and determine the interface element function corresponding to the interface element based on the image recognition result.

[0118] In some embodiments, the terminal 410 extracts the interface elements in the second screen through a target detection model (You Only Look Once, YOLO), and obtains icons corresponding to document 1, document 2, document 3, and a delete control 430.

[0119] In some embodiments, the terminal may determine the interface element functions corresponding to the above interface elements through template matching or model recognition methods.

[0120] Exemplarily, the terminal may perform similarity matching between the trash can pattern of the delete control 430 and the patterns corresponding to the various functional type controls pre-stored in the template library, so as to determine that the interface element function corresponding to the delete control 430 is the delete function.

[0121] When the interface element function “delete function” corresponding to the delete control 430 matches the target control function “delete document 2”, the delete control 430 is determined as the target control.

[0122] In some embodiments, the terminal may also determine the target operation based on the target control function and the target control. For example, when the target control function is "delete document 2" and the target control is delete control 430, the terminal may determine that the target operation is a click operation based on the preset delete operation and the operation correspondence between the controls.

[0123] Therefore, when the target control is the delete control 430 and the target operation is a click operation, the terminal calls the automated testing framework and performs a click operation at the location where the delete control 430 is located to implement the function of deleting document 2 .

[0124] After the click operation is performed at the location where the delete control 430 is located, the user interface is updated to a third screen, which includes a prompt bar 440, and the prompt bar 440 includes a cancel control 441 and a confirmation control 442. For the interface operation instruction <click to confirm deletion>, the terminal determines that the target control function is to confirm deletion of document 2 through the target extraction model.

[0125] In some embodiments, terminal 410 may perform optical character recognition on the interface elements, extract the text contained in the interface elements, and determine the interface element function corresponding to the interface element based on the text.

[0126] Exemplarily, the terminal can perform optical character recognition on cancel control 441 and confirmation control 442, and extract that the text contained in cancel control 441 is "Cancel", and its corresponding interface element function is "Cancel deletion"; the text contained in confirmation control 442 is "Confirm", and its corresponding interface element function is "Confirm deletion".

[0127] Among them, the interface element function "Confirm deletion" corresponding to the confirmation control 442 matches the target control function "Confirm deletion of document 2", so the confirmation control 442 can be determined as the target control.

[0128] In some embodiments, the terminal may also determine the target operation based on the target control function and the target control. For example, when the target control function is "confirm deletion of document 2" and the target control is confirmation control 442, the terminal may determine that the target operation is a click operation based on the preset corresponding relationship between the confirmation deletion operation and the control.

[0129] Therefore, when the target control is the confirmation control 442 and the target operation is a click operation, the terminal calls the automated testing framework and performs a click operation at the location of the confirmation control 442 to implement the function of confirming the deletion of the document 2 .

[0130] At this point, all the interface operation sub-steps corresponding to the interface operation instructions in the interface operation instruction sequence have been executed. The document icon 420 displayed in the user interface of the terminal 410 has been deleted.

[0131] The method of determining the interface element function corresponding to the interface element includes at least one of the following methods.

[0132] (1) In some embodiments, the terminal performs image recognition on the interface element and determines the interface element function corresponding to the interface element based on the image recognition result.

[0133] For example Figure 4 In the example, the terminal performs image recognition on the delete control 430, and performs similarity matching between the recognized trash can pattern and the patterns corresponding to the functional type controls pre-stored in the template library, so as to determine that the interface element function corresponding to the delete control 430 is the delete function.

[0134] (2) In some embodiments, the terminal performs optical character recognition on the interface elements, extracts the text contained in the interface elements, and determines the interface element function corresponding to the interface element based on the text.

[0135] For example Figure 4In the figure, the terminal performs optical character recognition on the cancel control 441 and the confirmation control 442, and extracts that the text contained in the cancel control 441 is "Cancel", and its corresponding interface element function is "Cancel Delete"; the text contained in the confirmation control 442 is "Confirm", and its corresponding interface element function is "Confirm Delete".

[0136] (3) In some embodiments, the terminal determines the accessibility tag corresponding to the interface element through the accessibility service, and determines the interface element function corresponding to the interface element based on the accessibility tag.

[0137] like Figure 4 In the example, the terminal determines that the interface element functions corresponding to the document icon 420 for <document 2> include: selecting document 2, opening document 2, editing document 2, moving document 2, etc. based on the accessibility tag.

[0138] See also Figure 5 , Figure 5 This is a flow chart of determining the interface element function corresponding to an interface element provided by an exemplary embodiment of the present application. The flow chart includes the following steps.

[0139] Step 501: Determine an accessibility tag corresponding to an interface element through an accessibility service, and determine an interface element function corresponding to the interface element based on the accessibility tag.

[0140] Step 502, when no accessibility label can be obtained, optical character recognition is performed on the interface element to extract the text contained in the interface element, and the interface element function corresponding to the interface element is determined based on the text.

[0141] Step 503: When the text cannot be extracted, image recognition is performed on the interface element, and the interface element function corresponding to the interface element is determined based on the image recognition result.

[0142] In this embodiment, when the accessibility label cannot be obtained, optical character recognition is used, and when the text cannot be extracted, image recognition is used to optimize the logic of determining the function of the interface element, reduce the complexity of determining the function of the interface element, and thus improve the efficiency of the control terminal.

[0143] In some embodiments, the terminal may include an eye tracking component. The eye tracking component is used to record the eye movement trajectory characteristics of the user when processing visual information and output an eyeball image. Through the eye tracking component, the area where the user's gaze stays, that is, the human eye gaze area, can be determined based on the eyeball image.

[0144] Optionally, the eye tracking component includes an eye tracking device based on the pupil-corneal reflection method, and eye tracking is performed through a line connecting the center of the cornea and the center of the pupil.

[0145] Optionally, the eye tracking component includes an eye tracking device based on capturing eye images by transmitting eye images using mirrors or optical waveguides.

[0146] In some embodiments, the eye tracking component can also perform eye tracking based on other methods, such as calculating the gaze point based on retinal images, reconstructing a three-dimensional model, retinal / corneal reflection method, etc., and the embodiments of the present application do not impose any restrictions on this.

[0147] In a possible implementation, the target control may be determined in combination with the location of the area where the human eye is gazing.

[0148] In some embodiments, when the function of the interface element matches the function of the target control and the interface element corresponding to the interface element is located within the human eye gaze area, the interface element can be directly determined as the target control.

[0149] See also Figure 6 , Figure 6 It is a schematic diagram of determining a target control in combination with a human eye gaze area provided by an exemplary embodiment of the present application.

[0150] like Figure 6 As shown, when the target control function is "delete document 2", the terminal 610 determines that the interface element function corresponding to the delete control 630 displayed on the user interface is the delete function through image recognition.

[0151] The terminal may determine the human eye gaze area 620 based on the eyeball image output by the eye tracking component.

[0152] When the interface element function corresponding to the delete control 630 matches the target control function and the delete control 630 is located within the human eye gaze area 620, the delete control 630 may be determined as the target control.

[0153] In some embodiments, when the function of an interface element matches the function of a target control and the interface element is outside the area where the human eye is looking, the confidence level corresponding to the interface element can be determined based on the distance between the interface element and the area where the human eye is looking; when the confidence level meets the confidence level requirements, the interface element is determined as the target control.

[0154] Among them, the distance between the interface element and the human eye's gaze area is negatively correlated with the confidence level corresponding to the interface element.

[0155] For example, the upper right corner of the user interface also includes a delete button 640 , and the terminal 610 determines, through image recognition, that the interface element function corresponding to the delete button 640 is also a delete function.

[0156] However, the position of the delete button 640 is far away from the human eye gaze area 620, so the confidence corresponding to the delete button 640 is low. When the confidence is lower than the confidence threshold, the terminal will not use the delete button 640 as a target control.

[0157] In this embodiment, the target control is determined in combination with the human eye gaze area, which can improve the accuracy of the target control determination, thereby further improving the quality of terminal control.

[0158] To improve the recognition efficiency of the target control, the terminal can also recognize the interface elements only within the area where the human eye is looking.

[0159] In some embodiments, the terminal determines the area where the human eye is looking based on the eye image output by the eye tracking component, and determines the interface element functions corresponding to the interface elements in the area where the human eye is looking.

[0160] For example, the terminal may determine the interface element function only for the delete control 630 in the human eye gaze area 620. When the interface element function "delete function" matches the target control function, the delete control 630 is determined as the target control.

[0161] By identifying interface elements only in the area where the human eye is looking, there is no need to determine the interface element functions of all interface elements in the user interface, thereby reducing the amount of calculation, improving the efficiency of terminal control, and ensuring the accuracy of target control determination.

[0162] For the same target function, in some possible scenarios, the terminal may adopt at least two sets of different interface operation instruction sequences to implement it.

[0163] For example, for the target function "turn on Bluetooth", the terminal can take the interface operation instruction sequence A: <Open system settings; select Bluetooth connection; click the turn on button> to achieve the target function, or it can take the interface operation instruction sequence B: <Slide down to display the toolbar; click the Bluetooth icon> to achieve the target function.

[0164] In some embodiments, the generative large language model can output at least two groups of interface operation instruction sequences based on natural language instructions and send them to the server, and the server sends the at least two groups of interface operation instruction sequences to the terminal.

[0165] When receiving at least two groups of interface operation instruction sequences sent by the server, the terminal determines the length of the operation path corresponding to each group of interface operation instruction sequences.

[0166] In some embodiments, the length of the operation path is equal to the number of interface operation instructions contained in the interface operation instruction sequence. For example, the length of the operation path of the interface operation instruction sequence A: <Open system settings; Select Bluetooth connection; Click the turn on button> is 3; the length of the operation path of the interface operation instruction sequence B: <Slide down to display the toolbar; Click the Bluetooth icon> is 2.

[0167] In some embodiments, the terminal performs interface operations in the user interface based on a sequence of interface operation instructions corresponding to the shortest operation path.

[0168] For example, for the target function "turn on Bluetooth", the terminal performs an interface operation in the user interface based on the interface operation instruction sequence B.

[0169] In this embodiment, performing interface operations in the user interface based on the interface operation instruction sequence corresponding to the shortest operation path can reduce the number of interface operations executed and improve the execution efficiency of the interface operations, thereby achieving the target function more quickly.

[0170] In a possible scenario, the terminal may be in a multi-device collaboration mode with at least one other terminal, and in the multi-device collaboration mode, the terminal controls the terminal or other terminals to achieve target functions.

[0171] In this scenario, the natural language instruction is the natural language content that instructs the local terminal or other terminals to implement the target function.

[0172] For example, when a smartphone and a tablet are in multi-device collaboration mode, the user can input a natural language command to the smartphone, "connect the tablet to Wi-Fi," and the smartphone will automatically control the tablet through the multi-device collaboration mode.

[0173] For another example, when a smartphone and a smart air conditioner are in multi-device collaboration mode, the user can input a natural language command into the smartphone, "lower the air conditioner by two degrees," and the smartphone will then automatically control the smart air conditioner through the multi-device collaboration mode.

[0174] Among them, other terminals can be mobile terminals such as smart phones, tablets, laptop computers, car-mounted terminals, etc., they can also be terminals such as desktop computers, projection computers, etc., and they can also be smart home appliances such as smart TVs, smart air conditioners, smart refrigerators and smart sweeping robots. The embodiments of this application do not impose any restrictions on the specific types of other terminals.

[0175] In some embodiments, before the terminal sends an instruction parsing request including a natural language instruction to the server, the terminal can determine a target terminal for implementing a target function based on the natural language instruction.

[0176] For example, if the natural language instruction is "connect the tablet to the wifi", the target terminal for achieving the target function is determined to be a tablet computer that is in multi-device collaboration mode with the terminal; if the natural language instruction is "lower the air conditioner by two degrees", the target terminal for achieving the target function is determined to be a smart air conditioner that is in multi-device collaboration mode with the terminal.

[0177] When the target terminal is the local terminal, the terminal sends a command parsing request containing natural language instructions and the identification information of the local terminal to the server. The server generates an interface operation instruction sequence corresponding to the local terminal through a generative large language model, and sends the interface operation instruction sequence to the local terminal, so that the local terminal executes the target function based on the interface operation instruction sequence.

[0178] The local terminal identification information is information used to identify the terminal. For example, the local terminal identification information may be the model information, system version number information, etc. of the terminal.

[0179] When the target terminal is another terminal, an instruction parsing request containing natural language instructions and other terminal identification information is sent to the server, so that the server generates an interface operation instruction sequence corresponding to the other terminal through a generative large language model, and sends the interface operation instruction sequence to the other terminal, so that the other terminal can execute the target function based on the interface operation instruction sequence.

[0180] The other terminal identification information is information used to identify other terminals in the multi-device collaborative mode with the terminal. For example, the other terminal identification information may be model information, system version number information, etc. of other terminals in the multi-device collaborative mode with the terminal.

[0181] See also Figure 7 , Figure 7 It is a schematic diagram of a control terminal in a multi-device collaborative mode provided by an exemplary embodiment of the present application.

[0182] like Figure 7 As shown, the user inputs a natural language instruction into the terminal 710, and the terminal 710 determines, based on the natural language instruction, that the target terminal for implementing the target function is the terminal 710 or another terminal 720. Among them, the other terminal 720 is a terminal in a multi-device collaboration mode with the terminal 710.

[0183] Optionally, the terminal 710 may extract a target terminal from a natural language instruction based on natural language processing technology.

[0184] When the target terminal is the local terminal (terminal 710), the terminal 710 sends an instruction parsing request A1 containing natural language instructions and local terminal identification information to the server 730, so that the server 730 generates an interface operation instruction sequence A2 corresponding to the local terminal through the generative large language model 740, and sends the interface operation instruction sequence A2 to the terminal 710, so that the terminal 710 executes the target function based on the interface operation instruction sequence A2.

[0185] When the target terminal is another terminal 720, the terminal 710 sends an instruction parsing request B1 containing natural language instructions and other terminal identification information to the server 730, so that the server 730 generates an interface operation instruction sequence B2 corresponding to the other terminal 720 through the generative large language model 740, and sends the interface operation instruction sequence B2 to the other terminal 720, so that the other terminal 720 executes the target function based on the interface operation instruction sequence B2.

[0186] In this embodiment, when the terminal is in a multi-device collaborative mode with other terminals, the target terminal is determined based on natural language instructions. When the target terminal is other terminals, an instruction parsing request containing identification information of other terminals is sent to the server, so that the server can send an interface operation instruction sequence to other terminals, thereby realizing collaborative control of multiple devices and further expanding the usage scenarios of terminal control.

[0187] See also Figure 8 , Figure 8 This is a structural block diagram of a terminal control device provided by an exemplary embodiment of the present application. The device includes:

[0188] An acquisition module 801 is used to acquire a natural language instruction, where the natural language instruction is a natural language content that instructs the terminal to implement a target function;

[0189] A sending module 802 is used to send an instruction parsing request including the natural language instruction to a server, wherein the server is used to generate an interface operation instruction sequence corresponding to the natural language instruction through a generative large language model, wherein the interface operation instruction sequence includes interface operation instructions to be executed in sequence, and the interface operation instructions are instructions corresponding to the interface operations supported by the terminal;

[0190] The receiving module 803 is used to receive the interface operation instruction sequence sent by the server;

[0191] The execution module 804 is used to execute the interface operation in the user interface based on the interface operation instruction sequence to realize the target function.

[0192] Optionally, the execution module 804 is used to:

[0193] Obtaining the i-th interface operation instruction in the interface operation instruction sequence, where i is a positive integer;

[0194] Determine a target control function corresponding to the i-th interface operation instruction, where the target control function is a control function corresponding to a control that executes the i-th interface operation instruction;

[0195] Determine a target control and a target operation that match the function of the target control, wherein the target control is a control to be operated in the user interface, and the target operation refers to an operation performed on the target control;

[0196] By calling the automated testing framework, the target operation is performed on the target control;

[0197] Obtain the (i+1)th interface operation instruction in the interface operation instruction sequence.

[0198] Optionally, the execution module 804 is used to:

[0199] Input the i-th interface operation instruction into the target extraction model to extract the target control function corresponding to the i-th interface operation instruction; or,

[0200] The target control function sent by the server is received, where the target control function is extracted by the server through the target extraction model based on the i-th interface operation instruction.

[0201] Optionally, the execution module 804 is used to:

[0202] Determining an interface element function corresponding to an interface element in the user interface;

[0203] In the case where the function of the interface element matches the function of the target control, the interface element is determined as the target control.

[0204] Optionally, the execution module 804 is used to:

[0205] Performing image recognition on the interface element, and determining the interface element function corresponding to the interface element based on the image recognition result; or,

[0206] Performing optical character recognition on the interface element to extract the text contained in the interface element, and determining the interface element function corresponding to the interface element based on the text; or,

[0207] An accessibility label corresponding to the interface element is determined through an accessibility service, and a function of the interface element corresponding to the interface element is determined based on the accessibility label.

[0208] Optionally, the terminal includes an eye tracking component, an execution module 804, configured to:

[0209] Determine the gaze area of ​​the human eye based on the eyeball image output by the eye tracking component;

[0210] In the case where the function of the interface element matches the function of the target control, and the interface element corresponding to the interface element is located within the human eye gaze area, the interface element is determined as the target control.

[0211] Optionally, the execution module 804 is used to:

[0212] When the function of the interface element matches the function of the target control and the interface element is outside the human eye gaze area, determining the confidence level corresponding to the interface element based on the distance between the interface element and the human eye gaze area;

[0213] When the confidence level meets the confidence level requirement, the interface element is determined as the target control.

[0214] Optionally, the execution module 804 is used to:

[0215] Determine the gaze area of ​​the human eye based on the eyeball image output by the eye tracking component;

[0216] Determine the interface element function corresponding to the interface element in the human eye gaze area.

[0217] Optionally, the execution module 804 is used to:

[0218] Determining an interface update status of the user interface;

[0219] When the interface update situation represents an update of the user interface, an i+1th interface operation instruction in the interface operation instruction sequence is obtained.

[0220] Optionally, the execution module 804 is used to:

[0221] Upon receiving at least two groups of the interface operation instruction sequences sent by the server, determining the length of the operation path corresponding to each group of the interface operation instruction sequences;

[0222] An interface operation is performed in the user interface based on the interface operation instruction sequence corresponding to the shortest operation path.

[0223] Optionally, the terminal and at least one other terminal are in a multi-device collaboration mode, and in the multi-device collaboration mode, the terminal controls the terminal or the other terminal to implement the target function; the device further includes a target terminal determination module, which is used to:

[0224] Based on the natural language instruction, a target terminal for implementing the target function is determined.

[0225] Optionally, the sending module 802 is used to:

[0226] In the case where the target terminal is the local end, sending the instruction parsing request including the natural language instruction and the identification information of the local end to the server; so that the server generates an interface operation instruction sequence corresponding to the local end through the generative large language model, and sends the interface operation instruction sequence to the local end, so that the local end executes the target function based on the interface operation instruction sequence;

[0227] When the target terminal is another terminal, an instruction parsing request including the natural language instruction and the identification information of the other terminal is sent to the server, so that the server generates an interface operation instruction sequence corresponding to the other terminal through a generative large language model, and sends the interface operation instruction sequence to the other terminal, so that the other terminal executes the target function based on the interface operation instruction sequence.

[0228] Please refer to Fig. 9 , Fig. 9 It is a structural block diagram of a terminal provided by an exemplary embodiment of the present application.

[0229] The terminal can execute the terminal control method of the above embodiment. The terminal can be a smart phone, a laptop, a tablet computer, a navigation device, a smart TV, etc. The terminal may also be called a user equipment, a portable terminal, etc. The terminal may include one or more of the following components: a processor 910 and a memory 920.

[0230] Optionally, the processor 910 uses various interfaces and lines to connect various parts of the entire electronic device, and executes various functions of the electronic device and processes data by running or executing instructions, programs, code sets or instruction sets stored in the memory 920, and calling data stored in the memory 920. Optionally, the processor 910 can be implemented in at least one hardware form of digital signal processing (DSP), field-programmable gate array (FPGA), and programmable logic array (PLA). The processor 910 can integrate one or a combination of a central processing unit (CPU), a graphics processing unit (GPU), a neural network processor (NPU), and a baseband chip. Among them, the CPU mainly processes the operating system, user interface, and application programs; the GPU is responsible for rendering and drawing the content that needs to be displayed on the touch display; the NPU is used to implement artificial intelligence (AI) functions; and the baseband chip is used to process wireless communications. It is understandable that the above baseband chip may not be integrated into the processor 910, but may be implemented by a separate chip.

[0231] The memory 920 may include a random access memory (RAM) or a read-only memory (ROM). Optionally, the memory 920 includes a non-transitory computer-readable storage medium. The memory 920 may be used to store instructions, programs, codes, code sets, or instruction sets. The memory 920 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the following various method embodiments, etc.; the data storage area may store data created according to the use of the electronic device (such as audio data, a phone book), etc.

[0232] In addition, those skilled in the art will appreciate that the structure of the electronic device shown in the above figures does not constitute a limitation on the electronic device, and the electronic device may include more or fewer components than shown in the figures, or a combination of certain components, or a different arrangement of components.

[0233] The embodiment of the present application also provides a computer-readable storage medium, in which at least one instruction is stored, and the at least one instruction is loaded and executed by a processor to implement the method provided in the above embodiment. Optionally, the computer-readable storage medium may include: ROM, RAM, solid state drive (SSD, Solid State Drives) or optical disk, etc. Among them, RAM may include resistance random access memory (ReRAM, Resistance Random Access Memory) and dynamic random access memory (DRAM, Dynamic Random Access Memory).

[0234] The embodiment of the present application also provides a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. The processor of the computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the method provided in various optional implementations of the above aspects.

[0235] The above description is only an optional embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A terminal control method, characterized in that: The method comprises: Acquire a natural language instruction, where the natural language instruction is natural language content that instructs the terminal to implement a target function; Sending an instruction parsing request including the natural language instruction to a server, wherein the server is used to generate an interface operation instruction sequence corresponding to the natural language instruction through a generative large language model, wherein the interface operation instruction sequence includes interface operation instructions to be executed in sequence, and the interface operation instructions are instructions corresponding to the interface operations supported by the terminal; Receiving the interface operation instruction sequence sent by the server; An interface operation is performed in a user interface based on the interface operation instruction sequence to achieve the target function.

2. The method according to claim 1, characterized in that The performing the interface operation in the user interface based on the interface operation instruction sequence includes: Obtaining the i-th interface operation instruction in the interface operation instruction sequence, where i is a positive integer; Determine a target control function corresponding to the i-th interface operation instruction, where the target control function is a control function corresponding to a control that executes the i-th interface operation instruction; Determine a target control and a target operation that match the function of the target control, wherein the target control is a control to be operated in the user interface, and the target operation refers to an operation performed on the target control; By calling the automated testing framework, the target operation is performed on the target control; Obtain the (i+1)th interface operation instruction in the interface operation instruction sequence.

3. The method according to claim 2, characterized in that The determining of the target control function corresponding to the i-th interface operation instruction includes: Input the i-th interface operation instruction into the target extraction model to extract the target control function corresponding to the i-th interface operation instruction; or, The target control function sent by the server is received, where the target control function is extracted by the server through the target extraction model based on the i-th interface operation instruction.

4. The method according to claim 2, characterized in that The determining a target control that matches the function of the target control includes: Determining an interface element function corresponding to an interface element in the user interface; In the case where the function of the interface element matches the function of the target control, the interface element is determined as the target control.

5. The method according to claim 4, characterized in that The determining of the interface element function corresponding to the interface element in the user interface includes: Performing image recognition on the interface element, and determining the interface element function corresponding to the interface element based on the image recognition result; or, Performing optical character recognition on the interface element to extract the text contained in the interface element, and determining the interface element function corresponding to the interface element based on the text; or, An accessibility label corresponding to the interface element is determined through an accessibility service, and a function of the interface element corresponding to the interface element is determined based on the accessibility label.

6. The method according to claim 4, characterized in that The terminal includes an eye tracking component, and when the function of the interface element matches the function of the target control, determining the interface element as the target control includes: Determine the gaze area of ​​the human eye based on the eyeball image output by the eye tracking component; In the case where the function of the interface element matches the function of the target control, and the interface element corresponding to the interface element is located within the human eye gaze area, the interface element is determined as the target control.

7. The method according to claim 6, characterized in that The method further comprises: When the function of the interface element matches the function of the target control and the interface element is outside the human eye gaze area, determining the confidence level corresponding to the interface element based on the distance between the interface element and the human eye gaze area; When the confidence level meets the confidence level requirement, the interface element is determined as the target control.

8. The method according to claim 4, characterized in that The terminal includes an eye tracking component, and the determining the interface element function corresponding to the interface element in the user interface includes: Determine the gaze area of ​​the human eye based on the eyeball image output by the eye tracking component; Determine the interface element function corresponding to the interface element in the human eye gaze area.

9. The method according to claim 2, characterized in that: The obtaining the (i+1)th interface operation instruction in the interface operation instruction sequence includes: Determining an interface update status of the user interface; When the interface update situation represents an update of the user interface, an i+1th interface operation instruction in the interface operation instruction sequence is obtained.

10. The method according to claim 1, characterized in that The performing the interface operation in the user interface based on the interface operation instruction sequence includes: Upon receiving at least two groups of the interface operation instruction sequences sent by the server, determining the length of the operation path corresponding to each group of the interface operation instruction sequences; An interface operation is performed in the user interface based on the interface operation instruction sequence corresponding to the shortest operation path.

11. The method according to claim 1, characterized in that: The terminal and at least one other terminal are in a multi-device collaboration mode, and in the multi-device collaboration mode, the terminal controls the terminal or the other terminal to implement the target function; Before sending the instruction parsing request including the natural language instruction to the server, the method further includes: Based on the natural language instruction, determining a target terminal for implementing the target function; The sending the instruction parsing request including the natural language instruction to the server includes: In the case where the target terminal is the local end, sending the instruction parsing request including the natural language instruction and the identification information of the local end to the server, so that the server generates an interface operation instruction sequence corresponding to the local end through the generative large language model, and sends the interface operation instruction sequence to the local end, so that the local end executes the target function based on the interface operation instruction sequence; When the target terminal is another terminal, an instruction parsing request including the natural language instruction and the identification information of the other terminal is sent to the server, so that the server generates an interface operation instruction sequence corresponding to the other terminal through a generative large language model, and sends the interface operation instruction sequence to the other terminal, so that the other terminal executes the target function based on the interface operation instruction sequence.

12. A terminal control device, characterized in that: The device comprises: An acquisition module, used to acquire a natural language instruction, wherein the natural language instruction is a natural language content instructing the terminal to implement a target function; A sending module, used to send an instruction parsing request containing the natural language instruction to a server, wherein the server is used to generate an interface operation instruction sequence corresponding to the natural language instruction through a generative large language model, wherein the interface operation instruction sequence includes interface operation instructions to be executed in sequence, and the interface operation instructions are instructions corresponding to the interface operations supported by the terminal; A receiving module, used for receiving the interface operation instruction sequence sent by the server; An execution module is used to execute interface operations in a user interface based on the interface operation instruction sequence to achieve the target function.

13. A terminal, characterized in that: The terminal includes a processor and a memory; the memory stores at least one instruction, and the at least one instruction is used to be executed by the processor to implement the terminal control method according to any one of claims 1 to 11.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores at least one instruction, and the instruction is loaded and executed by the processor to implement the terminal control method according to any one of claims 1 to 11.

15. A computer program product, characterized in that The computer program product includes computer instructions, which are stored in a computer-readable storage medium; a processor of the terminal reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the terminal executes the terminal control method as described in any one of claims 1 to 11.

Images