Method and device for automatic operation of terminal equipment

Abstract

The embodiment of the invention discloses a method and a device for automatic operation of terminal equipment. In the embodiment of the invention, user instruction information is acquired; generating a virtual display screen at the background of the terminal equipment according to the user instruction information; an application program corresponding to the user instruction information is loaded on the virtual display screen; obtaining a target image displayed on the virtual display screen at the current moment; inputting the user instruction information and the target image into a large language model to generate an automatic operation step; and executing the automatic operation step on the virtual display screen. By means of the method, automatic operation of the terminal equipment can be achieved on the virtual display screen under the condition that the screen of the user terminal equipment is not occupied, and the user experience is improved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and more specifically, to a method and device for automated operation of terminal equipment. Background Art

[0002] In recent years, with the development of artificial intelligence technology, various smart terminal devices have entered people's daily lives, providing a lot of convenience for people's lives. Taking smartphones as an example, users' work, life, travel, shopping and entertainment can all be achieved through various applications in smartphones. When users are busy, they need smartphones to realize automated operations and save their time.

[0003] In the prior art, with the widespread application of Large Language Model (LLM) technology, the above-mentioned LLM has made significant progress in understanding natural language and executing complex instructions. The above-mentioned LLM can understand the content on the screen of the terminal device and perform operations on the mobile device according to the user's instructions, such as purchasing goods on a shopping website. However, since LLM needs to analyze the screen content, understand the user's instructions and generate operation steps, the whole process takes a long time. During this period, the automated operation process will occupy the mobile phone screen and system resources, interfere with the user's normal use of the mobile phone, and affect the user experience.

[0004] In summary, how to realize the automated operation of the terminal device without occupying the screen of the user's terminal device and improve the user experience is a problem that needs to be solved at present. Summary of the invention

[0005] In view of this, an embodiment of the present invention provides a method and apparatus for automatic operation of a terminal device, which can realize automatic operation of a terminal device without occupying the screen of the user's terminal device, thereby improving the user experience.

[0006] In a first aspect, an embodiment of the present invention provides a method for automated operation of a terminal device, the method comprising:

[0007] Get user command information;

[0008] Generate a virtual display screen in the background of the terminal device according to the user instruction information;

[0009] Loading the application corresponding to the user instruction information on the virtual display screen;

[0010] Acquire the target image displayed on the virtual display screen at the current moment;

[0011] Inputting the user instruction information and the target image into a large language model to generate automated operation steps;

[0012] The automated operation steps are performed on the virtual display screen.

[0013] Optionally, the method further includes:

[0014] The target image displayed on the display screen is updated according to the automated operation steps.

[0015] Optionally, the method further includes:

[0016] Determining a result generated by the automated operation step;

[0017] The result is sent to the real display screen of the terminal device.

[0018] Optionally, the method further includes:

[0019] Confirmation feedback information sent by the user is received, wherein the confirmation feedback information indicates that the user confirms the generated result.

[0020] Optionally, the method further includes:

[0021] A modification instruction message sent by a user is received, wherein the modification instruction message indicates that the user modifies the generated result.

[0022] Optionally, the method further includes:

[0023] The virtual display screen is displayed in the form of a small window on the real display screen of the terminal device, wherein the small window occupies a partial area of ​​the real display screen.

[0024] Optionally, loading the application corresponding to the user instruction information on the virtual display screen specifically includes:

[0025] Analyzing the user instruction information according to the large language model to determine the application corresponding to the user instruction information;

[0026] The application is loaded on the virtual display screen.

[0027] In a second aspect, an embodiment of the present invention provides a device for automatic operation of a terminal device, the device comprising:

[0028] An acquisition unit, used for acquiring user instruction information;

[0029] A generating unit, configured to generate a virtual display screen in the background of a terminal device according to the user instruction information;

[0030] A processing unit, configured to load an application corresponding to the user instruction information on the virtual display screen;

[0031] The acquisition unit is also used to acquire the target image displayed on the virtual display screen at the current moment;

[0032] The generating unit is further used to input the user instruction information and the target image into a large language model to generate automated operation steps;

[0033] An execution unit is used to execute the automated operation steps on the virtual display screen.

[0034] Optionally, the device further comprises:

[0035] An updating unit is used to update the target image displayed on the display screen according to the automated operation steps.

[0036] Optionally, the device further comprises:

[0037] A determination unit, used to determine the result generated by the automated operation step;

[0038] A sending unit is used to send the result to the real display screen of the terminal device.

[0039] Optionally, the device further comprises:

[0040] The receiving unit is used to receive confirmation feedback information sent by the user, wherein the confirmation feedback information indicates that the user confirms the generated result.

[0041] Optionally, the receiving unit is further used for:

[0042] A modification instruction message sent by a user is received, wherein the modification instruction message indicates that the user modifies the generated result.

[0043] Optionally, the device further comprises:

[0044] The display unit is used to display the virtual display screen in the form of a small window on the real display screen of the terminal device, wherein the small window occupies a partial area of ​​the real display screen.

[0045] Optionally, the processing unit is specifically used for:

[0046] Analyzing the user instruction information according to the large language model to determine the application corresponding to the user instruction information;

[0047] The application is loaded on the virtual display screen.

[0048] In a third aspect, an embodiment of the present invention provides an electronic device, comprising a memory and a processor, the memory being used to store one or more computer program instructions, wherein the one or more computer program instructions are executed by the processor to implement a method as described in the first aspect or any possible embodiment of the first aspect.

[0049] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium storing computer program instructions, which, when executed by a processor, implement a method as described in the first aspect or any possible embodiment of the first aspect.

[0050] In an embodiment of the present invention, user instruction information is obtained; a virtual display screen is generated in the background of a terminal device according to the user instruction information; an application corresponding to the user instruction information is loaded on the virtual display screen; a target image displayed on the virtual display screen at the current moment is obtained; the user instruction information and the target image are input into a large language model to generate automated operation steps; and the automated operation steps are executed on the virtual display screen. Through the above method, the corresponding application can be loaded on the virtual display screen and the automated operation steps can be executed without occupying the screen of the user's terminal device, thereby realizing automated operation of the terminal device and improving the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] The above and other objects, features and advantages of the present invention will become more apparent through the following description of the embodiments of the present invention with reference to the accompanying drawings, in which:

[0052] Figure 1 is a flow chart of a method for automated operation of a terminal device in an embodiment of the present invention;

[0053] Figure 2 is a schematic diagram of a virtual display screen in an embodiment of the present invention;

[0054] Figure 3 is a schematic diagram of another virtual display screen in an embodiment of the present invention;

[0055] Figure 4 is a flow chart of another method for automated operation of a terminal device in an embodiment of the present invention;

[0056] Figure 5 is a flow chart of another method for automated operation of a terminal device in an embodiment of the present invention;

[0057] Figure 6 is a schematic diagram of a virtual display screen in an embodiment of the present invention;

[0058] Figure 7is a flow chart of another method for automated operation of a terminal device in an embodiment of the present invention;

[0059] Figure 8 is a flow chart of another method for automated operation of a terminal device in an embodiment of the present invention;

[0060] Fig. 9 is a schematic diagram of another virtual display screen in an embodiment of the present invention;

[0061] Fig.10 is a system schematic diagram of automated operation of a terminal device in an embodiment of the present invention;

[0062] Fig.11 is a schematic diagram of a display screen of a terminal device in an embodiment of the present invention;

[0063] Fig.12 is a schematic diagram of a device for automated operation of a terminal device in an embodiment of the present invention;

[0064] Fig.13 is a schematic diagram of an electronic device in an embodiment of the present invention. DETAILED DESCRIPTION

[0065] The present application is described below based on embodiments, but the present application is not limited to these embodiments. In the detailed description of the present application below, some specific details are described in detail. It is possible for those skilled in the art to fully understand the present application without the description of these details. In order to avoid confusing the essence of the present application, known methods, processes, flows, components and circuits are not described in detail.

[0066] In addition, persons of ordinary skill in the art will appreciate that the drawings provided herein are for illustration purposes and are not necessarily drawn to scale.

[0067] Unless the context clearly requires otherwise, the words "include", "comprising" and similar words throughout the application should be interpreted as including rather than exclusive or exhaustive; that is, the meaning is "including but not limited to".

[0068] In the description of this application, it should be understood that the terms "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying relative importance. In addition, in the description of this application, unless otherwise specified, the meaning of "plurality" is two or more.

[0069] In the prior art, the large language model LLM combined with multimodal capabilities can understand the content on the screen of the terminal device and perform operations on the mobile device according to the user's instructions, such as purchasing goods on a shopping website. However, since LLM needs to analyze the screen content, understand the user's instructions and generate operation steps, and automatically operate the terminal device through the generated operation steps, the whole process is time-consuming and the operation speed is slow. During this period, the process of automatic operation of the terminal device will occupy the mobile phone screen and system resources. The user cannot use the mobile phone to perform other operations during this process, which interferes with the user's normal use of the mobile phone and affects the user's experience. Specifically, the terminal device automatic operation refers to the automatic operation software performing automatic operations on the terminal device, such as installing applications, opening applications, entering text, etc., to improve operational efficiency and reduce human errors.

[0070] For example, suppose a user sends a shopping instruction to a terminal device, and the terminal device gradually executes the shopping steps on the real screen. Here the real screen is completely occupied. The user can no longer use the terminal device to make calls, send messages, play games, or handle work tasks, etc.; therefore, how to realize the automatic operation of the terminal device without occupying the user's terminal device screen and improve the user experience is a problem that needs to be solved at present.

[0071] In order to solve the above problems, an embodiment of the present invention proposes a method for automatic operation of a terminal device, as follows: Figure 1 As shown, the method includes:

[0072] Step S101: Obtain user instruction information.

[0073] Specifically, the terminal device obtains the user command information in the form of voice or text, that is, the user command information can be divided into user voice command information and user text command information.

[0074] Wherein, an automated operation application is installed in the terminal device or an automated operation function is set.

[0075] In one possible implementation, the automated operation application or automated operation function has the authority to control other applications or functions installed in the terminal device with the user's authorization; and, the automated operation application or automated operation function can also obtain user voice command information and user text command information issued by the user with the user's authorization.

[0076] For example, the content of the user instruction information is "order a bowl of beef noodles for me", "buy me a power bank", "take a taxi to a certain building at 5:30 pm" or "adjust the volume of the mobile phone to 10", which is only an exemplary description.

[0077] Step S102: Generate a virtual display screen in the background of the terminal device according to the user instruction information.

[0078] Specifically, after the automated operation application or automated operation function installed in the terminal device receives the user instruction information, a virtual display screen is generated in the background of the terminal device. The virtual display screen runs in the background of the terminal device for a complete system environment, wherein the system can be an Android system, an iOS system, etc., which is the same as the system environment of the terminal device.

[0079] Step S103: loading an application corresponding to the user instruction information on the virtual display screen.

[0080] Specifically, the user instruction information is analyzed according to the large language model to determine the application corresponding to the user instruction information; and the application is loaded on the virtual display screen.

[0081] For example, assuming that the user command information is "buy me a power bank", the user command information is analyzed by a large language model to determine that the application corresponding to the user command information is a shopping application, and the shopping application is loaded on the virtual display screen; or, the user command information is "take a taxi to a certain place at 5:30 pm", the user command information is analyzed by a large language model to determine that the application corresponding to the user command information is a travel application, and the travel application is loaded on the virtual display screen; or, the user command information is "adjust the volume of the mobile phone to 10", the user command information is analyzed by a large language model to determine that the user command information corresponds to a setting service of a terminal device, and the setting service of the terminal device is loaded on the virtual display screen.

[0082] Step S104: Acquire the target image displayed on the virtual display screen at the current moment.

[0083] Assume that when the travel application is loaded on the virtual display screen, the application interface of the travel application is displayed on the virtual display screen. If the current moment is the initial moment of entering the application, the application interface is the home page, for example Figure 2 As shown, it includes: "departure point", "current location", "destination point" user target location search box and map display, wherein the user target location search box can display "Where do you want to go?", which is only an exemplary description.

[0084] In a possible implementation, when the shopping application is loaded on the virtual display screen, the application interface of the shopping application is displayed on the virtual display screen, which is determined according to actual conditions.

[0085] Step S105: input the user instruction information and the target image into a large language model to generate automated operation steps.

[0086] Specifically, the large language model analyzes the user instruction information and the target image, understands the user's intention, and generates automated operation steps. For example, when filling in an address, the generated automated operation steps are to fill in "such and such building" to the address. Figure 2 The user destination search box in .

[0087] Step S106: executing the automated operation steps on the virtual display screen.

[0088] Specifically, execute the automated operation steps to fill in "XX Building" to the Figure 2 The user target location search box in the virtual display screen is automatically filled with "such and such building". Figure 3 shown.

[0089] In the embodiment of the present invention, after completing step S106, other steps are also included, specifically as follows: Figure 4 As shown, a method for automatic operation of a terminal device is shown, which specifically includes:

[0090] Step S107: updating the target image displayed on the display screen according to the automated operation steps.

[0091] In an embodiment of the present invention, if the task of the user instruction information is to be completed, it may not be possible to complete it by generating one automated operation step, and it is necessary to use a large language model multiple times to analyze the user instruction information and the target image that is updated multiple times to generate multiple automated operation steps; if multiple automated operation steps are required, after step S107, return to step S105 until the result corresponding to the user instruction information is generated.

[0092] In a possible implementation, after completing step S106, other steps are further included, specifically: Figure 5 As shown, a method for automatic operation of a terminal device is shown, which specifically includes:

[0093] Step S108: Determine the result generated by the automated operation step.

[0094] Specifically, the background of the terminal device determines the result generated by the virtual display screen. For example, the shopping application in the virtual display screen has completed the purchase of the power bank and determined the product information and order amount of the power bank, or the travel application in the virtual display screen has completed taking a taxi to a certain building and determined the vehicle type, estimated time and estimated cost, etc. This is only an exemplary description.

[0095] Step S109: Send the result to the real display screen of the terminal device.

[0096] In the embodiment of the present invention, since relatively important payment amount and travel information are involved, it is necessary to send the result to the real display screen of the terminal device for the user to confirm again to ensure the user's financial security or travel safety.

[0097] In a possible implementation, step S108 and step S109 may also be arranged after step S107.

[0098] In a possible implementation, the result is displayed on the actual display screen of the terminal device in the form of a local pop-up window, for example Figure 6 As shown, at this time, if the user is using other applications on the real display screen of the terminal device, the local pop-up window does not affect the use of the other applications.

[0099] In a possible implementation, the result may also be sent to the terminal device by voice broadcast.

[0100] In a possible implementation, after completing step S109, other steps are also included, specifically as follows: Figure 7 As shown, a method for automatic operation of a terminal device is shown, which specifically includes:

[0101] Step S110: receiving confirmation feedback information sent by the user, wherein the confirmation feedback information indicates that the user confirms the generated result.

[0102] In an embodiment of the present invention, the user can directly click the "Confirm" box displayed in the local pop-up window to send confirmation feedback information to the virtual display screen in the background; or, the user can switch the result interface of the virtual display screen to the real display screen for confirmation.

[0103] In a possible implementation, after completing step S109, other steps are also included, specifically as follows: Figure 8 As shown, a method for automatic operation of a terminal device is shown, which specifically includes:

[0104] Step S111: receiving modification instruction information sent by a user, wherein the modification instruction information indicates that the user modifies the generated result.

[0105] In an embodiment of the present invention, the user can directly click on the "Modify" box displayed in the local pop-up window to modify the address or the purchased product signal. The user can modify any information that needs to be modified and can also cancel it before making the final confirmation.

[0106] In a possible implementation, the terminal device sends modification instruction information to the virtual display screen in the background; or, the user can switch the result interface of the virtual display screen to the real display screen for modification.

[0107] In the embodiment of the present invention, the virtual display screen runs in the background, and the user cannot observe the virtual display screen under normal conditions. However, in order for the user to understand the current progress, the virtual display screen can be displayed in the form of a small window on the real display screen of the terminal device, wherein the small window occupies a local area of ​​the real display screen, specifically as follows: Fig. 9 As shown, this is only an exemplary description, and the position and size of the small window are determined according to actual conditions.

[0108] Through the above embodiments, firstly, by creating an invisible virtual display screen in the background, LLM is allowed to perform automated operations in a virtual environment without occupying the real display screen that the user is using, which can ensure that the user will not be disturbed when using the terminal device, thereby improving the user experience; secondly, by utilizing the natural language processing capability of the large language model, complex user instructions can be understood and analyzed, and at the same time, combined with the graphical user interface (GUI), the multimodal capability of the LLM can generate operation steps more accurately; thirdly, since tasks are performed independently in the virtual display screen, the separation of automated operations and user activities is ensured, thereby improving the overall operational efficiency and reducing the operational delay caused by the simultaneous processing of multiple tasks; finally, the background sends the results to the user for confirmation, which ensures the user's control and sense of security, avoids the improper execution of automated operations, and increases the reliability of operations. Moreover, automated operations are performed in a virtual environment, which helps to protect the user's privacy and data security and reduces potential risks.

[0109] In the embodiment of the present invention, the above method can be used not only for shopping and traveling, but also for scenarios such as ticket booking, automated testing, and auxiliary functions. That is, the scenarios that can be realized by the real terminal device can be realized on the virtual display screen in the background.

[0110] In a possible implementation, if the terminal device is a computer, tablet computer, foldable mobile phone or other device capable of split-screen display, one of the split-screens may be selected to display the content on the virtual display screen, for example, Fig.10 As shown, Fig.10 This is a schematic diagram of the display screen of a foldable mobile phone. Assume that the display screen of the foldable mobile phone is divided into screen 1 and screen 2, wherein screen 1 can display the content in the virtual display screen, that is, the automated operation is implemented in screen 1, and screen 2 is provided for the user to use other applications normally, and the automated operation of screen 1 will not affect screen 2.

[0111] In a possible implementation, the terminal device may be a multi-split screen, such as a third split screen or a fourth split screen, in addition to the above-mentioned two-split screen. The automated operation may be displayed in any of the multi-split screens as long as it does not affect the user's normal use of the terminal device.

[0112] In an embodiment of the present invention, a system for automatic operation of a terminal device is provided, such as Fig.11 As shown, it includes an instruction receiving module 1101, a virtual display module 1102, an application loading module 1103, an LLM reasoning module 1104, an operation execution module 1105 and a result confirmation module 1106; wherein, the instruction receiving module 1101 is used to receive user instruction information; the virtual display module 1102 is used to create and manage a virtual display screen; the application loading module 1103 is used to load a target application; the LLM reasoning module 1104 is used to analyze the user instruction information and the target application interface on the virtual display screen, and generate automated operation steps; the operation execution module 1105 is used to execute automated operation steps on the virtual display screen; and the result confirmation module 1106 is used to send the operation result to the user for confirmation.

[0113] In an embodiment of the present invention, a device for automatic operation of a terminal device is provided, such as Fig.12 As shown, it specifically includes: an acquisition unit 1201, a generation unit 1202, a processing unit 1203 and an execution unit 1204; wherein the acquisition unit 1201 is used to acquire user instruction information; the generation unit 1202 is used to generate a virtual display screen in the background of the terminal device according to the user instruction information; the processing unit 1203 is used to load the application corresponding to the user instruction information on the virtual display screen; the acquisition unit 1201 is also used to acquire the target image displayed on the virtual display screen at the current moment; the generation unit 1202 is also used to input the user instruction information and the target image into a large language model to generate automated operation steps; the execution unit 1204 is used to execute the automated operation steps on the virtual display screen.

[0114] Furthermore, the device also includes:

[0115] An updating unit is used to update the target image displayed on the display screen according to the automated operation steps.

[0116] Furthermore, the device also includes:

[0117] A determination unit, used to determine the result generated by the automated operation step;

[0118] A sending unit is used to send the result to the real display screen of the terminal device.

[0119] Furthermore, the device also includes:

[0120] The receiving unit is used to receive confirmation feedback information sent by the user, wherein the confirmation feedback information indicates that the user confirms the generated result.

[0121] Furthermore, the receiving unit is also used for:

[0122] A modification instruction message sent by a user is received, wherein the modification instruction message indicates that the user modifies the generated result.

[0123] Furthermore, the device also includes:

[0124] The display unit is used to display the virtual display screen in the form of a small window on the real display screen of the terminal device, wherein the small window occupies a partial area of ​​the real display screen.

[0125] Furthermore, the processing unit is specifically used for:

[0126] Analyzing the user instruction information according to the large language model to determine the application corresponding to the user instruction information;

[0127] The application is loaded on the virtual display screen.

[0128] Fig.13 Schematic diagram of the structure of the electronic device in the embodiment of the present invention. Fig.13As shown, it includes a general computer hardware structure, which at least includes a processor 1301 and a memory 1302. The processor 1301 and the memory 1302 are connected via a bus 1303. The memory 1302 is suitable for storing instructions or programs executable by the processor 1301. The processor 1301 can be an independent microprocessor or a collection of one or more microprocessors. Thus, the processor 1301 executes the instructions stored in the memory 1302, thereby executing the method flow of the embodiment of the present invention as described above to realize the processing of data and the control of other devices. The bus 1303 connects the above-mentioned multiple components together, and at the same time connects the above-mentioned components to the display controller 1304 and the display device and the input / output (I / O) device 1305. The input / output (I / O) device 1305 can be a mouse, a keyboard, a modem, a network interface, a touch input device, a somatosensory input device, a printer, and other devices known in the art. Typically, the input / output device 1305 is connected to the system through an input / output (I / O) controller 1306.

[0129] Among them, the instructions stored in the memory 1302 are executed by at least one processor 1301 to achieve: obtaining user command information; generating a virtual display screen in the background of the terminal device according to the user command information; loading the application corresponding to the user command information on the virtual display screen; obtaining the target image displayed on the virtual display screen at the current moment; inputting the user command information and the target image into a large language model to generate automated operation steps; and executing the automated operation steps on the virtual display screen.

[0130] Specifically, the electronic device includes: one or more processors 1301 and a memory 1302, Fig.13 Take a processor 1301 as an example. The processor 1301 and the memory 1302 may be connected via a bus or other means. Fig.13 In the example, the bus connection is used. The memory 1302 is a non-volatile computer-readable storage medium that can be used to store non-volatile software programs, non-volatile computer executable programs and modules. The processor 1301 executes various functional applications and data processing of the device by running the non-volatile software programs, instructions and modules stored in the memory 1302, that is, the method for determining the automatic operation of the terminal device is realized.

[0131] The memory 1302 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and applications required for at least one function; the data storage area may store a list of options, etc. In addition, the memory 1302 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 1302 may optionally include a memory remotely arranged relative to the processor 1301, and these remote memories may be connected to an external device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0132] One or more modules are stored in the memory 1302, and when executed by one or more processors 1301, the method for automatic operation of the terminal device in any of the above method embodiments is executed.

[0133] As will be appreciated by those skilled in the art, various aspects of the embodiments of the present invention may be implemented as a system, method or computer program product. Therefore, various aspects of the embodiments of the present invention may take the form of a complete hardware implementation, a complete software implementation (including firmware, resident software, microcode, etc.), or an implementation that combines software aspects with hardware aspects, which may generally be referred to herein as a "circuit," "module," or "system." In addition, various aspects of the embodiments of the present invention may take the form of a computer program product implemented in one or more computer-readable media having a computer-readable program code implemented thereon.

[0134] Any combination of one or more computer-readable media can be utilized. Computer-readable media can be computer-readable signal media or computer-readable storage media. Computer-readable storage media can be, for example, (but not limited to) electronic, magnetic, optical, electromagnetic, infrared or semiconductor systems, devices or apparatuses, or any suitable combination of the foregoing. More specific examples (non-exhaustive enumeration) of computer-readable storage media will include the following: an electrical connection with one or more wires, a portable computer floppy disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of an embodiment of the present invention, a computer-readable storage medium can be any tangible medium that can contain or store a program used by an instruction execution system, device or apparatus or a program used in conjunction with an instruction execution system, device or apparatus.

[0135] A computer readable signal medium may include a propagated digital signal having a computer readable program code implemented therein, such as in baseband or as part of a carrier wave. Such propagated signals may take any of a variety of forms, including but not limited to electromagnetic, optical, or any suitable combination thereof. A computer readable signal medium may be any of the following computer readable media: not a computer readable storage medium, and may communicate, propagate, or transmit a program used by or in conjunction with an instruction execution system, device, or apparatus.

[0136] Program code embodied on a computer readable medium may be transmitted using any appropriate medium including, but not limited to, wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0137] The computer program code for performing operations for various aspects of the embodiments of the present invention can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, Smalltalk, C++, etc., and conventional process programming languages ​​such as "C" programming language or similar programming languages. The program code can be executed completely on the user's computer, partially on the user's computer as a stand-alone software package; partially on the user's computer and partially on a remote computer; or completely on a remote computer or server. In the latter case, the remote computer can be connected to the user's computer through any type of network including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (e.g., by using the Internet of an Internet service provider).

[0138] The flowchart legend and / or block diagram of the method, device (system) and computer program product according to the embodiment of the present invention described above describe various aspects of the embodiment of the present invention.It will be understood that each block of the flowchart legend and / or block diagram and the combination of the blocks in the flowchart legend and / or block diagram can be implemented by computer program instructions.These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer or other programmable data processing device to produce a machine so that the instructions (executed by the processor of the computer or other programmable data processing device) create a device for implementing the function / action specified in the flowchart and / or block diagram block or block.

[0139] These computer program instructions may also be stored in a computer-readable medium that can direct a computer, other programmable data processing device, or other apparatus to operate in a particular manner, so that the instructions stored in the computer-readable medium produce an article of manufacture including instructions for implementing the functions / actions specified in the flowchart and / or block diagram blocks or blocks.

[0140] The computer program instructions may also be loaded onto a computer, other programmable data processing device or other apparatus so that a series of operable steps are performed on the computer, other programmable device or other apparatus to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide a process for implementing the functions / actions specified in the flowchart and / or block diagram block or blocks.

[0141] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

[0142] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation portals for users to choose to authorize or refuse. Users' refusal to process personal information other than the necessary information required for basic functions will not affect the user's use of basic functions.

Claims

1. A method for automated operation of a terminal device, characterized in that: The method comprises: Get user command information; Generate a virtual display screen in the background of the terminal device according to the user instruction information; Loading the application corresponding to the user instruction information on the virtual display screen; Acquire the target image displayed on the virtual display screen at the current moment; Inputting the user instruction information and the target image into a large language model to generate automated operation steps; The automated operation steps are performed on the virtual display screen.

2. The method according to claim 1, characterized in that The method further comprises: The target image displayed on the display screen is updated according to the automated operation steps.

3. The method according to claim 1, characterized in that The method further comprises: Determining a result generated by the automated operation step; The result is sent to the real display screen of the terminal device.

4. The method according to claim 3, characterized in that The method further comprises: Confirmation feedback information sent by the user is received, wherein the confirmation feedback information indicates that the user confirms the generated result.

5. The method according to claim 3, characterized in that: The method further comprises: A modification instruction message sent by a user is received, wherein the modification instruction message indicates that the user modifies the generated result.

6. The method according to claim 1, characterized in that The method further comprises: The virtual display screen is displayed on the real display screen of the terminal device in the form of a small window, wherein the small window occupies a partial area of ​​the real display screen.

7. The method according to claim 1, characterized in that The step of loading the application corresponding to the user instruction information on the virtual display screen specifically includes: Analyzing the user instruction information according to the large language model to determine the application corresponding to the user instruction information; The application is loaded on the virtual display screen.

8. A device for automatic operation of terminal equipment, characterized in that: The device comprises: An acquisition unit, used for acquiring user instruction information; A generating unit, configured to generate a virtual display screen in the background of a terminal device according to the user instruction information; A processing unit, configured to load an application corresponding to the user instruction information on the virtual display screen; The acquisition unit is also used to acquire the target image displayed on the virtual display screen at the current moment; The generating unit is further used to input the user instruction information and the target image into a large language model to generate automated operation steps; An execution unit is used to execute the automated operation steps on the virtual display screen.

9. An electronic device comprising a memory and a processor, characterized in that: The memory is used to store one or more computer program instructions, wherein the one or more computer program instructions are executed by the processor to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images