Method for controlling cloud mobile phone, related device, intelligent agent and computer program product

Abstract

The invention provides a method and device for controlling a cloud mobile phone, an intelligent agent, electronic equipment, a computer readable storage medium and a computer program product, and relates to the technical field of artificial intelligence such as cloud computing, automatic control and deep learning. A specific embodiment of the method comprises the following steps: receiving task description information which is provided by a user in a natural language form and aims at a cloud mobile phone; analyzing the task description information to obtain function description information; based on the function description information, extracting a group of target actions from a robot process automation action library to form an action sequence; and issuing the action sequence to the target cloud mobile phone, and controlling the target cloud mobile phone to execute the action sequence. According to the embodiment, the cloud mobile phone can be used and configured by the user more conveniently, and the use and configuration difficulty and cost of the cloud mobile phone by the user are reduced. Correspondingly, according to the implementation mode, the cloud mobile phone can be used and configured in more service scenes, and the use scene of the cloud mobile phone is expanded.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, specifically to the field of artificial intelligence technology such as cloud computing, automatic control, and deep learning, and especially to methods, devices, intelligent entities, electronic devices, computer-readable storage media, intelligent entities, and computer program products for controlling cloud phones. Background Art

[0002] With the development of computer technology, cloud phone technology has emerged to improve the computing power and user experience of user devices. Cloud phone is a "mobile phone" that uses cloud computing technology to network terminal services and implements cloud services through cloud servers. For example, a "server" can be used as a cloud computing power provider to enable users to obtain cloud phone services by interacting with the "server".

[0003] This kind of "mobile phone" that is deeply integrated with network services can obtain more powerful computing power with its own system and the network terminals set up by the manufacturer to provide more functions for the mobile phone used by the user. Therefore, how to facilitate users to use cloud phones and make cloud phones more convenient for users to use is worthy of attention and urgent need. Summary of the invention

[0004] The embodiments of the present disclosure provide a method, device, intelligent entity, electronic device, computer-readable storage medium and computer program product for controlling a cloud phone.

[0005] In the first aspect, an embodiment of the present disclosure proposes a method for controlling a cloud phone, including: receiving task description information for a cloud phone provided by a user in natural language; parsing the task description information to obtain function description information; based on the function description information, extracting a set of target actions from a robot process automation action library to form an action sequence; sending the action sequence to a target cloud phone, and controlling the target cloud phone to execute the action sequence.

[0006] In the second aspect, an embodiment of the present disclosure proposes a device for controlling a cloud phone, comprising: an input receiving unit, configured to receive task description information for the cloud phone provided by a user in natural language; an information parsing unit, configured to parse the task description information to obtain function description information; an action sequence composition unit, configured to extract a set of target actions from a robot process automation action library based on the function description information to form an action sequence; an execution control unit, configured to send the action sequence to a target cloud phone and control the target cloud phone to execute the action sequence.

[0007] In the third aspect, the embodiments of the present disclosure provide an intelligent entity, including: an input receiving module, configured to receive task description information for a cloud phone provided by a user in natural language; an information parsing module, configured to parse the task description information to obtain function description information; an action sequence composition module, configured to extract a set of target actions from a robot process automation action library based on the function description information to form an action sequence; an execution control module, configured to send the action sequence to a target cloud phone and control the target cloud phone to execute the action sequence.

[0008] In a fourth aspect, an embodiment of the present disclosure provides an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that when the at least one processor executes, it can implement the method of controlling a cloud phone as described in any implementation method in the first aspect.

[0009] In a fifth aspect, an embodiment of the present disclosure provides a non-transitory computer-readable storage medium storing computer instructions, which are used to enable a computer to implement the method of controlling a cloud phone as described in any implementation method in the first aspect when executed.

[0010] In a sixth aspect, an embodiment of the present disclosure provides a computer program product comprising a computer program, which, when executed by a processor, can implement the method for controlling a cloud phone as described in any implementation manner in the first aspect.

[0011] The method, device, intelligent body, electronic device, computer-readable storage medium and computer program product for controlling a cloud phone provided by the embodiments of the present disclosure receive task description information for a cloud phone provided by a user in natural language; parse the task description information to obtain function description information; based on the function description information, extract a set of target actions from a robot process automation action library to form an action sequence; send the action sequence to a target cloud phone, and control the target cloud phone to execute the action sequence.

[0012] The present disclosure can make cloud phones more convenient for users to use and configure, reducing the difficulty and cost of using and configuring cloud phones for users. Accordingly, cloud phones can also be used and configured in more business scenarios, expanding the use scenarios of cloud phones.

[0013] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Other features, objects and advantages of the present disclosure will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0015] Figure 1 is an exemplary system architecture in which the present disclosure may be applied;

[0016] Figure 2 A flowchart of a process of controlling a cloud phone provided by an embodiment of the present disclosure;

[0017] Figure 3 A flowchart of a process for generating a robotic process automation action library provided by an embodiment of the present disclosure;

[0018] Figure 4 A flowchart of a process of controlling a target cloud phone to execute an action sequence provided by an embodiment of the present disclosure;

[0019] Figure 5 A schematic diagram of an architecture capable of implementing a process of controlling a cloud phone in an application scenario provided by an embodiment of the present disclosure;

[0020] Figure 6 A structural block diagram of a device for controlling a cloud phone provided in an embodiment of the present disclosure;

[0021] Figure 7 A schematic diagram of the structure of an electronic device suitable for executing a method for controlling a cloud phone provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0022] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description. It should be noted that the embodiments in the present disclosure and the features in the embodiments may be combined with each other without conflict.

[0023] In addition, the acquisition, storage, use, processing, transportation, provision and disclosure of user personal information involved in the technical solutions of this disclosure are in compliance with the relevant laws and regulations and do not violate public order and good morals. For example, the acquisition of the description information provided by the user and the acquisition of the action video stream of the user when using the target cloud phone mentioned below are all known to the user and authorized and approved by the user.

[0024] Figure 1An exemplary system architecture 100 is shown to which embodiments of the method, apparatus, agent, electronic device, and computer-readable storage medium for controlling a cloud phone disclosed in the present invention may be applied.

[0025] like Figure 1 As shown, the system architecture 100 may include terminal devices 101, 102, 103, a network 104, and servers 105 and 106. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103, the servers 105, and the servers 106. The network 104 may include various connection types, such as wired, wireless communication links, or optical fiber cables, etc.

[0026] Users can use terminal devices 101, 102, 103 to interact with servers 105 and 106 through network 104 to receive or send messages, etc. Terminal devices 101, 102, 103 and server 105 can be installed with various applications for realizing information communication between the two, such as cloud phone management applications, cloud computing applications, instant messaging applications, etc. Correspondingly, servers 105 and 106 can also realize communication based on network 104 in a similar manner.

[0027] Terminal devices 101, 102, 103 and servers 105, 106 can all be hardware, or software individually or as a whole. When terminal devices 101, 102, 103 are hardware, they can be various electronic devices with display screens, including but not limited to smart phones, tablet computers, laptop portable computers, desktop computers, etc.; when terminal devices 101, 102, 103 are software, they can be installed in the electronic devices listed above, which can be implemented as multiple software or software modules, or as a single software or software module, which is not specifically limited here. When servers 105, 106 are hardware, they can be implemented as a distributed server cluster composed of multiple servers (for example, a server cluster that can be used to provide cloud services), or as a single server; when the server is software, it can be implemented as multiple software or software modules, or as a single software or software module, which is not specifically limited here.

[0028] The user can use the terminal devices 101, 102, 103 to interact with the server 105, server 106 through the network 104 to receive or send messages, etc. For example, when the server 106 is illustrated as a subject providing a "cloud phone" service, the user can use the terminal devices 101, 102, 103 to communicate with the server 106 through the network 104 to obtain the "cloud phone service".

[0029] Accordingly, various applications for realizing information communication between the terminal devices 101, 102, 103 and the server 106 may be installed, such as cloud phone computing, cloud service and cloud gaming applications.

[0030] In some other scenarios, when multiple "cloud phones" are provided, the server 105 can exemplarily be embodied as a "cloud phone service platform" that has management capabilities for the "cloud phones" and can configure the "cloud phones" for users to use.

[0031] Accordingly, the user can request the server 105 to configure and allocate a cloud phone for him / her by communicating with the server 105 (for example, the server 105 can configure the server 106 as a "cloud phone" for the terminal devices 101, 102, and 103 according to the user's request).

[0032] Accordingly, the server 105 and the server 106 can also similarly communicate based on the network 104. For example, the server 106 responds to the instructions of the server 105 and the configured permissions, and acts as a computing power provider of the "cloud phone" to provide services to the user.

[0033] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to the implementation requirements.

[0034] Please refer to Figure 2 , Figure 2 A flowchart of a process of controlling a cloud phone is provided in an embodiment of the present disclosure, including process 200.

[0035] The process 200 specifically includes the following steps:

[0036] Step 201: receiving task description information for a cloud phone provided by a user in a natural language form;

[0037] In the embodiments of the present disclosure, the execution subject of the method for controlling the cloud phone (for example Figure 1 The terminal devices 101, 102, and 103 shown in the figure can receive task description information for the cloud phone provided by the user in natural language when being used by the user. For example, the execution subject can collect the "voice" in natural language recorded by the user through its associated sound collection device (e.g., microphone).

[0038] Natural Language Form refers to the way of expressing and describing things, problems, concepts or instructions using natural language (such as Chinese, English, etc.). Natural language is the language used in daily human communication. It has rich grammar, vocabulary and semantics, and can flexibly express various ideas and emotions. Accordingly, "task description information in natural language form" can be understood as the content that users can directly express according to their own expression and expression habits, rather than the expression and expression content that they deliberately adjust according to a specific language form.

[0039] In some scenarios, the interaction mode that allows interaction in the form of user natural language may also be referred to as interaction based on a language user interface (LUI).

[0040] Furthermore, after receiving the voice provided by the user, the execution subject can determine whether it is "task description information for cloud phones" by analyzing its semantics. For example, the execution subject can determine whether it is directed to "cloud phones" or "voice" with the intention of using or configuring cloud phones by analyzing the entities included therein.

[0041] In some scenarios, in order to more accurately distinguish the intention of the "natural language form" voice provided by the user, the execution subject can also configure trigger controls, etc., so that the user can indicate his intention by triggering the control. For example, the execution subject can determine the voice received after the control is triggered as the above-mentioned task description information for the cloud phone.

[0042] The task description information may include the content of the task that the user expects to achieve, for example, expecting to use the cloud phone to complete a certain calculation, execute a certain algorithm, etc. For example, the task description information may be "use the cloud phone to open the application, and after executing XX in the application, return the result of executing XX".

[0043] In some optional implementations of this embodiment, the task description information provided by the user may also be in text form. For example, the user provides the task description information to the execution entity based on the entered text information.

[0044] In some optional implementations of this embodiment, the task description information may be for a cloud phone autonomous task under a graphical user interface. A graphical user interface (GUI) may be a graphical interface for a user to interact with software or hardware, usually displayed through elements such as icons, buttons, windows, and menus.

[0045] Accordingly, the “task” corresponding to the task description information may be a cloud phone autonomous task related to the GUI.

[0046] "Cloud phone autonomous tasks" refer to tasks that can independently perform certain cloud phone operations without the need for further human interaction or intervention. For example, cloud phone autonomous tasks under the GUI can be test tasks for applications in the cloud phone, "repetitive call tasks" for applications, and so on. For example, the form of task description information can be exemplified as: in the GUI interface of an application in a cloud phone, certain actions are performed, such as clicking a control in the page of the GUI of the application (or continuously clicking a group of controls) to interact with the application and obtain the services provided by the application. Alternatively, such a task may also be to achieve "a target result", for example, the task description information indicates (using a cloud phone) to test a function or a service in the GUI interface (for example, to fully or partially test an access or service acquisition link).

[0047] Therefore, through the autonomous tasks of the cloud phone under the GUI, users can conveniently use the cloud phone to simulate the "scenarios of manual operation and manual use of the application's GUI interface" through the "hosting" method, without having to actually and continuously perform operations on the cloud phone, thereby reducing the user's consumption of the cloud phone.

[0048] It should be pointed out that, usually based on the need for real-time interaction, the task description information is often obtained based on the user's real-time behavior. However, in some scenarios, the task description information can also be pre-collected by the execution subject, or obtained from other non-local devices (for example, because the user sets a delayed provision after collection, or the user instructs the execution subject to provide the task description information from other devices). In this case, the task description information can be obtained by the above-mentioned execution subject directly from the local storage device, or from a non-local storage device (for example Figure 1 As shown, the task description information is obtained from a terminal device (not used by the user among the terminal devices 101, 102, and 103). The local storage device may be a data storage module provided in the execution subject, such as a server hard disk. In this case, the task description information can be quickly read locally; the non-local storage device may also be any other electronic device provided for storing data, such as some user terminals, etc. In this case, the execution subject may obtain the required task description information by sending an acquisition command to the electronic device.

[0049] Step 202: parse the task description information to obtain function description information;

[0050] In an embodiment of the present disclosure, after the execution subject determines and receives the task description information based on the above step 201, the execution subject can parse the task description information to obtain the function description information. For example, the execution subject can determine the various "functions" that the user wants to obtain based on the actions and "purpose" entities included in the task description information, and then obtain the function description information.

[0051] Function description information can usually correspond to a function and be used to characterize the "information" of the function. For example, for the image zoom function, the function description information can actually be "reduce the image to XXX" or "enlarge the image to YYY", etc. Accordingly, the execution subject can accurately understand the "function" that needs to be provided through the function description information.

[0052] It should be understood that, for example, due to different difficulties and directions of the "functions", the execution subject can determine one or more functional description information by parsing the task description information. For example, in some "high-level functional integration" scenarios, the functional description information may include "integrated functions" such as continuous zooming, cropping, and adjusting the brightness of an image. In such a case, the "functional description information" may be one item. In some "low-level functional integration" scenarios, the functional description information may actually be three functional description information such as "zooming in the image", "cropping the image", and "adjusting the brightness of the image".

[0053] In some optional implementations of this embodiment, in order to improve the ability to parse task description information, more accurately determine the functions in the task description information, and provide more accurate functional description information, the execution entity may choose to use a large language model to parse the task description information and obtain the functional description information.

[0054] Large Language Model (LLM) is an artificial intelligence model designed to understand and generate human language, and LLM can perform processing operations accordingly based on its understanding to obtain corresponding processing results. For example, after obtaining the task description information, LLM can parse the functions included in the task description information and provide the functional description information of these functions accordingly after understanding the instructions (for example, parsing the functions included therein and providing the functional description information of these functions accordingly).

[0055] LLM can be trained on a large amount of text data and can perform a wide range of tasks, including text summarization, translation, sentiment analysis, and the like. LLM is characterized by its large scale, and it can usually include a large number of parameters to help them learn complex patterns in language data. These models are usually based on deep learning architectures, such as converters, which help them provide better processing performance on various NLP tasks. In an embodiment of the present disclosure, the execution subject can use LLM as a processing task description information to parse it to obtain the corresponding (a set of) functional description information. In some scenarios, in view of the analytical ability of LLM, LLM may be able to meet the ability requirements of parsing task description information and obtaining functional description information without being trained in advance. In this case, the execution subject can also directly instruct LLM to refer to some existing functions (for example, the actions corresponding to the functions, the purposes and effects that the functions can achieve) to parse the task description information and obtain the functional description information.

[0056] In addition, for the large language model, it can be configured by default to omit the "guide words". For example, for the purpose of processing and parsing task description information to obtain functional description information, LLM can, based on the default configuration, naturally understand the parsing operation that needs to be performed on the input task description information. Thus, by means of the default configuration, the large language model can stably and directionally process and parse the task description information, determine the "functions" included therein, expected to be obtained, and required, and provide functional description information of these "functions". Thus, not only can the large language model be used to perform the parsing process of the task description information more efficiently and with higher quality.

[0057] In some embodiments, as discussed above, a method of configuring the GUI action corpus may also be selected to provide a reference for the LLM so that the functional description information can be determined more accurately.

[0058] Step 203: extracting a set of target actions from the robotic process automation action library based on the function description information to form an action sequence;

[0059] In an embodiment of the present disclosure, after determining the function description information based on the above step 202, the execution entity can extract a set of target actions correspondingly from an RPA library including at least one Robotic Process Automation (RPA) action based on the function description information.

[0060] As discussed above, the RPA library (i.e., the Robotic Process Automation action library) may include one or more RPAs. RPA, by using software robots or "bots", automatically executes repetitive, regularized, and structured business processes. Correspondingly, the target action can also be understood as the target RPA. For example, the RPA in the RPA library that can implement the function corresponding to the function description information is extracted as the "target action".

[0061] Then, the execution subject can form an action sequence based on these target actions. For example, the execution subject can determine the execution order between each target action based on whether there is a dependency relationship, sequence, etc. between each target action. For example, for target actions that do not have a dependency relationship, the corresponding execution order can be determined based on the order of the time they are determined and extracted. For example, a target action with an earlier extraction time can have an earlier execution order than a target action with a later extraction time. For example, a target action with a dependency relationship, a target action that depends on other target actions, can have a later execution order than the other target actions that are dependent.

[0062] For easier understanding, please refer to Figure 3 , Figure 3 A flowchart of a process of generating a robotic process automation action library is provided in an embodiment of the present disclosure, which includes process 300.

[0063] Accordingly, in some optional implementations of this embodiment, the robotic process automation action library can be determined based on the method shown in the following process 300.

[0064] The process 300 specifically includes the following steps:

[0065] Step 301: Obtain a sample terminal operation flow in video format;

[0066] Specifically, developers (e.g., the above-mentioned users, or other verified and authorized development users) can generate a sample terminal operation flow in video format by recording the action flow. For example, developers can "use" the application to perform a specific function locally on the terminal device they use, and then record the entire process by recording to generate a sample terminal operation flow in video format.

[0067] Accordingly, the execution subject can obtain such a sample terminal operation flow in video format.

[0068] Step 302: extracting the video stream and the control stream from the sample terminal operation stream;

[0069] Specifically, based on the above step 301, the execution subject can extract the video stream and the control stream from the sample terminal operation stream.

[0070] The video stream is a sequence of images captured during the recording of the sample terminal operation flow. The RPA system based on image recognition will obtain the user's interface operation by capturing the video stream and use image processing technology to identify interface elements.

[0071] The control flow is an automated script or command flow generated by user operations. In other words, the control flow represents specific operation instructions, such as "click a button", "enter text", or "execute conditional judgment". Execution can be based on user operations, and the corresponding control flow will be generated by the RPA tool.

[0072] For example, the sample terminal operation flow can record every user operation (such as clicks, inputs, etc.). Correspondingly, the extracted control flow can include the steps and commands in the RPA script, such as simulated clicks, text inputs, waiting conditions, etc.

[0073] Step 303: based on the control stream and the video stream, convert the control stream into a specific control language and generate a conversion result;

[0074] Specifically, the execution subject can identify the image elements in the interface (or window) of the application in the sample terminal operation flow through optical character recognition (OCR), template matching or computer vision algorithm based on the RPA system. For example, buttons, text boxes, labels, etc.

[0075] Then, based on the elements captured in the video stream, the position, size, color and other characteristics of the operation are determined to accurately simulate user clicks and inputs.

[0076] Next, by identifying and analyzing the elements in the video stream, the various operations involved in the control flow (e.g., click operations on control elements, etc.) can be converted into corresponding specific control languages ​​to generate corresponding conversion results, such as "click a button", "enter text", "wait for element loading", etc.

[0077] In some embodiments, after determining the video stream, the executing entity may also process the video stream by deduplication and noise reduction to obtain "timing key frames" (i.e., video frames that can provide, or have action or functional reference value) to remove video frames in the video stream that lack "action contribution" or "reference value".

[0078] "Specific control language" refers to a specialized programming language or scripting language used to describe and control automated processes. It is usually designed for specific RPA tools or platforms to help users write, manage and optimize automated tasks.

[0079] Step 304: Determine the corresponding RPA based on the function identifier of the conversion result;

[0080] Specifically, after determining the conversion result of the specific control language form based on the above step 303, the execution subject can determine the function corresponding to the part of the "code" based on the various functions included in the conversion result, and "arrange" the corresponding part of the code based on the corresponding function to determine it as a "code module". Then, the "code module" is associated with the function identifier corresponding to the function to determine multiple robot process automation actions.

[0081] Accordingly, after the executing entity determines the "function description information", the determined RPA can extract and determine the robotic process automation action based on the matching of the function corresponding to the "function description information" and the "function" corresponding to the "function identifier".

[0082] Step 305: Generate a robotic process automation action library based on each RPA.

[0083] Specifically, after RPA is determined based on the above process, each robotic process automation action determined in the above process can be constructed and generated into a "robotic process automation action library" to facilitate subsequent execution entities to extract the required robotic process automation actions from the "robotic process automation action library".

[0084] For example, the RPA library may maintain the RPA actions and the corresponding function identifiers. Then, the execution subject may extract the required RPA from the RPA library based on the “function” matching method as discussed above.

[0085] Therefore, using the RPA library can further reduce the user's cloud phone configuration costs. For example, users can use the RPA library to refer to various RPAs provided by other users and developers to achieve their configuration goals, without the need for users to configure each action they want the cloud phone to perform one by one.

[0086] In some optional implementations of this embodiment, as discussed above, the "function" can be recorded in the form of "function identification" to facilitate matching between "functions". Accordingly, as an alternative or replacement of this step 203, the execution subject can first select to determine the target function identification based on the function description information when executing the process of "extracting a set of target actions from the RPA library based on the function description information to form an action sequence".

[0087] For example, the execution subject can obtain the task description information of the corresponding GUI autonomous task in natural language form as discussed above, and then parse the task description information to determine the required "functions". Then, based on the similar method of generating function identifiers discussed above, determine the target function identifiers corresponding to these functions (or, the execution subject directly uses "function" as the target function identifier for subsequent matching).

[0088] Then, the execution entity uses the determined target function identifier to match the "function identifier" included in the RPA library, and extracts a set of target actions (i.e., RPA) corresponding to the target function identifier from the robotic process automation action library to form an action sequence.

[0089] Therefore, by using the "function identification" method, the required target actions can be extracted from the robot process automation action library more quickly and accurately.

[0090] Step 204: Send the action sequence to the target cloud phone, and control the target cloud phone to execute the action sequence.

[0091] In an embodiment of the present disclosure, after determining the action sequence based on the above step 202, the execution entity can provide and send the action sequence to the target cloud phone that is expected to be used and configured (for example, the execution entity can send the action sequence to the server 106 exemplified as a "cloud phone").

[0092] Then, the execution subject controls the target cloud phone to execute the action sequence, so that the target cloud phone can be directly and quickly configured and provide services based on the "task description information in natural language form" provided by the user.

[0093] The method for controlling a cloud phone provided in an embodiment of the present disclosure receives task description information for a cloud phone provided by a user in natural language; parses the task description information to obtain function description information; based on the function description information, extracts a set of target actions from a robot process automation action library to form an action sequence; sends the action sequence to a target cloud phone, and controls the target cloud phone to execute the action sequence. As a result, the cloud phone can be used and configured more conveniently by the user, reducing the difficulty and cost of using and configuring the cloud phone for the user. Accordingly, the cloud phone can also be used and configured in more business scenarios, expanding the use scenarios of the cloud phone.

[0094] In some embodiments, as discussed above, there may also be a "cloud phone service platform" for managing, configuring, and scheduling the "cloud phone". In this case, the process of issuing the action sequence of the execution subject can actually be implemented through the "cloud phone service platform".

[0095] For this, please refer to Figure 4 . Figure 4 A flowchart of a process of controlling a target cloud phone to execute an action sequence provided by an embodiment of the present disclosure includes process 400. For example, process 400 may be a replacement or alternative implementation of step 204 in the above process 200.

[0096] The process 400 specifically includes the following steps:

[0097] Step 401: Communicate the action sequence to the cloud phone service platform;

[0098] Specifically, after the execution subject determines the action sequence based on step 203, it can provide the action sequence to the mobile service platform (for example, Figure 1 Server 105 shown).

[0099] Step 402: Control the cloud phone service platform to determine a target cloud phone for implementing the action sequence from a group of candidate cloud phones;

[0100] Specifically, after providing and communicating the action sequence to the cloud phone service platform based on the above step 401, the execution subject can control the cloud phone service platform to determine the target cloud phone for implementing the action sequence from a group of candidate cloud phones that it manages and has configuration authority. That is, the execution subject can control the cloud phone service platform to configure and allocate a target cloud phone that can be used for the execution subject.

[0101] Step 403: Control the cloud phone service platform to send the action sequence to the target cloud phone;

[0102] Specifically, after the cloud phone service platform determines and configures the target cloud phone based on the above step 402, the execution entity can control the cloud phone service platform, and based on the pre-configured communication link between the cloud phone service platform and the target cloud phone, the cloud phone service platform sends the action sequence to the target cloud phone.

[0103] Step 404: Control the target cloud phone to execute the action sequence through the cloud phone service platform.

[0104] Specifically, the execution subject communicates with the cloud phone service platform to use the cloud phone service platform to instruct and control the target cloud phone to execute the action sequence (for example, the execution subject can send a corresponding action sequence start instruction to the cloud phone service platform to instruct the cloud phone service platform to control the target cloud phone to execute the action sequence).

[0105] It should be understood that in some scenarios, the cloud phone service platform can also be configured to directly send the corresponding action sequence to the target cloud phone after determining the target cloud phone, and directly control the target cloud phone to execute the sent action sequence, so as to reduce the interaction process with the execution entity and reduce the interaction cost.

[0106] Accordingly, in such a case, step 403 may not be included in process 400, and the purpose of step 404 may be achieved directly through the "automatically implemented control process" of the cloud phone service platform.

[0107] In some optional implementations of this embodiment, for the cloud phone service platform, the communication link between it and the candidate cloud phone and the target cloud phone can be configured based on the Android Debug Bridge. In other words, the communication link between the cloud phone and the candidate cloud phone and the target cloud phone is implemented based on the Android Debug Bridge (ADB).

[0108] Android Debug Bridge can be used to debug applications, install APKs, transfer files, execute device commands, and other operations. Accordingly, the cloud phone service platform can be configured to establish communication with candidate cloud phones and target cloud phones in the form of a long connection using Android Debug Bridge. This enables a continuous and stable connection to be established between the cloud phone service platform and candidate cloud phones and target cloud phones through the ADB command line tool. This enables continuous communication between the cloud phone service platform and candidate cloud phones and target cloud phones to avoid re-establishing a connection each time an operation is performed, thereby improving communication efficiency.

[0109] In some embodiments, in order to facilitate the user to understand the execution of the "task" and to facilitate the user to grasp the progress of the "task", the execution subject can also choose to obtain and record the action video stream of the target cloud phone (in the process of executing the action sequence). Then, based on the feedback and the status of the "task execution" presented in the video frame in the action video stream (for example, the control is clicked, the text is generated, etc.), the execution progress of the task is determined.

[0110] For example, you can maintain the (image) state that the task should have at each progress node in advance for different tasks, and maintain a set of preset video frames. Accordingly, each preset video frame can correspond to a task progress information (for example, 10%, 20%, etc. of the task progress).

[0111] Then, the execution entity can compare the latest video frame recorded in the action video stream with a set of preset video frames. If the latest video frame recorded hits a target preset video frame (for example, the similarity between the video frame and the target preset video frame in a set of preset video frames is greater than or equal to a similarity threshold), the execution entity can determine the task progress of the "task" currently executed to the target preset video frame.

[0112] Accordingly, the execution entity may feed back the task progress information corresponding to the target preset video frame to the user (for example, the execution entity may prompt and present the task progress to the user based on its display component).

[0113] In some embodiments, the target cloud phone can return the execution result of the action sequence to the execution entity after completing the action sequence (for example, after the "task" produces a specific result, the target cloud phone can return the result; for example, when the "task" is to complete a process, the target cloud phone can also return "process completion information" as the execution result).

[0114] Accordingly, if the target cloud phone returns an execution result for the action sequence, the execution subject can respond to it, present it, and provide it to the user for reporting.

[0115] In some optional implementations of this embodiment, in order to facilitate user understanding, the execution subject can also choose to process the execution result in a text-to-speech manner, obtain the corresponding text-to-speech transcription result, and generate voice feedback information based on the transcription result. For example, for the processing result that is directly in "text form", the execution subject can directly transcribe it. For the result in non-text form, the execution subject can obtain the corresponding "content in text form" for broadcasting by, for example, reading its semantics, generating description information using LLM, etc.

[0116] Therefore, the execution entity can provide, feedback and report the processing results to the user in the form of "voice broadcast" through the voice feedback information, so as to facilitate the user to obtain, understand and receive the execution results and enhance the user's interactive experience.

[0117] In some embodiments, in order to facilitate the use of users, the above-mentioned execution subject can actually be an intelligent agent. "Intelligent agent" refers to a system that can perceive its environment, make decisions and take actions based on the perceived information. It is not just a tool that passively responds to the environment, but can also actively learn, adapt, plan and optimize behavior to achieve specific goals.

[0118] Such an intelligent agent may include: an input receiving module, which is configured to receive task description information for a cloud phone provided by a user in natural language; an information parsing module, which is configured to parse the task description information and obtain function description information; an action sequence composition module, which is configured to extract a set of target actions from the robot process automation action library based on the function description information to form an action sequence; and an execution control module, which is configured to send the action sequence to the target cloud phone and control the target cloud phone to execute the action sequence.

[0119] Therefore, by utilizing the intelligent agent, the execution subject can directly complete the process of controlling the cloud phone under the various embodiments discussed above by interacting with the intelligent agent.

[0120] In some scenarios, the intelligent agent can also be encapsulated as an "application" that is easier for users to understand, such as an online assistant, a question-and-answer assistant, etc. Accordingly, the user can actually utilize the intelligent agent and interact with the intelligent agent through interaction with the "application" to complete the control and configuration of the cloud phone, so as to achieve the purpose of more convenient use and configuration of the cloud phone discussed above.

[0121] To deepen understanding, this disclosure also provides a specific implementation solution in combination with a specific application scenario. Figure 5 .

[0122] Figure 5 An exemplary architecture 500 capable of implementing a process of controlling a cloud phone in a specific application scenario provided by an embodiment of the present disclosure is shown.

[0123] For easier understanding, you can refer to Figure 1 The architecture 100 is shown to illustrate the architecture 500 .

[0124] In the architecture 500, the user 510 can provide task description information to the agent 520 in the form of LUI through interaction with the agent 520, in order to control, configure and use the cloud phone. Accordingly, the server 105 can serve as a "cloud phone service platform" to provide the user 510 with a configuration process for the server 106 of the "cloud phone".

[0125] For example, the user 510 may provide the agent 520 with task description information 511 in natural language form, so as to complete the configuration and use of the cloud phone through interaction with the agent 520 .

[0126] Accordingly, the agent 520 can refer to the knowledge base 525 based on the task description information 511 by using the LLM 523 to parse the task description information 511 to obtain corresponding function description information (not shown in the figure).

[0127] Then, based on the determined functional description information, the agent 520 extracts a set of target actions (not shown in the figure) from the RPA library 540 to form an action sequence 530.

[0128] Then, agent 520 can provide action sequence 530 to server 105 exemplified as a “cloud phone service platform” to control server 105 as a “cloud phone service platform” to configure a (target) cloud phone for user 510. For example, the configured target cloud phone is server 106.

[0129] Then, the server 105 serving as the “cloud phone service platform” can send the action sequence 530 to the server 106 , so as to use the server 106 as the subject of providing calculation examples to provide the “cloud phone” service to the intelligent agent 520 and the user 510 .

[0130] Then, after the server 106 completes the "cloud phone" service (for example, after executing the action sequence 530), the execution result 535 can be returned to the intelligent agent 520 (for example, through direct communication with the intelligent agent 520, or forwarded by using the server 105).

[0131] After receiving the execution result 535, the intelligent agent 520 can continue to provide it to the user 510 so that the user 510 can complete the configuration and use process of the "cloud phone" (for example, directly presenting the execution result 535, or feedback the execution result 535 through voice broadcast as discussed above).

[0132] Furthermore, to facilitate understanding of the present disclosure, a portion for configuring the RPA library 540 is also shown in the architecture 500. For example, in the architecture 500, the (robotic process automation) actions in the RPA library 540 may come from a sample terminal operation flow 550 in a video format.

[0133] As discussed above, with respect to the sample terminal operation flow 550 , the video stream 560 and the control stream 565 may be first extracted and determined from the sample terminal operation flow 550 .

[0134] For the video stream 560, a group of video frames 551 for recording "action" may be included. Accordingly, the group of video frames 551 may be processed by deduplication, noise reduction, etc., and only the time sequence key frames included therein may be retained. For example, after processing the group of video frames 551, a group of video frames 552 may be obtained.

[0135] Then, based on the control stream 565 and the video stream 560 (more specifically, a set of video frames 552 ), the control stream 565 may be converted into a specific control language to generate a conversion result 567 .

[0136] Subsequently, the corresponding RPA may be determined based on the function identifier (not shown in the figure) of the conversion result, and then the RPA library 540 may be finally generated based on each RPA.

[0137] Further references Figure 6 As an implementation of the methods shown in the above figures, the present disclosure provides an embodiment of a device for controlling a cloud phone. Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0138] like Figure 6 As shown, the device 600 for controlling a cloud phone in this embodiment may include: an input receiving unit 601, an information parsing unit 602, an action sequence composition unit 603, and an execution control unit 604. Among them, the input receiving unit 601 is configured to receive task description information for a cloud phone provided by a user in a natural language form; the information parsing unit 602 is configured to parse the task description information to obtain function description information; the action sequence composition unit 603 is configured to extract a set of target actions from the robot process automation action library based on the function description information to form an action sequence; the execution control unit 604 is configured to send the action sequence to the target cloud phone and control the target cloud phone to execute the action sequence.

[0139] In this embodiment, in the device 600 for controlling the cloud phone, the specific processing of the input receiving unit 601, the information parsing unit 602, the action sequence composition unit 603 and the execution control unit 604 and the technical effects thereof can be referred to in Figure 2 The relevant descriptions of steps 201 - 204 in the corresponding embodiment are not repeated here.

[0140] In some optional implementations of this embodiment, the information parsing unit 602 is further configured to parse the task description information using a large language model to obtain the function description information.

[0141] In some optional implementations of this embodiment, the execution control unit 604 includes: an action sequence communication subunit, configured to communicate the action sequence to the cloud phone service platform; a service platform control subunit, configured to control the cloud phone service platform to determine a target cloud phone for implementing the action sequence from a group of candidate cloud phones; control the cloud phone service platform to send the action sequence to the target cloud phone; and control the target cloud phone to execute the action sequence through the cloud phone service platform.

[0142] In some optional implementations of this embodiment, the cloud phone service platform communicates with the target cloud phone based on the Android debug bridge.

[0143] In some optional implementations of this embodiment, the device 600 also includes: an action video stream acquisition unit, configured to acquire the action video stream of the target cloud phone; a task progress feedback unit, configured to feedback the task progress information corresponding to the target preset video frame to the user in response to the similarity between the latest video frame in the action video stream and the target preset video frame in a group of preset video frames being greater than or equal to a similarity threshold.

[0144] In some optional implementations of this embodiment, the device 600 also includes: a voice feedback information generation unit, configured to generate voice feedback information in response to the target cloud phone returning the execution result of the action sequence, based on the text-to-speech transcription result of the execution result; and an execution result feedback unit, configured to communicate the voice feedback information to the user.

[0145] In some optional implementations of this embodiment, the action sequence composition unit 603 includes: a function identifier determination subunit, configured to determine the target function identifier based on the function description information; an action sequence composition subunit, configured to extract a set of target actions corresponding to the target function identifier from the robot process automation action library to form an action sequence.

[0146] In some optional implementations of this embodiment, the device 600 also includes: a robot process automation action library determination unit, which is configured to determine the robot process automation action library based on the following method: obtaining a sample terminal operation flow in a video format; extracting a video stream and a control flow from the sample terminal operation flow; based on the control flow and the video stream, converting the control flow into a specific control language to generate a conversion result; based on the function identifier of the conversion result, determining the corresponding robot process automation action; based on each robot process automation action, generating a robot process automation action library.

[0147] In some optional implementations of this embodiment, the task description information is for the cloud phone autonomous tasks under the graphical user interface.

[0148] This embodiment exists as a device embodiment corresponding to the above method embodiment. The device for controlling a cloud phone provided in this embodiment receives task description information for a cloud phone provided by a user in natural language; parses the task description information to obtain function description information; based on the function description information, extracts a set of target actions from the robot process automation action library to form an action sequence; sends the action sequence to the target cloud phone, and controls the target cloud phone to execute the action sequence. As a result, the cloud phone can be used and configured more conveniently by the user, reducing the difficulty and cost of using and configuring the cloud phone for the user. Accordingly, the cloud phone can also be used and configured in more business scenarios, expanding the use scenarios of the cloud phone.

[0149] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0150] Figure 7 A schematic block diagram of an example electronic device 700 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0151] like Figure 7As shown, the device 700 includes a computing unit 701, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 702 or a computer program loaded from a storage unit 708 into a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the device 700 can also be stored. The computing unit 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0152] A number of components in the device 700 are connected to the I / O interface 705, including: an input unit 706, such as a keyboard, a mouse, etc.; an output unit 707, such as various types of displays, speakers, etc.; a storage unit 708, such as a disk, an optical disk, etc.; and a communication unit 709, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 709 allows the device 700 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0153] The computing unit 701 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 701 performs the various methods and processes described above, such as a method for controlling a cloud phone. For example, in some embodiments, the method for controlling a cloud phone may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 708. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 700 via ROM 702 and / or communication unit 709. When the computer program is loaded into RAM 703 and executed by the computing unit 701, one or more steps of the method for controlling a cloud phone described above may be performed. Alternatively, in other embodiments, the computing unit 701 may be configured to perform a method for controlling a cloud phone in any other appropriate manner (e.g., by means of firmware).

[0154] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0155] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0156] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0157] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0158] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0159] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or a cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and virtual private servers (VPS) services. The server can also be divided into a server of a distributed system, or a server combined with a blockchain.

[0160] According to the technical solution of the embodiment of the present disclosure, the task description information for the cloud phone provided by the user in the form of natural language is received; the task description information is parsed to obtain the function description information; based on the function description information, a set of target actions are extracted from the robot process automation action library to form an action sequence; the action sequence is sent to the target cloud phone, and the target cloud phone is controlled to execute the action sequence. As a result, the cloud phone can be used and configured more conveniently by the user, reducing the difficulty and cost of using and configuring the cloud phone for the user. Accordingly, the cloud phone can also be used and configured in more business scenarios, expanding the use scenarios of the cloud phone.

[0161] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions provided by this disclosure can be achieved, and this document does not limit this.

[0162] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A method for controlling a cloud phone, comprising: Receiving task description information for the cloud phone provided by the user in natural language; Parsing the task description information to obtain function description information; Based on the functional description information, a set of target actions are extracted from a robotic process automation action library to form an action sequence; The action sequence is sent to the target cloud phone, and the target cloud phone is controlled to execute the action sequence.

2. The method according to claim 1, wherein: The step of parsing the task description information to obtain the function description information includes: The task description information is parsed using a large language model to obtain the function description information.

3. The method according to claim 1, wherein: The sending of the action sequence to the target cloud phone and controlling the target cloud phone to execute the action sequence includes: Communicating the action sequence to a cloud phone service platform; Controlling the cloud phone service platform to determine a target cloud phone for implementing the action sequence from a group of candidate cloud phones; Controlling the cloud phone service platform to send the action sequence to the target cloud phone; The target cloud phone is controlled by the cloud phone service platform to execute the action sequence.

4. The method according to claim 3, wherein: The cloud phone service platform communicates with the target cloud phone based on the Android debugging bridge.

5. The method according to claim 1, further comprising: Obtaining the action video stream of the target cloud phone; In response to the similarity between the latest video frame in the action video stream and a target preset video frame in a group of preset video frames being greater than or equal to a similarity threshold, task progress information corresponding to the target preset video frame is fed back to the user.

6. The method according to claim 1, further comprising: In response to the target cloud phone returning an execution result for the action sequence, generating voice feedback information based on a text-to-speech transcription result of the execution result; The voice feedback information is communicated to the user.

7. The method according to claim 1, wherein: Based on the functional description information, a set of target actions are extracted from the robot process automation action library to form an action sequence, including: Based on the function description information, determining a target function identifier; A group of target actions corresponding to the target function identifier is extracted from the robot process automation action library to form an action sequence.

8. The method according to claim 1, wherein: The robotic process automation action library is determined based on the following: Get sample terminal operation stream in video format; Extracting a video stream and a control stream from the sample terminal operation stream; Based on the control stream and the video stream, convert the control stream into a specific control language to generate a conversion result; Determining a corresponding robotic process automation action based on the function identifier of the conversion result; Based on each of the robotic process automation actions, a robotic process automation action library is generated.

9. The method according to any one of claims 1 to 8, wherein: The task description information is for the cloud phone autonomous tasks under the graphical user interface.

10. A device for controlling a cloud phone, comprising: An input receiving unit, configured to receive task description information for the cloud phone provided by a user in a natural language form; an information parsing unit, configured to parse the task description information to obtain function description information; An action sequence composition unit is configured to extract a set of target actions from a robot process automation action library based on the function description information to form an action sequence; The execution control unit is configured to send the action sequence to the target cloud phone and control the target cloud phone to execute the action sequence.

11. The device according to claim 10, wherein: The information parsing unit is further configured to parse the task description information using a large language model to obtain the function description information.

12. The device according to claim 10, wherein: The execution control unit comprises: an action sequence communication subunit, configured to communicate the action sequence to a cloud phone service platform; The service platform control subunit is configured to control the cloud phone service platform to determine a target cloud phone for implementing the action sequence from a group of candidate cloud phones; control the cloud phone service platform to send the action sequence to the target cloud phone; and control the target cloud phone to execute the action sequence through the cloud phone service platform.

13. The device according to claim 12, wherein: The cloud phone service platform communicates with the target cloud phone based on the Android debugging bridge.

14. The apparatus according to claim 10, further comprising: An action video stream acquisition unit, configured to acquire an action video stream of the target cloud phone; The task progress feedback unit is configured to feedback the task progress information corresponding to the target preset video frame to the user in response to the similarity between the latest video frame in the action video stream and the target preset video frame in a group of preset video frames being greater than or equal to a similarity threshold.

15. The apparatus according to claim 10, further comprising: A voice feedback information generating unit is configured to generate voice feedback information based on a text-to-speech transcription result of the execution result in response to the target cloud phone returning the execution result of the action sequence; The execution result feedback unit is configured to communicate the voice feedback information to the user.

16. The device according to claim 10, wherein: The action sequence component unit includes: a function identifier determination subunit, configured to determine a target function identifier based on the function description information; The action sequence composition subunit is configured to extract a group of target actions corresponding to the target function identifier from the robot process automation action library to form an action sequence.

17. The apparatus according to claim 10, further comprising: The robot process automation action library determining unit is configured to determine the robot process automation action library based on the following method: Get sample terminal operation stream in video format; Extracting a video stream and a control stream from the sample terminal operation stream; Based on the control stream and the video stream, convert the control stream into a specific control language to generate a conversion result; Determining a corresponding robotic process automation action based on the function identifier of the conversion result; Based on each of the robotic process automation actions, a robotic process automation action library is generated.

18. The device according to any one of claims 10 to 17, wherein: The task description information is for the cloud phone autonomous tasks under the graphical user interface.

19. An intelligent agent, comprising: An input receiving module is configured to receive task description information for the cloud phone provided by a user in a natural language form; An information parsing module, configured to parse the task description information to obtain function description information; an action sequence composition module, configured to extract a set of target actions from a robot process automation action library based on the function description information to form an action sequence; The execution control module is configured to send the action sequence to the target cloud phone and control the target cloud phone to execute the action sequence.

20. An electronic device, comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method for controlling a cloud phone according to any one of claims 1-9.

21. A non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable the computer to execute the method for controlling a cloud phone according to any one of claims 1-9.

22. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements the method for controlling a cloud phone according to any one of claims 1 to 9.

Images