Android mobile phone intelligent control method and system based on large model agent

Abstract

The invention belongs to the technical field of artificial intelligence, and relates to an Android mobile phone intelligent control method and system based on a large model agent, and the method comprises the steps: 1) obtaining a task inputted by a user; 2) judging whether the task input by the user is within the processing range of the Agent model, if so, entering step 3), otherwise, ending the execution of the task input by the user; 3) obtaining page information of the Android mobile phone; (4) judging and processing the non-standard state of the page; 5) carrying out compression simplification and page completion processing on the obtained page XML; 6) inputting the processed page XML and the processed page screenshot into an Agent model, and outputting the type and the position of the operation executed by the Android mobile phone in the next step by the Agent model; and (7) according to the output of the Agent model, operation execution is carried out on the Android mobile phone on the basis of the accessible service (AccessibilityService). The mobile phone page state can be recognized and judged, and operation is planned and executed in combination with the intention of a user.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence technology, and relates to an Android mobile phone intelligent control method and system, and in particular to an Android mobile phone intelligent control method and system based on a large model intelligent body. Background Art

[0002] Smart terminals such as mobile phones are excellent carriers for the implementation of big model technology. The combination of smart terminals and big models can bring users subversive product experience and highly intelligent services. Currently, the smart assistants and RPA tools on the market still have multiple limitations in assisting users to complete operations. The scope of application of their functions is extremely limited, and they are not suitable for highly variable tasks. The initial deployment and maintenance costs are high, and there are also many difficulties in understanding users' complex language instructions and context associations.

[0003] Among them, mobile smart assistants such as Siri perform tasks involving application operations in a variety of ways. These methods include using deep integration and API access to start and operate third-party applications through the Intent mechanism provided by the operating system (such as in the Android system), and using App Shortcut and Siri Shortcuts to activate shortcut commands within the application. In addition, the mobile smart assistant first understands the user's intentions through natural language processing (NLP), then converts these intentions into specific tasks, and interacts with the application through protocols and APIs. Sometimes, mobile smart assistants also support dynamic frameworks and plug-in systems, allowing developers to create specific plug-ins to achieve customized communication. For example, users can send text messages, play music, or open applications through Siri, and these tasks are implemented through APIs such as SiriKit or MusicKit. Such a variety of methods ensure that smart assistants can perform a wide range of tasks, thereby improving user convenience and system flexibility. However, for tasks related to operation execution proposed by users, mobile smart assistants generally implement task execution by calling APIs or SDKs. However, not all applications are willing to open their APIs to smart assistants, which limits smart assistants in certain tasks. In addition, complex function calls of some applications require specific permissions, and smart assistants may not be able to access them directly, and can only assist users in completing simpler tasks.

[0004] The method of implementing automatic task execution by RPA (Robotic Process Automation) tools on Android phones developed by third parties mainly includes UI automation through screen coordinate clicks, image recognition, OCR and other technologies; control identification by means of control ID, class name or XPath; scripting for complex operations; recording and repeating user operations through recording and playback functions; directly calling Android system API; and analyzing and intercepting network requests. The whole process involves preparing equipment, defining and recording tasks, debugging and testing, and finally deploying and running to achieve task automation. Although using RPA technology to automate tasks can significantly improve efficiency, there are also some limitations, such as script failure caused by UI changes, limited processing capabilities for complex tasks, compatibility issues caused by application updates and system updates, and performance and resource consumption during runtime. In addition, RPA also faces challenges in anti-interference ability and multi-tasking. RPA usually requires users to build their own workflow to complete it, and the high threshold for use is also one of the problems faced by the current implementation and promotion of related technologies.

[0005] AI Agent is a software system based on artificial intelligence technology. It can be designed to autonomously perform specific tasks or solve problems. It can run in a specific environment and has certain autonomous decision-making and task planning capabilities. The agent analyzes the task based on the "brain" composed of a large model, designs and plans problem solutions and paths, and has the ability to call tools and perform operations. As a new technical solution, it can be used to solve many problems faced by mobile phone intelligent control tasks. However, there is currently no intelligent control method for Android mobile phones based on large model agents. Even if some mobile phone intelligent control methods use large model technology, they still have certain defects in page information acquisition, non-standard page judgment processing, and page XML processing.

[0006] Therefore, in view of the defects existing in the above-mentioned prior art, it is necessary to develop a new type of intelligent control method for Android mobile phones based on large model intelligent agents. Summary of the invention

[0007] In order to overcome the defects of the prior art, the present invention proposes an intelligent control method and system for an Android mobile phone based on a large model intelligent body, which can identify and judge the status of the mobile phone page, and plan and execute operations based on the user's intention.

[0008] In order to achieve the above object, the present invention provides the following technical solutions:

[0009] An Android mobile phone intelligent control method based on a large model agent, characterized in that it includes the following steps:

[0010] 1) The task of obtaining user input;

[0011] 2) Determine whether the task input by the user is within the processing range of the Agent model. If so, proceed to step 3). Otherwise, terminate the execution of the task input by the user.

[0012] 3) Obtaining page information of an Android phone, wherein the page information includes page XML and page screenshots;

[0013] 4) Based on the page information, a determination process is performed on the non-standard state of the page to ensure that the page information obtained is that of a standard page;

[0014] 5) compressing and simplifying the acquired page XML and performing page completion processing to obtain the processed page XML;

[0015] 6) Input the processed page XML and page screenshots into the Agent model, and the Agent model outputs the type and location of the next operation performed by the Android phone;

[0016] 7) According to the output of the Agent model, operations are performed on the Android phone based on the Accessibility Service.

[0017] Preferably, the step 3) specifically includes:

[0018] 31) Based on the accessibility service AccessibilityService, obtain the accessibility service node information AccessibilityServiceNodeInfo and the page screenshot Screenshot of the Android phone page;

[0019] 32) Parse the node Node in the accessibility service node information AccessibilityServiceNodeInfo, and convert the parsing result into a variable character sequence StringBuilder;

[0020] 33) Adjust the structure of the variable character sequence StringBuilder, and parse the child nodes ChildNod of the corresponding node Node in parallel;

[0021] 34) Obtain the parsing results of each child node ChildNod and assemble them into a variable character sequence StringBuilder, and adjust the structure of the assembled variable character sequence StringBuilder again;

[0022] 35) Obtain the final page XML based on the variable character sequence StringBuilder that has been adjusted again;

[0023] 36) Performing quality compression on the page screenshot Screenshot without changing the original resolution of the page screenshot Screenshot to obtain a final page screenshot.

[0024] Preferably, the step 4) specifically includes:

[0025] 41) Page waiting processing, which is used to process a page that needs to wait before it can become a standard page, so as to ensure that the page information of a page that does not need to wait before it can become a standard page is obtained;

[0026] 42) Pop-up window processing, which is used to handle the situation where a pop-up window exists in a page to ensure that the page information of a page without a pop-up window is obtained.

[0027] Preferably, the step 41) specifically includes:

[0028] 411) Get the page XML of the Android phone again;

[0029] 412) Determine whether the difference between the total number of nodes in the page XMLs obtained twice is less than D, and determine whether the position attributes of the first k nodes in the page XMLs obtained twice are completely consistent. If the difference is less than D and the position attributes are completely consistent, the page XML obtained again is used as the page XML of the page that does not need to wait before becoming a standard page; if the difference is not less than D and / or the position attributes are not completely consistent, determine whether the total time consumed is greater than or equal to T seconds. If it is greater than or equal to T seconds, the page XML obtained again is used as the page XML of the page that does not need to wait before becoming a standard page. If it is less than T seconds, return to step 411);

[0030] 413) Get the page screenshot of the Android phone again;

[0031] 414) Input the page screenshot obtained again into the single-image Wait model and input the page screenshots obtained twice adjacently into the dual-image Wait model to respectively determine whether the loading is completed. If the output results of the single-image Wait model and the dual-image Wait model are both loading completed, the page screenshot obtained again is used as a page screenshot of the page that does not need to be in a waiting state before it can be changed into a standard page; if the output result of the single-image Wait model and / or the dual-image Wait model is that the loading is not completed, determine whether the number of cycles is greater than or equal to n times. If it is greater than or equal to n times, the page screenshot obtained again is used as a page screenshot of the page that does not need to be in a waiting state before it can be changed into a standard page. If it is less than n times, return to step 413).

[0032] Preferably, the step 42) specifically includes:

[0033] 421) Get page screenshots on Android phones;

[0034] 422) Detect whether there is a pop-up window in the page screenshot through the pop-up window detection model. If not, take the page screenshot as a screenshot of a page without a pop-up window; if so, determine whether it has been cycled once. If it has been cycled once, leave it to the user to operate. If it has not been cycled once, leave it to the pop-up window closing model and the pop-up window closing model will try to close the pop-up window. If the pop-up window closing model can return to the closed position, the pop-up window will be automatically closed according to the closed position and return to step 421). If the pop-up window closing model cannot return to the closed position, leave it to the user to operate.

[0035] Preferably, the step 5) specifically includes:

[0036] 51) Determine whether it is necessary to retain the off-screen nodes in the page XML, if not, proceed to step 52), if yes, proceed to step 53);

[0037] 52) Delete off-screen nodes in the page XML;

[0038] 53) Delete redundant nodes in page XML;

[0039] 54) Determine whether the page of the Android phone is a special page, if yes, proceed to step 55), if no, proceed to step 56);

[0040] 55) Delete or add information to special pages;

[0041] 56) using an optical character recognition method to identify all texts and locations of the texts in the page of the Android phone, comparing the texts recognized by the optical character recognition method with the texts in the page XML, removing the texts in the texts recognized by the optical character recognition method that are repeated with the texts in the page XML, and inserting the remaining texts into the page XML;

[0042] 57) Simplify the attributes in the page XML after insertion.

[0043] Preferably, the Android mobile phone intelligent control method based on the large model agent further comprises:

[0044] 8) Display the results of the operations executed on the Android phone.

[0045] In addition, the present invention also provides an Android mobile phone intelligent control system based on a large model intelligent agent, which is characterized by comprising:

[0046] A task input module, which is used to obtain tasks input by users;

[0047] The task discrimination model is used to determine whether the task input by the user is within the processing range of the Agent model, so as to determine whether to continue to execute the task input by the user;

[0048] A data acquisition module, which is used to obtain page information of an Android phone, wherein the page information includes page XML and page screenshots;

[0049] A page status judgment processing module, which is used to judge the non-standard status of the page based on the page information to ensure that the page information obtained is that of a standard page;

[0050] A data processing module is used to compress and simplify the acquired page XML and complete the page to obtain a processed page XML;

[0051] Agent model, which is used to output the type and location of the next operation to be performed by the Android phone based on the processed page XML and page screenshots;

[0052] The task execution module is used to perform operations on the Android phone based on the accessibility service AccessibilityService according to the output of the Agent model.

[0053] Moreover, the present invention also provides an Android mobile phone intelligent control device based on a large model intelligent body, which is characterized by comprising:

[0054] one or more processors;

[0055] A memory for storing one or more programs;

[0056] When the one or more programs are executed by the one or more processors, the one or more processors implement the Android mobile phone intelligent control method based on the large model intelligent agent as described above.

[0057] Finally, the present invention provides a computer-readable storage medium having a computer program stored thereon, characterized in that when the program is executed by a processor, the steps of the method for intelligent control of an Android mobile phone based on a large model intelligent agent as described above are implemented.

[0058] Compared with the prior art, the Android mobile phone intelligent control method and system based on the large model intelligent body of the present invention has one or more of the following beneficial technical effects:

[0059] 1. The present invention can identify and judge the status of mobile phone pages, and plan and execute operations based on user intentions.

[0060] 2. The present invention has a wide range of applications, low initial deployment and maintenance costs, and no difficulty in accurately understanding user complex language instructions and context associations. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 It is a flow chart of the intelligent control method of an Android mobile phone based on a large model intelligent body of the present invention.

[0062] Figure 2 It is a flow chart of obtaining page information of an Android mobile phone in the present invention.

[0063] Figure 3 It is a flow chart of the determination process of the non-standard state of a page in the present invention.

[0064] Figure 4 It is a flow chart of page waiting processing in the present invention.

[0065] Figure 5 It is a flow chart of pop-up window processing in the present invention.

[0066] Figure 6 This is a flowchart of the present invention for compressing and simplifying the acquired page XML and performing page completion processing.

[0067] Figure 7 It is a schematic diagram of the structure of the Android mobile phone intelligent control system based on the large model intelligent body of the present invention. DETAILED DESCRIPTION

[0068] Before describing in detail any embodiment of the present invention, it should be understood that the present invention is not limited in its application to the construction and arrangement details of the components described below or illustrated in the following figures. The present invention can have other embodiments and can be practiced or carried out in various ways. In addition, it should be understood that the words and terms used here are for descriptive purposes and should not be considered restrictive. The use of "including" or "having" and its variations herein is intended to cover the items and their equivalents and additional items displayed below. Unless otherwise specified or limited, the terms "install", "connect", "support" and "couple" and their variations are widely used and cover direct installation and indirect installation, connection, support and connection. In addition, "connect" and "couple" are not limited to physical or mechanical connections or connections.

[0069] Furthermore, on the first aspect, in the disclosure of the present invention, the orientation or positional relationship indicated by terms such as "longitudinal", "transverse", "upper", "lower", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside" and "outside" are based on the orientation or positional relationship shown in the accompanying drawings, which are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operate in a specific orientation, and therefore the above terms cannot be understood as limitations on the present invention; on the second aspect, the term "one" should be understood as "at least one" or "one or more", that is, in one embodiment, the number of an element may be one, while in another embodiment, the number of the element may be multiple, and the term "one" cannot be understood as a limitation on the quantity.

[0070] Before introducing the specific content of the present invention, some technical terms used in the present invention are briefly introduced to help those skilled in the art to better understand the present invention.

[0071] 1. RPA, the full name of which is Robotic Process Automation, is a technology that uses software robots to imitate regular, repetitive tasks performed by humans on computers. RPA robots usually automatically perform tasks such as data input, data processing, and data transmission through user interface interactions, thereby greatly improving efficiency and reducing human errors.

[0072] 2. Android's Accessibility Service was originally designed to help users with visual and hearing impairments to operate their phones more conveniently. This special permission allows applications to perform some operations that are usually only available to users, such as clicking buttons, sliding screens, and entering text, so it is widely used in areas such as automated testing and task automation.

[0073] 3. ASR is the abbreviation of Automatic Speech Recognition. This technology can convert voice signals into text. Important applications of ASR include voice assistants (such as Siri, Google Assistant), voice input methods, and smart devices with voice control functions.

[0074] 4. XML (Extensible Markup Language) is a widely used format in Android development for defining user interface layouts, resource files, and various configuration files. It is mainly used to define the UI layout of an application. Developers can describe the location, attributes, and values ​​of UI components through a hierarchical XML structure. These layout files usually exist in the form of files, which can intuitively display the interface design, thereby improving the readability and maintainability of the code.

[0075] 5. Bounding Box (bounding box of a node), referred to as bbox. In the accessibility service developed by Android, the AccessibilityNodeInfo class is used to describe the elements in the view hierarchy, each element is called a node. The Bounding Box defines its position and size on the screen. It is a rectangular area used to accurately describe the boundary position of the node on the screen, including the left boundary, top boundary, right boundary and bottom boundary.

[0076] Figure 1 The flowchart of the method for intelligently controlling an Android mobile phone based on a large model agent of the present invention is shown. Figure 1 As shown, the Android mobile phone intelligent control method based on the large model intelligent agent of the present invention includes the following steps:

[0077] 1. The task of obtaining user input.

[0078] In the present invention, the user can input the task in two forms: voice and text. If the user inputs the task in the form of voice, the client of the Android phone calls the ASR tool to convert the input voice into text.

[0079] Second, determine whether the task input by the user is within the processing range of the Agent model. If so, continue processing and enter step three. Otherwise, it means that the Agent model cannot process the task input by the user, and the execution of the task input by the user is terminated.

[0080] In the present invention, a task discrimination model can be used to determine whether the task input by the user is within the processing range of the Agent model. The task discrimination model can be a binary classifier obtained by fine-tuning and training based on the GLM-4-9B model. Before use, some data with input as user-input tasks and output as the operation type that the Agent model can process can be collected as a training data set, and the GLM-4-9B model can be fine-tuned and trained using the training data set to obtain the task discrimination model. This part of the content is a technology that can be understood and mastered by technicians in the field of large language models. For the sake of simplicity, it will not be described in detail here.

[0081] 3. Obtain page information of the Android phone, wherein the page information includes page XML and page screenshots.

[0082] like Figure 2 As shown, in the present invention, obtaining the page information of the Android phone specifically includes:

[0083] 1. Get the accessibility service node information AccessibilityServiceNodeInfo and page screenshot Screenshot of the Android phone page based on the accessibility service AccessibilityService.

[0084] 2. Parse the node Node in the accessibility service node information AccessibilityServiceNodeInfo, and convert the parsing result into a mutable character sequence StringBuilder.

[0085] In the present invention, the accessibility service node information AccessibilityServiceNodeInfo is passed into RecursiveTask (a subclass of ForkJoinPool, specifically used for recursive tasks with return values) to perform node Node parsing. RecursiveTask parses the passed-in node Node to obtain key fields, and then converts it into StringBuilder. In addition, when parsing the node Node, it is parsed from the root node RootNode first.

[0086] The RecursiveTask code logic is:

[0087] (1) Parameter passing. The parameters passed include: Node, which is the Node object to be parsed; index, which is the order of the current node Node in the parent node FatherNode. When assembling the results, it is spliced ​​according to the index order to prevent the XML structure from being disordered; depth, which is the depth of the current node Node in the node tree NodeTree. When the format is indented, it is used to fill in the number of spaces.

[0088] (2) Method. The method is compute(), which is the core method of RecursiveTask. If the current task needs to be split, the task is split into more subtasks, and the fork() method is used to process these subtasks in parallel. After the parsing is completed, the join() method is called to obtain the result of the corresponding task. The node parsing logic is executed asynchronously in this method.

[0089] In the compute() method, the key fields of the current node Node are obtained first, and then the child node ChildNode of the current node Node is obtained. Then, according to the number of child nodes ChildNode, it is split into a fixed number of subtasks, and each child node ChildNode and the sequential index of the current node Node are passed to the new RecursiveTask, and fork() is called to execute the parsing task of each child node ChildNode, and the parsing result of the child node ChildNode is obtained through join().

[0090] In the present invention, the node Node in the accessibility service node information AccessibilityServiceNodeInfo is parsed, and the parsing result is converted into a variable character sequence StringBuilder, which specifically includes:

[0091] (1) Create an XmlSerializer object (the present invention uses the XmlSerializer object to ensure that the data complies with the XML specification), and set the StringWriter object and the startDocument attribute and the startTag attribute. The sample code for the specific settings is as follows:

[0092] valstringWriter=StringWriter()

[0093] valserializer=Xml.newSerializer()

[0094] serializer.setOutput(stringWriter)

[0095] serializer.startDocument("UTF-8",true)

[0096] serializer.startTag("","node").

[0097] (2) Obtain the key fields of the node Node in the accessibility service node information AccessibilityServiceNodeInfo.

[0098] The node itself carries some information fields, and the key fields of the node can be obtained directly through method calls. The following is an example of how the XmlSerializer object loads the key fields of the node:

[0099] serializer.attribute("","class",node.className.toString());

[0100] serializer.attribute("","text",node.text.toString()).

[0101] Among them, node.className can get the information of the className field of the node Node, and node.text can get the information of the text field of the node Node.

[0102] In the present invention, the key fields of the node Node include: className, contentDescription, text, viewIdResourceName, isClickable, isEnabled, isChecked, isFocused, isFocusable, isScrollable, isSelected, isLongClickable and boundsInScreen.

[0103] (3) Set the end tag endTag property and the end document endDocument property of the XmlSerialize object, insert the key field of the obtained node Node into the XmlSerializer object, and use the StringWriter object to convert it into a variable character sequence StringBuilder. The sample code for the specific settings is as follows:

[0104] serializer.endTag("","node")

[0105] serializer.endDocument()

[0106] valstringBuilder=StringBuilder()

[0107] stringBuilder.append(stringWriter.toString()).

[0108] Here, stringWriter is the StringWriter object set in step (1).

[0109] Thus, an exemplary StringBuilder is obtained as follows:

[0110] <? xml version='1.0'encoding='UTF-8'standalone='yes'? > <node NAF="true" index="0" text="" resource-id=""class="android.view.ViewGroup"package="com.zhipu.agent"content-desc=""checkable="false"checked="false"clickable="false"enabled="true"focusable="false"focused="false"scrollable="false"long-clickable="false"password="false"selected="false"

[0111] bounds="[0,148][1224,2776]" / >.

[0112] 3. Adjust the structure of the variable character sequence StringBuilder, and perform parallel parsing on the child nodes ChildNod of the corresponding node Node. Specifically, it includes:

[0113] (1) Remove the version, encoding, and standalone information of the XmlSerializer object in the mutable character sequence StringBuilder, and only keep the version, encoding, and standalone information of the outermost node Node to avoid duplication. The sample code is as follows:

[0114] StringBuilder.toString().replace("<?xml version='1.0'encoding='UTF-8'

[0115] standalone='yes'? >","").

[0116] (2) Determine whether the node Node corresponding to the variable character sequence StringBuilder contains a child node ChildNode. If the node Node corresponding to the variable character sequence StringBuilder contains a child node ChildNode, remove the end tag endTag of the node Node in the variable character sequence StringBuilder, otherwise a structural error will occur when assembling the child node ChildNode. The sample code is as follows:

[0117] StringBuilder.toString().replace(" / >",">").

[0118] (3) Obtain the key fields of each child node ChildNod in parallel and set a depth parameter for each child node ChildNod to record the depth of each child node ChildNod. Save the key fields and depth parameters of each child node ChildNod obtained into RecursiveTaskList.

[0119] The key fields of each child node ChildNod are obtained in the same manner as the key fields of the node Node in step 2, and will not be described for simplicity. However, the key fields of multiple child nodes ChildNod of a node Node may be obtained in parallel, thereby improving parsing efficiency.

[0120] Of course, in order to facilitate subsequent assembly, the index of each child node ChildNod also needs to be saved in RecursiveTaskList.

[0121] 4. Obtain the parsing results of each child node ChildNod and assemble them into a variable character sequence StringBuilder, and adjust the structure of the assembled variable character sequence StringBuilder again. Specifically, it includes:

[0122] (1) Traverse the RecursiveTaskList and obtain the key fields and depth parameters of each child node ChildNod. Of course, if there are multiple child nodes ChildNod, it is also necessary to obtain the index of each child node ChildNod.

[0123] (2) The key fields of each child node ChildNod obtained are concatenated into a variable character sequence StringBuilder, and the number of indentation spaces is set according to the depth parameter to complete the format indentation. The sample code of the format indentation is as follows:

[0124] StringBuilder.append("\r\n").append("".repeat(depth)).

[0125] Among them, "\r\n" is a line break character, and depth is the depth of the current child node ChildNod.

[0126] The sample code for concatenating the key fields of each child node ChildNod obtained into a variable character sequence StringBuilder is as follows:

[0127] StringBuilder.append(childNodeStr).

[0128] Among them, childNodeStr is the result of parsing the child node ChildNode. If there are multiple child nodes ChildNode, childNodeStr needs to be assembled according to the index order to prevent the overall structure from being chaotic.

[0129] (3) After all the key fields of the child nodes ChildNod are concatenated into the variable character sequence StringBuilder, the end tag endTag of the variable character sequence StringBuilder is completed. The sample code is as follows:

[0130] StringBuilder.append("\r\n").append("".repeat(depth-1)).append("").

[0131] Thus, starting from the parsing of the root node RootNode, through the gradual parsing and splicing of its child node ChildNod and the child node ChildNod of the child node ChildNod, all the nodes Node in the accessibility service node information AccessibilityServiceNodeInfo of the page will be assembled into a StringBuilder.

[0132] 5. Get the final page XML based on the adjusted variable character sequence StringBuilder.

[0133] The final StringBuilder, which is the final XML of the current page, can be obtained through the ForkJoinPool.invoke() method.

[0134] An example final XML is as follows:

[0135] <? xml version='1.0'encoding='UTF-8'standalone='yes'? >

[0136] <hierarchy rotation="0">

[0137] <node index="0"text=""resource-id=""

[0138] class="android.widget.FrameLayout"package="com.autonavi.minimap"content-desc=""checkable="false"checked="false"clickable="false"enabled="true"focusable="true"focused="false"scrollable="false"long-clickable="false"password="false"selected="false"

[0139] bounds="[0,0][1224,2776]">

[0140] <node index="0"text=""resource-id=""

[0141] class="android.widget.LinearLayout"package="com.autonavi.minimap"content-desc=""checkable="false"checked="false"clickable="false"enabled="true"focusable="false"focused="false"scrollable="false"long-clickable="false"password="false"selected="false"

[0142] bounds="[0,0][1224,2776]">

[0143] <node index="0"text=""resource-id="android:id / content"

[0144] class="android.widget.FrameLayout"package="com.autonavi.minimap"content-desc=""checkable="false"checked="false"clickable="false"enabled="true"focusable="false"focused="false"scrollable="false"long-clickable="false"password="false"selected="false"

[0145] bounds="[0,0][1224,2776]">

[0146] <node index="0"text=""

[0147] resource-id="com.autonavi.minimap:id / root_view"

[0148] class="android.widget.LinearLayout"package="com.autonavi.minimap"content-desc=""checkable="false"checked="false"clickable="false"enabled="true"focusable="false"focused="false"scrollable="false"long-clickable="false"password="false"selected="false"

[0149] bounds="[0,0][1224,2776]">

[0150] <node index="0"text=""

[0151] resource-id="com.autonavi.minimap:id / fl_content_view"

[0152] class="android.widget.FrameLayout"package="com.autonavi.minimap"content-desc=""checkable="false"checked="false"clickable="false"enabled="true"focusable="false"focused="false"scrollable="false"long-clickable="false"password="false"selected="false"

[0153] bounds="[0,0][1224,2776]">

[0154] <node index="0"text=""

[0155] resource-id="com.autonavi.minimap:id / map_container"

[0156] class="android.widget.FrameLayout"package="com.autonavi.minimap"content-desc=""checkable="false"checked="false"clickable="false"enabled="true"focusable="false"focused="false"scrollable="false"long-clickable="false"password="false"selected="false"

[0157] bounds="[0,0][1224,2776]">

[0158] <node index="0"text=""

[0159] resource-id="com.autonavi.minimap:id / atmapsView"

[0160] class="android.widget.FrameLayout"package="com.autonavi.minimap"content-desc=""checkable="false"checked="false"clickable="false"enabled="true"focusable="false"focused="false"scrollable="false"long-clickable="false"password="false"selected="false"

[0161] bounds="[0,0][1224,2776]">

[0162] <node index="0"text=""resource-id=""class="android.view.View"package="com.autonavi.minimap"content-desc=""checkable="false"checked="false"clickable="false"enabled="true"focusable="false"focused="false"scrollable="false"long-clickable="false"password="false"selected="false"bounds="[0,0][1224,2776]" / >

[0163]

[0164]

[0165] <node index="1"text=""

[0166] resource-id="com.autonavi.minimap:id / temporary_layer"

[0167] class="android.widget.FrameLayout"package="com.autonavi.minimap"content-desc=""checkable="false"checked="false"clickable="false"enabled="true"focusable="false"focused="false"scrollable="false"long-clickable="false"password="false"selected="false"

[0168] bounds="[0,0][1224,2776]" / >

[0169] <node index="2"text=""resource-id=""

[0170] class="android.widget.RelativeLayout"package="com.autonavi.minimap"content-desc=""checkable="false"checked="false"clickable="false"enabled="true"focusable="false"focused="false"scrollable="false"long-clickable="false"password="false"selected="false"

[0171] bounds="[0,0][1224,2776]">

[0172] <node index="0"text=""

[0173] resource-id="com.autonavi.minimap:id / notification_layer"

[0174] class="android.widget.FrameLayout"package="com.autonavi.minimap"content-desc="" checkable="false" checked="false" clickable="false" enabledd="true" focusable="false" focused="false" scrollable="false" long-clickable="false" password="false" selected ="false"

[0175] bounds="[0,0][1224,2776]" / >

[0176]

[0177]

[0178]

[0179]

[0180]

[0181]

[0182] .

[0183] 6. Perform quality compression on the page screenshot Screenshot without changing the original resolution of the page screenshot Screenshot to obtain a final page screenshot.

[0184] The screenshot Screenshot may be compressed in quality without changing the original resolution of the screenshot Screenshot, so as to speed up the image transmission efficiency and reduce the traffic consumption, thereby obtaining the final screenshot of the page.

[0185] In the present invention, the Bitmap.compress() method can be used to compress the Bitmap image data after the screenshot, and the method signature is: public boolean compress(Bitmap.CompressFormat format, int quality, OutputStream stream). Among them, the parameter format is the format of the compressed image, and the Bitmap.CompressFormat enumeration is used to specify the format, such as JPEG, PNG and WEBP; the parameter quality is the quality of the compressed image, and the integer range is 0 to 100, where 0 represents the lowest quality (minimum compression) and 100 represents the highest quality (minimum compression). If the format is PNG, this parameter will be ignored because PNG is lossless compression; the parameter stream is the output stream to which the image compression data will be written.

[0186] The present invention obtains the child node ChildNode information by parallel parsing based on the accessibility service node information AccessibilityServiceNodeInfo, obtains the information of multiple child nodes ChildNode at the same time, and then sequentially combines the obtained child node ChildNode information to obtain the complete XML content, which greatly improves the time. Moreover, the present invention retains the information of the corresponding node Node at the end of each node Node parsing, and then reversely completes the node tree NodeTree to obtain the complete XML, which improves the efficiency while ensuring the integrity of the page information. Through a large number of experiments, it is found that under the same mobile phone interface, the speed of obtaining page information by the present invention saves nearly half of the time of recursive parsing and recursive parsing, and the integrity of XML is also well guaranteed.

[0187] Fourth, based on the page information, a judgment process is performed on the non-standard state of the page to ensure that the page information obtained is that of a standard page.

[0188] The existing technology for parsing and processing Android GUI (Graphical User Interface) pages is based on standard pages (completely loaded, no pop-up windows appear). However, in the process of the Agent model actually operating Android phones, non-standard page states (such as waiting for loading, page jumps, pop-up windows, etc.) are often encountered. The existing technology cannot judge and process these non-standard pages, so it will fail. The problem to be solved in this step is to distinguish these non-standard page states from standard page states and provide a method for processing non-standard pages.

[0189] like Figure 3 As shown, in the present invention, the determination process of the non-standard state of a page specifically includes:

[0190] 1. The page is waiting to be processed.

[0191] Page waiting processing is used to process "page loading", "page jumping" and other pages that need to wait for a while before becoming a standard page, so as to ensure that the page information of a page that does not need to wait before becoming a standard page (ie, a standard page) is obtained.

[0192] Figure 4 The flowchart of the page waiting process in the present invention is shown. This process can be effective at any time when the application is running on an Android phone. After this process, the page XML and page screenshots obtained can be guaranteed not to be in non-standard page states such as "page loading" and "page jumping".

[0193] In the present invention, the page waiting processing is divided into two parallel processing processes: the XML (Extensible Markup Language) side and the screenshot side. The core idea is to cyclically obtain the page XML and page screenshots of the Android phone's page, and judge whether it is no longer necessary to wait based on the page XML and page screenshots respectively. If the judgment conclusion is that waiting is still required, enter the next cycle until the judgment conclusion is that waiting is no longer required or the number of cycles or the time consumed exceeds a preset threshold.

[0194] Specifically, Figure 4 As shown, the page waiting for processing specifically includes:

[0195] (1) Get the page XML and page screenshots of the Android phone's page.

[0196] In the present invention, since step three has already obtained the page XML and page screenshot of the page of the Android phone, this step can be omitted when the page is waiting for processing.

[0197] After obtaining the page XML and page screenshots of the Android phone's page, it is divided into two parallel processing processes: XML (Extensible Markup Language) side and screenshot side.

[0198] The processing process on the XML side is as follows:

[0199] (2) Get the page XML of the Android phone's page again.

[0200] You can still use the method in step 3 to get the page XML of the Android phone's page again.

[0201] (3) Determine whether the difference in the total number of nodes (Node class nodes in the present invention) in the page XML obtained twice is less than D, and determine whether the position attributes of the first k nodes (Node class nodes in the present invention) in the page XML obtained twice are completely consistent. If the difference is less than D and the position attributes are completely consistent, it means that it is not in a non-standard page state such as "page loading" or "page jump", and the page XML obtained again can be used as the page XML of the page that does not need to wait before it can become a standard page; if the difference is not less than D and / or the position attributes are not completely consistent, determine whether the total time consumed on the XML side is greater than or equal to T seconds. If it is greater than or equal to T seconds, the page XML obtained again is used as the page XML of the page that does not need to wait before it can become a standard page. If it is less than T seconds, return to step (2), obtain the page XML of the page of the Android phone again, and start the next cycle.

[0202] In the present invention, by traversing the obtained XML file tree, the total number of node class nodes and the difference calculation can be realized. The XML file tree is traversed in pre-order to obtain the first k node class nodes and their bounds attributes (the bounds attribute represents the position pixel value of the element on the screen, which is a position attribute). And, if the total time consumed on this side is greater than or equal to T seconds, the loop is jumped out to avoid an infinite loop.

[0203] Preferably, D is between 10 and 20, k is between 5 and 10, and T is between 1.5 and 2.5.

[0204] The processing process on the screenshot side is as follows:

[0205] (4) Take a screenshot of the page on the Android phone again.

[0206] You can still use the method in step three to get a screenshot of the page on your Android phone again.

[0207] (5) Input the page screenshot obtained again into the single-image Wait model and input the page screenshots obtained twice adjacently into the dual-image Wait model to respectively determine whether the loading is complete. If the output results of the single-image Wait model and the dual-image Wait model are both loading completed, the page screenshot obtained again is used as a page screenshot of the page that does not need to wait before it can be changed to a standard page; if the output result of the single-image Wait model and / or the dual-image Wait model is that the loading is not complete, it is determined whether the number of cycles is greater than or equal to n times (preferably, n is between 5 and 10). If it is greater than or equal to n times, the page screenshot obtained again is used as a page screenshot of the page that does not need to wait before it can be changed to a standard page. If it is less than n times, return to step (4) and start the next cycle.

[0208] Among them, the single-image Wait model mainly focuses on the following situations: (1) there is an obvious prompt on the page that it is in a loading state; (2) the page is obviously in a transition state between two pages. The dual-image Wait model mainly focuses on the following situations that the single-image wait model cannot handle: (1) no page jump occurs, but some changes in the page have not yet ended; (2) a page jump occurs, but due to various reasons (such as good mobile phone performance and network conditions), the screenshots before and after are relatively complete. Therefore, through the single-image Wait model and the dual-image Wait model, it is possible to realize the judgment of various "page loading", "page jump" and other pages that require a short wait before becoming a standard page.

[0209] In the present invention, the single-image Wait model and the dual-image Wait model are both trained end-to-end prediction models based on convolutional neural networks. In specific use, CNN (convolutional neural network) can be used as the single-image Wait model and the dual-image Wait model, and training data can be collected and manually annotated, and the constructed single-image Wait model and the dual-image Wait model can be trained with the annotated training data to obtain the single-image Wait model and the dual-image Wait model.

[0210] The training data of the single-image Wait model are pages that are clearly marked as being in a loading state after being manually annotated, and pages that are clearly in a transitional state of two-page jumps after being manually annotated. The training data of the dual-image Wait model are pages that have not undergone page jumps after being manually annotated, but some changes within the page have not yet been completed, and pages that have undergone page jumps after being manually annotated, but the screenshots before and after are relatively complete due to various reasons.

[0211] 2. Pop-up window processing.

[0212] Pop-up processing is used to handle the situation where there are pop-up windows in the page, so as to ensure that the page information of a page without pop-up windows (i.e., standard page) is obtained. In other words, pop-up processing is responsible for handling the situation where there are "pop-up windows" in the page. Here, "pop-up windows" mainly refer to unexpected interactive windows that appear on the operation path. The appearance of such windows is uncertain and interferes with and hinders the normal completion of the original task.

[0213] Figure 5 FIG. 2 shows a flowchart of pop-up window processing in the present invention. Figure 5 As shown, the pop-up window processing specifically includes:

[0214] (1) Get a screenshot of the page on the Android phone.

[0215] Since the screenshot of the page of the Android phone has been obtained in step three, this step can be omitted when starting the pop-up window processing, and only executed in the subsequent loop process.

[0216] Of course, in this step, you can still use the method in step three to obtain a screenshot of the page on the Android phone.

[0217] (2) Detect whether there is a pop-up window in the page screenshot through the pop-up window detection model. If not, take the page screenshot as a page screenshot of a page without a pop-up window; if so, determine whether it has been cycled once (that is, whether it has been handed over to the pop-up window closing model for processing once). If it has been cycled once, hand it over to the user for operation. If it has not been cycled once, hand it over to the pop-up window closing model and the pop-up window closing model attempts to close the pop-up window. If the pop-up window closing model can return to the closed position, the pop-up window is automatically closed according to the closed position and returns to step (1). If the pop-up window closing model cannot return to the closed position, hand it over to the user for operation.

[0218] The pop-up window detection model and the pop-up window closing model are also trained end-to-end prediction models based on convolutional neural networks. The pop-up window detection model is used to determine whether there is a pop-up window in the page screenshot, and the pop-up window closing model is used to determine whether there is a click position in the page screenshot that can close the pop-up window, and if so, predict its position.

[0219] In specific use, CNN (convolutional neural network) can be used as the pop-up window detection model and pop-up window closing model, and training data can be collected and manually labeled. The constructed pop-up window detection model and pop-up window closing model can be trained with the labeled training data to obtain the pop-up window detection model and pop-up window closing model.

[0220] The training data of the pop-up window detection model are manually annotated pages with pop-up windows and their labels are "pop-up windows exist". The training data of the pop-up window closing model are manually annotated pages with pop-up windows and their labels are "pixel coordinates of the close button in the pop-up window".

[0221] In addition, in the present invention, when the user operates, the user can choose to manually close the pop-up window existing in the page, or manually confirm that there is no pop-up window on the current page, thereby ending the process. In the case of manually confirming that there is no pop-up window on the current page, the obtained page screenshot can be directly used as the page screenshot of the page without the pop-up window; in the case of choosing to manually close the pop-up window existing in the page, return to step (1) again to obtain the page screenshot again, and use the page screenshot obtained again as the page screenshot of the page without the pop-up window.

[0222] When performing judgment and processing of non-standard pages, the present invention takes into account the detection of various non-standard page states such as "page loading", "page jumping", "pop-up window", etc., and can provide corresponding processing methods for different non-standard page states. Therefore, the present invention is more comprehensive, flexible, and more adaptable to the needs of the Agent model.

[0223] 5. Compress, simplify and complete the obtained page XML to obtain the processed page XML.

[0224] like Figure 6 As shown, in the present invention, the compression and simplification of the acquired page XML and the page completion processing include the following steps:

[0225] 1. Determine whether it is necessary to retain the off-screen nodes in the page XML. If not, proceed to step 2; if yes, proceed to step 3.

[0226] Since page XML is used to define the layout and elements of the page of an Android phone, it contains all the components in a page, so the page XML contains a large number of nodes that exist only for structure and layout. These nodes do not contain actual useful page information, which is also the main reason why the page XML is too long. In addition, a page often contains more nodes than what is displayed on the screen, such as some swiping pages, so the page XML also contains many nodes outside the screen.

[0227] Whether to retain the nodes outside the screen is controlled by an entry parameter remain_nodes (retain nodes when remain_nodes = True), which can be selected by yourself. In the present invention, if the need is to summarize the information of the entire page, in order to save the number of operation steps (such as sliding the screen to see the complete text), the nodes outside the screen can be retained, so that the text information of the entire page can be directly obtained without additional operations. If the need is related to operation simulation, such as the need to simulate clicks or slides, you can choose to delete the nodes outside the screen to prevent interference with the Agent model.

[0228] 2. Delete the off-screen nodes in the page XML.

[0229] In the page XML, the bounds attribute of all nodes on the screen must be within the range of [0,0][Window_Height,Window_Width] and must be included by its parent node. Therefore, you only need to determine whether the bounds of the current node is included by its parent node to determine all nodes within the screen range. If you need to delete nodes outside the screen, after determining all nodes within the screen range, delete other nodes that are not within the screen range.

[0230] 3. Delete redundant nodes in the page XML.

[0231] The page XML also contains a large number of nodes that exist only for structural and layout needs. These nodes do not contain useful page information, so the present invention will delete these redundant nodes. The present invention will determine whether a node is redundant based on the attribute information of the node. If at least one of the attributes of a node such as "checkable", "checked", "clickable", "focusable", "scrollable", "long-clickable", "password" and "selected" is True, or the text and content-desc attributes are not empty, then the node is considered to be a functional node, and nodes that do not meet this requirement are redundant nodes. The present invention will delete all nodes that do not meet this requirement.

[0232] 4. Determine whether the page of the Android phone is a special page. If yes, go to step 5; if no, go to step 6.

[0233] Due to different ideas of APP developers during development or different models, some pages will get some page XMLs with different structures or even information, which will easily confuse the Agent model. At the same time, for components such as pop-up windows and drop-down boxes, it will cause overlapping between different components. However, in the page XML, these components are displayed in a tiled manner, which means that it is impossible to distinguish which component is on the top and which one is on the bottom only through the page XML (in actual operation, the components overlapping below will not be triggered by any operation), which is also easy to cause confusion in the Agent model. Therefore, the present invention designs a separate processing logic for special pages. The present invention will determine whether it is a special page based on the fixed text content of each page, and then execute the processing logic of the special page.

[0234] In the present invention, whether it is a special page will be determined based on the page content contained in the page XML. Specifically, the corresponding text information can be preset for all special pages in advance. For example, for pages containing drop-down boxes in various APPs, the preset text information can be selected according to the specific information contained in the drop-down box in the page. In this way, when the text or content-desc attributes of some nodes in the page XML contain text information preset for special pages, the current page is considered to be a special page.

[0235] 5. Delete or supplement information on special pages;

[0236] Special pages can be divided into two categories: special pages for deleting information and special pages for supplementing information.

[0237] The special page for deleting information is mainly aimed at the situation that multiple components overlap in some pages, where the components on the top can be operated, but the components on the bottom cannot be operated (covered by the components on the top). At this time, the nodes of the components on the bottom still exist in the page XML, which will confuse the Agent model. For example, when the pop-up window and various pages containing drop-down boxes appear when selecting specifications for ordering takeaway APP, the components originally in this position will be covered, resulting in overlapping phenomena. When the present invention processes these special pages for deleting information, it will first select a text node that is fixed and unchanged on the current page as a marker node (information that is not changed by time or network changes), and then find the top-level position of the node tree to be deleted according to the position of this node and delete it (XML is displayed in a tree structure, so it is only necessary to delete the root node of the subtree where all nodes to be deleted are located).

[0238] The special page of supplementary information is mainly aimed at some pages, which may not exist in the page XML for some information (some attributes of a node are missing) or some operable components due to the implementation logic. For example, the button for liking a circle of friends in WeChat Moments does not have a specific node corresponding in the page XML, and a node needs to be added through special processing. Similar to the processing method in the special page of deleted information, for the special page of supplementary information, the present invention will also select a marker node, find the node that needs supplementary information according to its position, and supplement the information (supplement the missing attributes or add a new node under this node).

[0239] 6. Use an optical character recognition method to identify all texts and text locations in the page of the Android phone, compare the text recognized by the optical character recognition method with the text in the page XML, remove the text in the text recognized by the optical character recognition method that is repeated with the text in the page XML, and insert the remaining text into the page XML.

[0240] In many pages, the page XML does not contain all the text information in the page, but in many cases, this text information is critical for the Agent model to understand the page information. The present invention will introduce OCR (Optical Character Recognition) technology to supplement the missing text information in the page XML. Specifically, OCR will first be used to identify all the text and the location (bounds) of the text in the page of the Android phone. Then compare the text recognized by OCR with the text (text and content-desc) contained in the page XML, remove the duplicate text and insert the remaining text into the page XML.

[0241] Specifically, an algorithm based on MinHash can be used to compare text similarity and attempt to determine whether the given OCR text contains the same or most of the information as the text in the existing page XML (sometimes limited by the accuracy of OCR, the text recognized by OCR may not be exactly the same as that in the page XML). Technologies such as MinHash and Levenshtein edit distance are used to process text matching, and the coverage rate of the matching is determined under a certain threshold. First, a MinHashLSH object is created for Locality-Sensitive Hashing (LSH) queries, and MinHash is generated for all page XML descriptions and inserted into the LSH. At the same time, punctuation marks and spaces are removed and saved in a dictionary. Then, MinHash is generated for each OCR text and an LSH query is performed. Those that reach a specific threshold are considered to have a successful match (parameters of the MinHash algorithm). If the match fails, further processing is carried out according to the text length: for texts with a length of 2 to 4 characters, the Levenshtein edit distance is used for comparison, and a distance less than or equal to 1 is considered a match; for single-character texts, direct character matching is performed. Finally, all non-repeating texts are obtained. For these non-repeating texts, according to their corresponding position information (bounds), they are inserted into the page XML as a node. According to the fact that the bounds of the parent node must contain the child node as described above, the present invention will select the one with the smallest bounds among all those that contain the OCR recognized text bounds as its parent node for insertion.

[0242] The following is an example. Suppose all page XML descriptions correspond to a text list ["How!","document."], and the text list recognized by OCR is ["HoW!","丶document."]. First, all punctuation and special symbols are removed to obtain the text list ["How","document"] for page XML descriptions and the text list ["HoW","document"] for OCR recognized texts. Then, MinHash values are generated for each element in the text list of page XML descriptions and inserted into the LSH. After that, for each element in the text list recognized by OCR, MinHash values are also generated and matched with the elements in the LSH (MinHash corresponding to the XML list). Finally, document can be successfully matched, which means that "document." and "丶document." correspond to the same content, so this text does not need to be supplemented into the page XML. However, HoW and How do not match. For the next step of judgment, since the Levenshtein edit distance between How and HoW is 1, it means they match, and it is also considered that this OCR recognized text and the text in the page XML are repeated.

[0243] 7. Simplify the attributes in the page XML after insertion.

[0244] The descriptions of each attribute in the page XML are too redundant and will consume a very large number of Tokens. Finally, the present invention will simplify the descriptions of these attributes. For functional attributes such as "checkable", "checked", "clickable", "focusable", "scrollable", "long-clickable", "password", and "selected", since in most cases, most of them are False, only the attributes with the value of True will be shown. The three attributes "index", "resource-id", and "package" are not helpful for the Agent model to understand the page and will be directly deleted. The "class" attribute also represents the main function of the node to a certain extent, so the last part will be retained (the "class" must be composed of the format of x.x.x.x, the number of dots is variable, and the present invention only retains the content after the last dot. For example, in android.widget.FrameLayout, only FrameLayout is retained). The two attributes "text" and "content-desc" represent the text information of the node, and the two will be combined and shown separately. The "bounds" attribute represents the position of the node in the page and is one of the most critical attributes, and it will also be shown separately.

[0245] Finally, for the following node:

[0246] <node index="0"text="汗蒸"resource-id=""class="android.view.View"

[0247] package="com.autonavi.minimap"content-desc=""checkable="false"checked="false"

[0248] clickable="false"enabled="true"focusable="false"focused="false"scrollable="false"

[0249] long-clickable="false"password="false"selected="false"bounds="[290,844][346,885]" / >

[0250] It will be simplified to:

[0251] [n42]View;;;Sweat steaming;[290,844][346,885]

[0252] The present invention deletes redundant nodes and nodes that are not on the screen through node deletion, simplifies the attribute description of the nodes, rewrites the page XML into a new format to obtain a more streamlined page XML, and deletes or supplements information for a specific page through special page processing and OCR technology, thereby improving the processing capability of the Agent model.

[0253] 6. Input the processed page XML and page screenshots into the Agent model, and the Agent model outputs the type and location of the next operation performed by the Android phone.

[0254] In the present invention, the Agent model can be obtained by fine-tuning the GLM-4 model. Before use, some data with processed page XML and page screenshots as input and operation type and operation location as output can be collected as a training data set, and the GLM-4 model can be fine-tuned using the training data set to obtain the Agent model. This part of the content is a technology that can be understood and mastered by technicians in the field of large language models. For the sake of simplicity, it will not be described in detail here.

[0255] In the present invention, after fine-tuning training, the input of the Agent model is mainly the processed page XML and page screenshots. The following Table 1 is the operation type that the Agent can output and its corresponding output content, and element is the bbox corresponding to the node Node in the page XML. In addition to some conventional operation types such as click, long press, slide, and rollback, the present invention can also handle some special types of operations, such as call api, which is for some tasks that require text content understanding and generation (for example: help me write an abstract for this WeChat public account article / help me send a 300-word apology letter to xx). The Agent model will combine the task input by the user, generate instructions, and combine the page XML to call the GLM-4-public api to complete the task and generate information message.

[0256] In addition, for some special operation types, it is necessary to combine page screenshots for processing. Among them, if the action predicted by the Agent model is the Note and interact operation type, it is necessary to combine the fine-tuned GLM-4v visual model for further reasoning and prediction.

[0257] The Note operation exists in the form of annotations. For some tasks that require the use of information browsed in the middle (for example: which one is closer to me, the Summer Palace or the Forbidden City? - This task requires remembering the distance between the Summer Palace and the current location, and then comparing it with the current distance of the Forbidden City to draw a conclusion), the Agent model will predict the steps that need to be remembered, add "#Note:True" before the output, and extract all the steps that the Agent model predicts need to be noted in this trace and the screenshot of the current page before finishing, and splice them into a large picture, call the visual model to generate the final information finish message based on the information therein, and feed it back to the client.

[0258] The interact function mainly generates interactive information based on page screenshots. During the task, multiple results that meet the user's needs may be found (for example: buy me a high-speed rail ticket to Shanghai tomorrow morning - which train, second-class seat or first-class seat, etc., all require the user to make further choices). For such cases, the Agent model will predict the steps that need to be interacted with, call the fine-tuned GLM-4v model to generate interactive information (for example: multiple results that meet the conditions such as xx and xx have been retrieved for you, which one do you want to choose?), and continue to execute the task after the user supplements.

[0259] Table 1 Types of operations processed by the Agent model

[0260]

[0261]

[0262] 7. According to the output of the Agent model, operations are performed on the Android phone based on the Accessibility Service.

[0263] The output of the Agent model will be sent to the client of the Android phone, and the client will simulate the operations such as clicking, sliding, long pressing, and input based on the Android accessibility service AccessibilityService.

[0264] 8. Display the results of operations executed on Android phones.

[0265] The client can display the message of operations such as call API and interact.

[0266] Figure 7 The schematic diagram of the structure of the Android mobile phone intelligent control system based on the large model intelligent body of the present invention is shown. Figure 7As shown, the Android mobile phone intelligent control system based on the large model intelligent body of the present invention includes:

[0267] 1. Task input module

[0268] The task input module is used to obtain the task input by the user.

[0269] 2. Task discrimination model.

[0270] The task discrimination model is used to judge whether the task input by the user is within the processing range of the Agent model, so as to determine whether to continue to execute the task input by the user.

[0271] 3. Data acquisition module.

[0272] The data acquisition module is used to obtain page information of an Android phone, and the page information includes page XML and page screenshots.

[0273] 4. Page status judgment processing module.

[0274] The page status judgment processing module is used to perform a judgment process on the non-standard status of the page based on the page information to ensure that the page information obtained is that of a standard page.

[0275] 5. Data processing module.

[0276] The data processing module is used to compress and simplify the acquired page XML and perform page completion processing to obtain the processed page XML.

[0277] 6.Agent model.

[0278] The Agent model is used to output the type and location of the next operation to be performed by the Android phone based on the processed page XML and page screenshots.

[0279] 7. Task execution module.

[0280] The task execution module is used to perform operations on the Android phone based on the accessibility service AccessibilityService according to the output of the Agent model.

[0281] Of course, the Android mobile phone intelligent control system based on the large model agent of the present invention may also include:

[0282] 8. Display module.

[0283] The display module is used to display the results of the operation executed on the Android mobile phone.

[0284] In addition, the present invention also provides an Android mobile phone intelligent control device based on a large model intelligent agent, which includes: one or more processors; a memory for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the Android mobile phone intelligent control method based on the large model intelligent agent as described above.

[0285] Finally, the present invention provides a computer-readable storage medium having a computer program stored thereon, characterized in that when the program is executed by a processor, the steps of the method for intelligent control of an Android mobile phone based on a large model intelligent agent as described above are implemented.

[0286] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention, rather than to limit the scope of protection of the present invention. Those skilled in the art can modify or replace the technical solution of the present invention according to the idea of ​​the present invention without departing from the essence and scope of the technical solution of the present invention.

Claims

1. An Android mobile phone intelligent control method based on a large model agent, characterized in that: The following steps are involved: 1) The task of obtaining user input; 2) Determine whether the task input by the user is within the processing range of the Agent model. If yes, proceed to step 3). Otherwise, terminate the execution of the task input by the user. 3) Obtaining page information of an Android phone, wherein the page information includes page XML and page screenshots; 4) Based on the page information, a judgment process is performed on the non-standard state of the page to ensure that the page information obtained is that of a standard page; 5) compressing and simplifying the acquired page XML and performing page completion processing to obtain the processed page XML; 6) Input the processed page XML and page screenshots into the Agent model, and the Agent model outputs the type and location of the next operation performed by the Android phone; 7) According to the output of the Agent model, operations are performed on the Android phone based on the Accessibility Service.

2. The method for intelligently controlling an Android mobile phone based on a large model agent according to claim 1 is characterized in that: The step 3) specifically includes: 31) Based on the accessibility service AccessibilityService, obtain the accessibility service node information AccessibilityServiceNodeInfo and the page screenshot Screenshot of the Android phone page; 32) Parse the node Node in the accessibility service node information AccessibilityServiceNodeInfo, and convert the parsing result into a variable character sequence StringBuilder; 33) Adjust the structure of the variable character sequence StringBuilder, and perform parallel parsing on the child nodes ChildNod of the corresponding node Node; 34) Obtain the parsing results of each child node ChildNod and assemble them into a variable character sequence StringBuilder, and adjust the structure of the assembled variable character sequence StringBuilder again; 35) Obtain the final page XML based on the variable character sequence StringBuilder that has been adjusted again; 36) Performing quality compression on the page screenshot Screenshot without changing the original resolution of the page screenshot Screenshot to obtain a final page screenshot.

3. The method for intelligently controlling an Android mobile phone based on a large model agent according to claim 2 is characterized in that: The step 4) specifically includes: 41) Page waiting processing, which is used to process a page that needs to wait before it can become a standard page, so as to ensure that the page information of a page that does not need to wait before it can become a standard page is obtained; 42) Pop-up window processing, which is used to handle the situation where a pop-up window exists in a page to ensure that the page information of a page without a pop-up window is obtained.

4. The method for intelligently controlling an Android mobile phone based on a large model agent according to claim 3 is characterized in that: The step 41) specifically includes: 411) Get the page XML of the Android phone again; 412) Determine whether the difference between the total number of nodes in the page XMLs obtained twice is less than D, and determine whether the position attributes of the first k nodes in the page XMLs obtained twice are completely consistent. If the difference is less than D and the position attributes are completely consistent, the page XML obtained again is used as the page XML of the page that does not need to wait before becoming a standard page; if the difference is not less than D and / or the position attributes are not completely consistent, determine whether the total time consumed is greater than or equal to T seconds. If it is greater than or equal to T seconds, the page XML obtained again is used as the page XML of the page that does not need to wait before becoming a standard page. If it is less than T seconds, return to step 411); 413) Get the page screenshot of the Android phone again; 414) Input the page screenshot obtained again into the single-image Wait model and input the page screenshots obtained twice adjacently into the dual-image Wait model to respectively determine whether the loading is completed. If the output results of the single-image Wait model and the dual-image Wait model are both loading completed, the page screenshot obtained again is used as a page screenshot of the page that does not need to be in a waiting state before it can be changed into a standard page; if the output result of the single-image Wait model and / or the dual-image Wait model is that the loading is not completed, determine whether the number of cycles is greater than or equal to n times. If it is greater than or equal to n times, the page screenshot obtained again is used as a page screenshot of the page that does not need to be in a waiting state before it can be changed into a standard page. If it is less than n times, return to step 413).

5. The method for intelligently controlling an Android mobile phone based on a large model agent according to claim 3 is characterized in that: The step 42) specifically includes: 421) Get page screenshots on Android phones; 422) Detect whether there is a pop-up window in the page screenshot through the pop-up window detection model. If not, take the page screenshot as a screenshot of a page without a pop-up window; if so, determine whether it has been cycled once. If it has been cycled once, leave it to the user to operate. If it has not been cycled once, leave it to the pop-up window closing model and the pop-up window closing model will try to close the pop-up window. If the pop-up window closing model can return to the closed position, the pop-up window will be automatically closed according to the closed position and return to step 421). If the pop-up window closing model cannot return to the closed position, leave it to the user to operate.

6. The method for intelligently controlling an Android mobile phone based on a large model agent according to any one of claims 1 to 5, characterized in that: The step 5) specifically includes: 51) Determine whether it is necessary to retain the off-screen nodes in the page XML, if not, proceed to step 52), if yes, proceed to step 53); 52) Delete off-screen nodes in the page XML; 53) Delete redundant nodes in the page XML; 54) Determine whether the page of the Android phone is a special page, if yes, proceed to step 55), if no, proceed to step 56); 55) Delete or add information to special pages; 56) using an optical character recognition method to identify all texts and locations of the texts in the page of the Android phone, comparing the texts recognized by the optical character recognition method with the texts in the page XML, removing the texts in the texts recognized by the optical character recognition method that are repeated with the texts in the page XML, and inserting the remaining texts into the page XML; 57) Simplify the attributes in the page XML after insertion.

7. The method for intelligently controlling an Android mobile phone based on a large model agent according to claim 6 is characterized in that: Further including: 8) Display the results of the operations executed on the Android phone.

8. An Android mobile phone intelligent control system based on a large model agent, characterized in that: include: A task input module, which is used to obtain tasks input by users; The task discrimination model is used to determine whether the task input by the user is within the processing range of the Agent model, so as to determine whether to continue to execute the task input by the user; A data acquisition module, which is used to obtain page information of an Android phone, wherein the page information includes page XML and page screenshots; A page status judgment processing module, which is used to judge the non-standard status of the page based on the page information to ensure that the page information obtained is that of a standard page; A data processing module is used to compress and simplify the acquired page XML and complete the page to obtain a processed page XML; Agent model, which is used to output the type and location of the next operation to be performed by the Android phone based on the processed page XML and page screenshots; The task execution module is used to perform operations on the Android phone based on the accessibility service AccessibilityService according to the output of the Agent model.

9. An Android mobile phone intelligent control device based on a large model intelligent agent, characterized in that: include: one or more processors; A memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the Android mobile phone intelligent control method based on the large model intelligent agent as described in any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method for intelligently controlling an Android mobile phone based on a large model intelligent agent are implemented as described in any one of claims 1-7.

Images