A mobile terminal application control method, device and system

Abstract

The embodiment of the application discloses a mobile terminal application control method, device and system, comprising: when it is determined that a current page where a target mobile terminal application is located is inconsistent with a target page, obtaining a current task script corresponding to an application operation task from a script library maintained by a community side intelligent agent, and running the current task script; if the current task script fails to execute, collecting an actual page screenshot of the target mobile terminal application, and generating an updated task script according to the actual page screenshot and the application operation task through a pre-constructed visual language model; alternately testing the current task script and the updated task script, and when the test times of the current task script and the updated task script both reach a preset test time, and the test success rate of the updated task script is greater than the test success rate of the current task script, running the updated task script to complete the application operation task, thereby improving the automation control efficiency and operation precision of the mobile terminal application.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence, and in particular to a mobile application control method, apparatus and system. Background Technology

[0002] With the widespread adoption of smartphones, tablets, and industrial handheld terminals, the number of mobile applications in scenarios such as office, marketing, testing, operation and maintenance, and the Internet of Things continues to expand. The work mode of relying on manual repetitive operations on mobile applications is inefficient, and automated control technology for mobile applications has become an urgent need for various industries.

[0003] In existing technologies, script generation and iterative updates typically rely on manual documentation, manual recording, or centralized training samples, resulting in low script update efficiency and poor usability of the updated scripts. Summary of the Invention

[0004] This invention provides a mobile application control method, device, and system that solves the problems of low script update efficiency and poor usability of updated scripts caused by relying on manual documents, manual recording, or centralized training samples for script generation and iterative updates. It improves script update efficiency, ensures the usability of updated scripts, and thus improves the automation control efficiency and operation accuracy of mobile applications.

[0005] In a first aspect, embodiments of the present invention provide a mobile application control method, applied to a mobile application control system, the mobile application control system comprising: a user-side intelligent agent and a community-side intelligent agent; This method is executed by the user-side agent and includes: Based on the application operation tasks sent to the target mobile application, determine the target page that the target mobile application needs to reach; When it is determined that the current page of the target mobile application is inconsistent with the target page, the current task script corresponding to the application operation task is obtained from the script library maintained by the community-side intelligent agent, and the current task script is run. If the current task script fails to execute, then the actual page screenshot of the target mobile application is collected, and an updated task script is generated based on the actual page screenshot and application operation task using a pre-built visual language model. The current task script and the update task script are tested alternately. When both the current task script and the update task script have reached the preset number of tests, and the success rate of the update task script is greater than that of the current task script, the update task script is run to complete the application operation task.

[0006] Secondly, embodiments of the present invention also provide a mobile application control device, applied to a mobile application control system, the mobile application control system including: a user-side intelligent agent and a community-side intelligent agent; The device is executed by a user-side intelligent agent and includes: The target page determination module is used to determine the target page that the target mobile application needs to reach based on the application operation tasks sent to the target mobile application. The script execution module is used to retrieve the current task script corresponding to the application operation task from the script library maintained by the community-side intelligent agent when it is determined that the current page of the target mobile application is inconsistent with the target page, and to run the current task script. The script update module is used to collect actual page screenshots of the target mobile application if the current task script fails to execute, and generate an update task script based on the actual page screenshots and application operation tasks using a pre-built visual language model. The script verification module is used to alternately test the current task script and the update task script. When both the current task script and the update task script have reached the preset number of tests, and the success rate of the update task script is greater than that of the current task script, the update task script is run to complete the application operation task.

[0007] Thirdly, embodiments of the present invention also provide a mobile application control system, which includes: a user-side intelligent agent and a community-side intelligent agent; The community-side intelligent agent is used to maintain a script library containing multiple task scripts; The user-side intelligent agent is used to determine the target page that the target mobile application needs to reach based on the application operation tasks sent to the target mobile application. When it is determined that the current page of the target mobile application is inconsistent with the target page, the current task script corresponding to the application operation task is obtained from the script library maintained by the community-side intelligent agent, and the current task script is run. If the current task script fails to execute, then the actual page screenshot of the target mobile application is collected, and an updated task script is generated based on the actual page screenshot and application operation task using a pre-built visual language model. The current task script and the update task script are tested alternately. When both the current task script and the update task script have reached the preset number of tests, and the success rate of the update task script is greater than that of the current task script, the update task script is run to complete the application operation task.

[0008] The technical solution of this invention, when it is determined that the current page of the target mobile application is inconsistent with the target page, obtains the current task script corresponding to the application operation task from the script library maintained by the community-side intelligent agent; if the current task script fails to execute, it collects a screenshot of the actual page of the target mobile application, and generates an updated task script based on the actual page screenshot and the application operation task through a pre-built visual language model; it alternately tests the current task script and the updated task script, and runs the updated task script to complete the application operation task when both the current task script and the updated task script have reached a preset number of tests, and the test success rate of the updated task script is greater than that of the current task script. This solves the problem of low script update efficiency and poor usability of the updated script caused by relying on manual documents, manual recording, or centralized training samples for script generation and iterative updates, thereby improving the efficiency of script update, ensuring the usability of the updated script, and thus improving the automation control efficiency and operation accuracy of mobile applications.

[0009] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0010] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1 This is a schematic diagram of the structure of a mobile application control system according to Embodiment 1 of the present invention; Figure 2 This is a flowchart of a mobile application control method provided in Embodiment 1 of the present invention; Figure 3 This is a flowchart of another mobile application control method provided according to Embodiment 2 of the present invention; Figure 4 This is a schematic diagram of the structure of a mobile application control device according to Embodiment 3 of the present invention. Detailed Implementation

[0012] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0013] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0014] Example 1 Figure 1 This is a schematic diagram of a mobile application control system according to Embodiment 1 of the present invention. This embodiment can be applied to the situation of automating the control of mobile applications.

[0015] like Figure 1 As shown in the figure, the mobile application control system disclosed in this embodiment includes: multiple user-side intelligent agents and one community-side intelligent agent.

[0016] Specifically, user-side intelligent agents can be understood as intelligent agent program units deployed on mobile terminals such as smartphones or tablets, possessing independent capabilities for perception, understanding, decision-making, execution, and reflection. User-side intelligent agents can be used to connect locally to mobile terminals to automate the control processes of mobile applications. Community-side intelligent agents can be used to achieve unified operation and maintenance management of the script library; they do not directly issue click, swipe, or input control commands to the mobile terminal. The aforementioned script library stores task scripts corresponding to various application operation tasks.

[0017] Figure 2 This is a flowchart of a mobile application control method provided according to Embodiment 1 of the present invention. Figure 2 As shown, this embodiment discloses a mobile application control method, which is executed by a user-side intelligent agent and specifically includes the following steps: S110. Based on the application operation task sent to the target mobile application, determine the target page that the target mobile application needs to reach.

[0018] In this embodiment, the target mobile application can be understood as a mobile application that can be automatically controlled by a user-side intelligent agent, such as a content platform, social platform, e-commerce platform, office application, or other mobile applications with complex user interface (UI) states. The application operation task may include several sequentially executed page operation steps and the target page to be reached. Each page operation step is recorded with a step name and a corresponding operation description. A page operation step may consist of multiple adaptive execution steps, each representing a state transition from the current page to the corresponding page.

[0019] In this step, specifically, the application operation tasks can be parsed to obtain the page operation steps that need to be controlled to be executed sequentially by the target mobile application, as well as the target page that the target mobile application needs to reach.

[0020] S120. When it is determined that the current page of the target mobile application is inconsistent with the target page, the current task script corresponding to the application operation task is obtained from the script library maintained by the community-side intelligent agent, and the current task script is run.

[0021] In this embodiment, the current task script can be understood as an instruction carrier that records the complete execution logic of the application operation task. The user-side intelligent agent can parse the current task script and execute the operation steps of each page sequentially. The current task script includes at least an execution function, and preferably also includes an independent verification function and metadata. The metadata includes at least the number of successes, the number of failures, the total number of successes, the number of consecutive successes, the average time taken, the most recent time taken, the number of times the visual language model was used as a fallback, the failure breakdown, the utility value, the confidence level, the target page, and the verification type.

[0022] In this step, specifically, the current page of the target mobile application can be determined based on a predefined page state graph corresponding to the target mobile application. The predefined page state graph can include information such as the target mobile application's page state, page metrics, transition edges, discovery paths, optional states, observation count, matching failure count, and image hash values. It can be used for scenarios such as page detection, navigation path finding, and script generation / hint injection.

[0023] Then, when the current image hash value of the current page or the number of text recognition metrics matching the current page and the target page can be determined, it can be determined whether the current page of the target mobile application is consistent with the target page based on the current image hash value and / or the number of text recognition metrics. When the current image hash value of the current page and the number of text recognition metrics matching the current page and the target page cannot be determined, it can be determined whether the current page of the target mobile application is consistent with the target page through a pre-built visual model. The current image hash value can be of type mean hash, perceptual hash, or difference hash, etc.

[0024] Finally, if it's determined that the current page of the target mobile application is different from the target page, the system checks if a current task script corresponding to the application's operation task exists in the script library. If a current task script exists, it is used as the current task script and executed. If no current task script exists, the system checks if a candidate task script exists. If a candidate task script exists, it is used as the current task script and executed. If no candidate task script exists, a seed template corresponding to the application's operation task is retrieved from the script library and executed as the current task script.

[0025] Taking the data viewing operation of a short video application as an example, if the application's operation task is "view account data center" and the current page is already in the data center, it will succeed directly; if the current page is the homepage, it will execute the multi-step operation of "enter personal page - enter creator center - enter data center" according to the current task script.

[0026] S130. If the current task script fails to execute, collect actual page screenshots of the target mobile application and generate an updated task script based on the actual page screenshots and application operation tasks using a pre-built visual language model.

[0027] In this embodiment, the visual language model can be understood as a deep learning model trained using historical operation tasks and historical page screenshots as samples.

[0028] In this step, specifically, a screenshot of the actual page of the target mobile application can be taken to obtain a screenshot of the target mobile application's actual page. Then, the actual page screenshot and the application operation task can be input into the visual language model to obtain the update task script.

[0029] The advantage of this setup is that by calling the visual language model to generate an updated task script only when the current task script fails, it avoids over-reliance on the visual language model, reduces script generation costs, and shortens task processing latency.

[0030] S140. Alternately test the current task script and the update task script. When the number of tests for both the current task script and the update task script reaches the preset number of tests, and the success rate of the update task script is greater than that of the current task script, run the update task script to complete the application operation task.

[0031] In this embodiment, the preset number of tests can be understood as a threshold number of tests determined based on user needs and historical experience. For example, the preset number of tests can be set to 3.

[0032] The technical solution of this embodiment determines the target page that the target mobile application needs to reach by issuing application operation tasks to the target mobile application. When it is determined that the current page of the target mobile application is inconsistent with the target page, the current task script corresponding to the application operation task is obtained from the script library maintained by the community-side intelligent agent. If the current task script fails to execute, a screenshot of the actual page of the target mobile application is collected, and an updated task script is generated based on the actual page screenshot and the application operation task using a pre-built visual language model. The current task script and the updated task script are tested alternately, and the updated task script is run when both the current task script and the updated task script have reached the preset number of tests, and the test success rate of the updated task script is greater than that of the current task script. This solves the problem of low script update efficiency and poor usability of updated scripts caused by relying on manual documents, manual recording, or centralized training samples for script generation and iterative updates. It improves script update efficiency, ensures the usability of updated scripts, and thus improves the automation control efficiency and operation accuracy of mobile applications.

[0033] Example 2 Figure 3 This is a flowchart of another mobile application control method provided by Embodiment 2 of the present invention. This embodiment is a further optimization and extension based on the above embodiments, and can be combined with various optional technical solutions in the above embodiments. Figure 3 As shown, the method includes: S210. Based on the application operation task sent to the target mobile application, determine the target page that the target mobile application needs to reach.

[0034] S220. Determine the current page of the target mobile application and the current image hash value of the current page, and obtain multiple reference image hash values ​​corresponding to the target page.

[0035] In this step, specifically, for the current page, the current image hash value can be determined based on all or part of the pixel values ​​of the current page. For the target page, multiple screenshots corresponding to the target page can be taken while the target page is in normal execution flow, and the reference image hash value corresponding to each screenshot can be determined based on the target pixel values ​​corresponding to each screenshot. The target pixel value determination process is as follows: crop the top status bar area of ​​the page screenshot to obtain the cropped target page; scale the cropped target page to a preset size to obtain the scaled target page; convert the scaled target page into a target grayscale image, and use all pixel values ​​in the target grayscale image as the target pixel value.

[0036] Optionally, determining the current image hash value of the current page includes: cropping the top status bar area of ​​the current page to obtain the cropped page, and scaling the cropped page to a preset size to obtain the scaled page; and generating the current image hash value corresponding to the current page based on all pixel values ​​of the scaled page.

[0037] Specifically, the scaled page can be converted into a scaled grayscale image, and the average pixel value of the scaled grayscale image can be determined based on all pixel values. Then, pixel values ​​in the scaled grayscale image greater than the average pixel value can be set to 1, and pixel values ​​less than or equal to the average pixel value can be set to 0, resulting in a pixel binarization sequence corresponding to the scaled grayscale image. Finally, based on the pixel binarization sequence corresponding to the scaled grayscale image, the current image hash value corresponding to the current page can be generated.

[0038] For example, the cropped page can be scaled to an 8x8 scaled page, and the 8x8 scaled page can be converted into a scaled grayscale image. Then, the pixel mean of the scaled grayscale image can be calculated, and a 64-bit current image hash value can be generated using the pixel mean as a binarization threshold.

[0039] S230. Calculate the Hamming distance between the current image hash value and the hash values ​​of each reference image respectively. When each Hamming distance is less than the preset distance threshold, count the number of text recognition indicators that match between the current page and the target page.

[0040] In this embodiment, the preset distance threshold can be understood as a distance threshold determined based on user needs and historical experience; for example, the preset distance threshold can be set to 12. The text recognition metric can consist of text content, matching type, and text source, or it can be composed of text content, control resource identifiers, layout structure, icon recognition results, and multimodal embedding.

[0041] In this step, specifically, if any Hamming distance is greater than or equal to a preset distance threshold, it can be determined that the current page and the target page are visually dissimilar, and thus, the current page and the target page can be directly determined to be inconsistent. When all Hamming distances are less than the preset distance threshold, it can be determined that the current page and the target page are visually similar, and then the number of matching text recognition metrics between the current page and the target page can be counted.

[0042] For example, the Hamming distance between the current image hash value and the hash values ​​of up to 5 reference images can be calculated. When any Hamming distance is less than 12, it is determined that the current page and the target page are visually similar, and the number of matching text recognition metrics between the current page and the target page is counted.

[0043] S240. If the number of text recognition indicators is less than the preset indicator threshold, it is determined that the current page is inconsistent with the target page, and the current task script corresponding to the application operation task is obtained from the script library maintained by the community-side intelligent agent.

[0044] In this embodiment, the preset index threshold can be understood as an index threshold determined based on user needs and historical experience. For example, the preset index threshold can be set to 1, 2 or 3.

[0045] Specifically, in this step, if the number of text recognition metrics is greater than or equal to a preset threshold, it can be determined that the text on the current page and the target page are similar, and thus the current page and the target page are consistent. If the number of text recognition metrics is less than the preset threshold, it can be determined that the text on the current page and the target page are not similar, and thus the current page and the target page are inconsistent.

[0046] S250. If the current task script fails to execute, collect actual page screenshots of the target mobile application and generate an updated task script based on the actual page screenshots and application operation tasks using a pre-built visual language model.

[0047] Specifically, in this step, if the current task script fails to execute, a screenshot of the actual page of the target mobile application can be collected, and it can be determined whether the actual page screenshot belongs to an unknown page or a dynamic page. If so, the current task script is determined to be normal. At this time, the application operation task can be executed based on the actual page screenshot using a visual language model. When the application operation task is executed successfully, the task execution trajectory is output. Then, based on the actual page screenshot, the task execution trajectory, and the operation instructions, an updated task script is generated. If not, the current task script is determined to be abnormal. At this time, the application operation task can be executed based on the actual page screenshot using a visual language model. When the application operation task is executed successfully, the task execution trajectory and the abnormal diagnosis result are output. Then, based on the actual page screenshot, the task execution trajectory, the operation instructions, and the abnormal diagnosis result, an updated task script is generated. After generating the updated task script, the updated task script can be stored as a candidate task script in the local cache of the user-side agent, so that the user-side agent can determine whether to use the updated task script to replace the current task script.

[0048] For example, if a step fails during the initial execution of the current task script due to a change in the application entry point, the visual language model finds a new entry point based on the actual page screenshot and completes the application operation task. When the application operation task is successfully executed, an updated task script is generated based on the actual page screenshot, the task execution trajectory, and the operation instructions, and the transition edges of the predefined page state graph are updated. During the update of the predefined page state graph, stability checks can be performed on unknown pages; for pages such as dynamic video streams, a pause and retry can be attempted first, followed by combining image hash values, text recognition metrics, and the task execution trajectory to complete page recognition.

[0049] The advantage of this setup is that by establishing a continuously expandable page state graph in the target mobile application, automated operations are no longer just a fixed coordinate sequence, but a state transition process of "detecting the current position - planning page transitions - executing - verifying", thus avoiding the problem of fixed scripts easily failing due to changes in control text or coordinates.

[0050] Optionally, determining whether an actual page screenshot belongs to an unknown page or a dynamic page may include: determining the actual image hash value of the actual page screenshot, and obtaining each known page corresponding to the target mobile application, as well as multiple known image hash values ​​corresponding to each known page; calculating the Hamming distance between the actual image hash value and the known image hash values ​​of each known page according to the preset priority of each known page; if the Hamming distance between the actual image hash value and the known image hash values ​​of each known page is greater than or equal to a preset distance threshold, then the actual page screenshot is determined to belong to an unknown page or a dynamic page. If the Hamming distance between the actual image hash value and the known image hash values ​​of each known page is less than the preset distance threshold, then the number of text recognition indicators matching between the actual page screenshot and each known page is counted sequentially. If any text recognition indicator is greater than or equal to a preset indicator number threshold, then the actual page screenshot is determined not to belong to an unknown page or a dynamic page; if the number of text recognition indicators is less than the preset indicator number threshold, then the actual page screenshot is determined to belong to an unknown page or a dynamic page.

[0051] The advantage of this setup is that when the actual page screenshot belongs to an unknown or dynamic page, the visual language model is called to generate an update task script, which can automatically complete the exploration, memorization, and generation of executable operations.

[0052] S260. Alternately test the current task script and the update task script. When the number of tests for both the current task script and the update task script reaches the preset number of tests, and the success rate of the update task script is greater than that of the current task script, run the update task script to complete the application operation task.

[0053] In this step, specifically, when alternating between testing the current task script and the update task script, independent verification can be performed using an external verification function or a built-in verification function within the script. When verification passes, the test result is recorded as successful. Then, the update task script can be run only after both the current and update task scripts have reached the preset number of tests, and the success rate of the update task script is greater than that of the current task script.

[0054] Optionally, the update task script is run, including: determining whether the test success rate of the update task script is greater than or equal to a first preset success rate threshold; if so, the update task script is run directly; if not, a preset quality reviewer is used to identify any abnormal issues in the update task script; the update task script is corrected based on the abnormal issues to obtain a corrected task script, and the corrected task script is run.

[0055] The first preset success rate threshold can be understood as a success rate threshold determined based on user needs and historical experience. For example, the first preset success rate threshold can be set to 0.9.

[0056] In one specific implementation, the update task script can be run directly when the test success rate of the update task script is greater than or equal to a first preset success rate threshold. When the test success rate of the update task script is less than the first preset success rate threshold, a preset quality reviewer is used to identify any abnormal issues in the update task script; the update task script is then corrected based on these abnormal issues to obtain a corrected task script, which is then run.

[0057] In another specific implementation, the update task script can be run directly when the test success rate of the update task script is greater than or equal to a first preset success rate threshold, and the visual language model fallback rate is less than or equal to a preset fallback rate threshold. When the test success rate of the update task script is less than the first preset success rate threshold, or the visual language model fallback rate is greater than the preset fallback rate threshold, a preset quality reviewer is used to identify abnormal problems in the update task script; the update task script is then corrected based on these abnormal problems to obtain the corrected task script. Furthermore, the corrected task script and the current task script can be tested alternately, and the corrected task script is run only when the number of tests for both the corrected and current task scripts reaches a preset number of tests threshold, the test success rate of the corrected task script is greater than that of the current task script, and the visual language model fallback rate of the corrected task script is less than or equal to that of the current task script.

[0058] For example, assuming that after several runs, the success rate of the update task script is less than 0.9%, or the visual language model fallback rate is greater than 10%, a preset quality review tool is used to identify abnormal issues in the update task script, such as page misclassification or insufficient entry loading time. Then, the update task script can be corrected based on these abnormal issues, resulting in a corrected task script. The corrected task script and the current task script are executed alternately. When both the corrected and current task scripts have been executed three times, if the success rate of the corrected task script is greater than that of the current task script, and the visual language model fallback rate of the corrected task script is less than or equal to that of the current task script, then the corrected task script is executed.

[0059] Optionally, run the update task script, including: generating a set of anthropomorphic operation primitives corresponding to the update task script; and executing each anthropomorphic operation primitive in the set of anthropomorphic operation primitives in sequence.

[0060] By setting the above, you can avoid directly hardcoding raw coordinates or directly calling the underlying click interface in the task update script, thus reducing the risk of being identified by the anti-automation mechanism of mobile applications.

[0061] S270. When the number of times the update task script is executed reaches the preset number of executions and the success rate of the update task script is greater than the second preset success rate threshold, the update task script replaces the current task script in the script library so that different user-side intelligent agents can use the update task script to perform application operation tasks.

[0062] The preset execution count can be understood as a threshold number of executions determined based on user needs and historical experience. For example, the second preset success rate threshold can be set to 5. The second preset success rate threshold can also be understood as a success rate threshold determined based on user needs and historical experience. For example, the second preset success rate threshold can be set to 0.7.

[0063] Specifically, this step involves determining whether a valid current task script exists in the script library. If not, the updated task script can replace the current task script in the script library if its execution success rate is greater than a second preset success rate threshold, such as 0.7. If it is, the updated task script's execution success rate is further determined to be greater than a third preset success rate threshold, such as 0.9. When the updated task script's execution success rate is greater than the third preset success rate threshold, the updated task script is considered to have excellent results, and it can then replace the current task script in the script library.

[0064] When the success rate of the update task script is less than or equal to the third preset success rate threshold, if the number of times the update task script is executed reaches a preset number of executions (e.g., 5), and the success rate of the update task script is greater than the second preset success rate threshold (e.g., 0.7), then the update task script replaces the current task script in the script library. Alternatively, if the number of times the update task script is executed reaches a preset number of executions (e.g., 5), and the success rate of the update task script is higher than the success rate of the current task script by at least a preset difference (e.g., 0.05), then the update task script replaces the current task script in the script library.

[0065] The advantage of this setup is that, compared to existing technologies that directly replace the current task script in the script library with the updated task script, the technical solution in this embodiment fully considers the number of script executions and the success rate, avoids replacing the original script with a script that has poor performance, and ensures the effective availability of the scripts stored in the script library.

[0066] Optionally, when the application operation task is a new task without predefined operations, the visual language model can be directly invoked to execute the application operation task based on the actual page screenshot. When the application operation task is successfully executed, the task execution trajectory and the actual page state diagram are output. Then, based on the actual page screenshot, task execution trajectory, and actual page state diagram, the actual task script is generated. Finally, the actual task script can be run to complete the application operation task.

[0067] Optionally, to ensure the accuracy of the actual page state diagram, a consensus page can be formed after the same page state is confirmed by at least two different user-side agents before it is distributed to unconfirmed user-side agents.

[0068] The technical solution of this embodiment determines whether the current page of the target mobile application is consistent with the target page by using the current image hash value of the current page and the number of text recognition indicators that match between the current page and the target page. This improves the accuracy of page consistency determination and reduces automated execution anomalies. Secondly, when the number of executions of the update task script reaches a preset number and the success rate of the update task script is greater than a second preset success rate threshold, the update task script replaces the current task script in the script library. This allows different user-side agents to use the update task script to perform application operation tasks, fully considering the number of script executions and the success rate, avoiding the use of ineffective scripts to replace the original scripts, and ensuring the effective availability of scripts stored in the script library.

[0069] Example 3 Figure 4 This is a schematic diagram of a mobile application control device according to Embodiment 3 of the present invention. This embodiment is applicable to the situation of automating the control of mobile applications. The mobile application control device can be implemented in hardware and / or software and can be configured in the user-side intelligent agent.

[0070] like Figure 4 As shown, the mobile application control device disclosed in this embodiment is applied to a mobile application control system, which includes: multiple user-side intelligent agents and one community-side intelligent agent. The device is executed by the user-side intelligent agents and includes: The target page determination module 31 is used to determine the target page that the target mobile application needs to reach based on the application operation task sent to the target mobile application. The script execution module 32 is used to obtain the current task script corresponding to the application operation task from the script library maintained by the community-side intelligent agent when it is determined that the current page of the target mobile application is inconsistent with the target page, and to run the current task script. The script update module 33 is used to collect actual page screenshots of the target mobile application if the current task script fails to execute, and generate an update task script based on the actual page screenshots and application operation tasks through a pre-built visual language model. The script verification module 34 is used to alternately test the current task script and the update task script. When both the current task script and the update task script have reached the preset number of tests, and the success rate of the update task script is greater than that of the current task script, the update task script is run to complete the application operation task. The technical solution in this embodiment, through the cooperation of the target page determination module 31, the script execution module 32, the script update module 33, and the script verification module 34, solves the problem of low script update efficiency and poor usability of the updated script caused by relying on manual documents, manual recording, or centralized training samples for script generation and iterative updates. It improves the script update efficiency, ensures the usability of the updated script, and thus improves the automation control efficiency and operation accuracy of mobile applications.

[0071] Optionally, the script execution module 32 includes: The hash value determination unit is used to determine the current image hash value of the current page and obtain the hash values ​​of multiple reference images corresponding to the target page; The identification index determination unit is used to calculate the Hamming distance between the current image hash value and the hash values ​​of each reference image, and when each Hamming distance is less than the preset distance threshold, count the number of text recognition indexes that match between the current page and the target page. The page determination unit is used to determine that the current page is inconsistent with the target page if the number of text recognition indicators is less than a preset indicator threshold.

[0072] Optionally, the hash value determination unit is specifically used for: cropping the top status bar area of ​​the current page to obtain the cropped page, scaling the cropped page to a preset size to obtain the scaled page; and generating the current image hash value corresponding to the current page based on all pixel values ​​of the scaled page.

[0073] Optionally, the execution flow of the visual language model includes: executing application operation tasks based on actual page screenshots, and outputting the task execution trajectory when the application operation tasks are successfully executed; obtaining the operation instructions that were completed before the current task script failed to execute, and generating an updated task script based on the actual page screenshots, task execution trajectories, and operation instructions.

[0074] Optionally, script update module 33 includes: The first script execution unit is used to determine whether the test success rate of the update task script is greater than or equal to the first preset success rate threshold; if so, the update task script is run directly; if not, a preset quality reviewer is used to identify the abnormal problems in the update task script; the update task script is corrected based on the abnormal problems to obtain the corrected task script, and the corrected task script is run. The second script execution unit is used to generate a set of anthropomorphic operation primitives corresponding to the update task script; and executes each anthropomorphic operation primitive in the set of anthropomorphic operation primitives in sequence.

[0075] Optionally, the device also includes a script replacement module, which is used to replace the current task script in the script library with the update task script when the number of times the update task script is executed reaches a preset number of executions and the success rate of the update task script is greater than a second preset success rate threshold.

[0076] The mobile application control device provided in this embodiment of the invention can execute the mobile application control method provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the method execution. Content not described in detail in this embodiment can be referred to the description in any method embodiment of this application.

[0077] Example 4 like Figure 1 As shown in the figure, the mobile application control system disclosed in this embodiment includes: multiple user-side intelligent agents and one community-side intelligent agent, and each user-side intelligent agent can communicate and connect with the community-side intelligent agent.

[0078] The system consists of two parts: a community-side intelligent agent, which maintains a script library containing multiple task scripts; and a user-side intelligent agent, which determines the target page that the target mobile application needs to reach based on the application operation task issued to the target mobile application. If the current page of the target mobile application is inconsistent with the target page, the user-side intelligent agent retrieves the current task script corresponding to the application operation task from the script library maintained by the community-side intelligent agent and runs the current task script. If the current task script fails to execute, the user captures a screenshot of the actual page of the target mobile application and generates an updated task script based on the actual page screenshot and the application operation task using a pre-built visual language model. The user then alternately tests the current task script and the updated task script. The updated task script is run only when both the current and updated task scripts have been tested a preset number of times, and the success rate of the updated task script is greater than that of the current task script, to complete the application operation task.

[0079] Optionally, when determining whether the current page and the target page are consistent, the user-side intelligent agent is specifically used to determine the current image hash value of the current page and obtain multiple reference image hash values ​​corresponding to the target page; calculate the Hamming distance between the current image hash value and each reference image hash value respectively, and when each Hamming distance is less than a preset distance threshold, count the number of matching text recognition indicators between the current page and the target page; if the number of text recognition indicators is less than a preset indicator threshold, then it is determined that the current page and the target page are inconsistent. Specifically, determining the current image hash value of the current page includes: cropping the top status bar area of ​​the current page to obtain a cropped page, scaling the cropped page to a preset size to obtain a scaled page; and generating the current image hash value corresponding to the current page based on all pixel values ​​of the scaled page.

[0080] Optionally, the execution flow of the visual language model includes: executing application operation tasks based on actual page screenshots, and outputting the task execution trajectory when the application operation tasks are successfully executed; obtaining the operation instructions that were completed before the current task script failed to execute, and generating an updated task script based on the actual page screenshots, task execution trajectories, and operation instructions.

[0081] Optionally, when running the update task script, the user-side intelligent agent is specifically used to determine whether the test success rate of the update task script is greater than or equal to the first preset success rate threshold; if so, the update task script is run directly; if not, a preset quality reviewer is used to identify the abnormal problems existing in the update task script; the update task script is corrected based on the abnormal problems to obtain the corrected task script, and the corrected task script is run.

[0082] Optionally, when running the update task script, the user-side intelligent agent is also specifically used to: generate a set of anthropomorphic operation primitives corresponding to the update task script; and execute each anthropomorphic operation primitive in the set of anthropomorphic operation primitives in sequence.

[0083] Optionally, after running the update task script, the user-side agent is also used to: replace the current task script in the script library with the update task script when the number of times the update task script is executed reaches the preset number of executions and the success rate of the update task script is greater than the second preset success rate threshold.

[0084] The technical solution of this invention provides a mobile application control system, which includes a user-side intelligent agent and a community-side intelligent agent. The community-side intelligent agent maintains a script library containing multiple task scripts. The user-side intelligent agent determines the target page that the target mobile application needs to reach based on the application operation task issued to the target mobile application. When it is determined that the current page of the target mobile application is inconsistent with the target page, the user-side intelligent agent retrieves the current task script corresponding to the application operation task from the script library maintained by the community-side intelligent agent and runs the current task script. If the current task script fails to execute, a screenshot of the actual page of the target mobile application is captured and processed through a pre-built visual... The perceptual language model generates update task scripts based on actual page screenshots and application operation tasks. It alternately tests the current task script and the update task script, and only runs the update task script when both scripts have reached a preset number of tests, and the update task script's success rate is higher than the current task script's success rate. This solves the problem of low script update efficiency and poor usability of updated scripts caused by relying on manual documentation, manual recording, or centralized training samples for script generation and iterative updates. It improves script update efficiency, ensures the usability of updated scripts, and thus improves the automation control efficiency and operational accuracy of mobile applications.

[0085] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0086] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A mobile application control method, characterized in that, Applied to a mobile application control system, the mobile application control system includes: a user-side intelligent agent and a community-side intelligent agent; The method is executed by the user-side intelligent agent, and the method includes: Based on the application operation task sent to the target mobile application, determine the target page that the target mobile application needs to reach; When it is determined that the current page of the target mobile application is inconsistent with the target page, the current task script corresponding to the application operation task is obtained from the script library maintained by the community-side intelligent agent, and the current task script is run. If the current task script fails to execute, then an actual page screenshot of the target mobile application is captured, and an updated task script is generated based on the actual page screenshot and the application operation task using a pre-built visual language model. The current task script and the updated task script are tested alternately. When the number of tests for both the current task script and the updated task script reaches the preset number of tests, and the success rate of the updated task script is greater than that of the current task script, the updated task script is run to complete the application operation task.

2. The method according to claim 1, characterized in that, Determining that the current page of the target mobile application is inconsistent with the target page includes: Determine the current image hash value of the current page, and obtain multiple reference image hash values ​​corresponding to the target page; Calculate the Hamming distance between the current image hash value and each of the reference image hash values ​​respectively, and when each of the Hamming distances is less than a preset distance threshold, count the number of text recognition indicators that match between the current page and the target page; If the number of text recognition metrics is less than a preset metric threshold, then the current page is determined to be inconsistent with the target page.

3. The method according to claim 2, characterized in that, Determining the current image hash value of the current page includes: The top status bar area of ​​the current page is cropped to obtain the cropped page, and the cropped page is scaled to a preset size to obtain the scaled page; Generate the current image hash value corresponding to the current page based on all pixel values ​​of the scaled page.

4. The method according to claim 1, characterized in that, The execution flow of the visual language model includes: Execute the application operation task based on the actual page screenshot, and output the task execution trajectory when the application operation task is successfully executed; Obtain the operation instructions that were completed before the current task script failed to execute, and generate an updated task script based on the actual page screenshot, the task execution trajectory, and the operation instructions.

5. The method according to claim 1, characterized in that, Running the update task script includes: Determine whether the test success rate of the update task script is greater than or equal to a first preset success rate threshold; If so, then run the update task script directly; If not, a preset quality reviewer is used to determine any abnormalities in the update task script; Based on the aforementioned anomaly, the update task script is corrected to obtain the corrected task script, and the corrected task script is then run.

6. The method according to claim 1, characterized in that, Running the update task script includes: Generate a set of anthropomorphic operation primitives corresponding to the update task script; Each anthropomorphic operation primitive in the set of anthropomorphic operation primitives is executed sequentially.

7. The method according to claim 1, characterized in that, After running the update task script, the following is also included: When the number of times the update task script is executed reaches the preset number of executions, and the success rate of the update task script is greater than the second preset success rate threshold, the update task script replaces the current task script in the script library.

8. A mobile application control device, characterized in that, Applied to a mobile application control system, the mobile application control system includes: a user-side intelligent agent and a community-side intelligent agent; The device is executed by the user-side intelligent agent, and the device includes: The target page determination module is used to determine the target page that the target mobile application needs to reach based on the application operation task sent to the target mobile application; The script execution module is used to retrieve the current task script corresponding to the application operation task from the script library maintained by the community-side intelligent agent when it is determined that the current page of the target mobile application is inconsistent with the target page, and to run the current task script. The script update module is used to collect actual page screenshots of the target mobile application if the current task script fails to execute, and generate an update task script based on the actual page screenshots and the application operation tasks using a pre-built visual language model. The script verification module is used to alternately test the current task script and the update task script, and when the number of tests for both the current task script and the update task script reaches a preset number of tests, and the test success rate of the update task script is greater than that of the current task script, the update task script is run to complete the application operation task.

9. A mobile application control system, characterized in that, The mobile application control system includes: a user-side intelligent agent and a community-side intelligent agent; The community-side intelligent agent is used to maintain a script library containing multiple task scripts; The user-side intelligent agent is used to determine the target page that the target mobile application needs to reach based on the application operation task sent to the target mobile application. When it is determined that the current page of the target mobile application is inconsistent with the target page, the current task script corresponding to the application operation task is obtained from the script library maintained by the community-side intelligent agent, and the current task script is run. If the current task script fails to execute, then an actual page screenshot of the target mobile application is captured, and an updated task script is generated based on the actual page screenshot and the application operation task using a pre-built visual language model. The current task script and the updated task script are tested alternately. When the number of tests for both the current task script and the updated task script reaches the preset number of tests, and the success rate of the updated task script is greater than that of the current task script, the updated task script is run to complete the application operation task.

10. The system according to claim 9, characterized in that, The user-side intelligent agent is specifically used for: Determine the current image hash value of the current page, and obtain multiple reference image hash values ​​corresponding to the target page; Calculate the Hamming distance between the current image hash value and each of the reference image hash values ​​respectively, and when each of the Hamming distances is less than a preset distance threshold, count the number of text recognition indicators that match between the current page and the target page; If the number of text recognition metrics is less than a preset metric threshold, then the current page is determined to be inconsistent with the target page.

Images