Automatic robot implementation method and device based on barrier-free service

Abstract

The invention discloses an automatic robot implementation method and device based on barrier-free service. The method comprises the following steps: initializing and starting a barrier-free service through a barrier-free service API (Application Program Interface), and registering real-time monitoring on interface elements; acquiring all accessible interface elements on a current screen by utilizing the barrier-free service, and identifying and classifying the interface elements according to the characteristic attributes of the elements; and automatically generating a task script based on the classified interface elements, and simulating user interface interaction by executing the task script. The technical problem that an existing automatic tool has a compatibility problem in different system versions and is difficult to adapt to all systems is solved.

Description

Technical Field

[0001] The present invention relates to the field of computers, and in particular, to a method and device for implementing an automated robot based on an accessibility service. Background Art

[0002] With the growth of the demand for mobile application automation, existing automation tools generally adopt a low-level control method based on coordinate clicking or Root permission, which has significant technical limitations. Traditional solutions rely on fixed screen coordinates for interface operations, making it difficult to adapt to devices with different resolutions, and often causing script failures due to changes in the interface layout after system version upgrades, resulting in high maintenance costs. In addition, some tools achieve control traversal by obtaining the system Root permission, which not only violates the application security specification, but also faces the risk of technical failure due to the tightening of Android system permissions, seriously restricting the universal development of automation technology.

[0003] In view of the above problems, no effective solution has been proposed yet. Summary of the Invention

[0004] Embodiments of the present invention provide a method and device for implementing an automated robot based on an accessibility service, so as to at least solve the technical problems that existing automation tools have compatibility problems in different system versions and are difficult to adapt to all systems.

[0005] According to one aspect of the embodiments of the present invention, there is provided a method for implementing an automated robot based on an accessibility service, including: initializing and starting an accessibility service through an accessibility service API, and registering real-time monitoring of interface elements; using the accessibility service to obtain all accessible interface elements on the current screen, and identifying and classifying the interface elements according to the characteristic attributes of the elements; based on the classified interface elements, automatically generating a task script, and simulating user interface interaction by executing the task script.

[0006] According to another aspect of the embodiments of the present invention, there is also provided a device for implementing an automated robot based on an accessibility service, including: an initialization module configured to initialize and start an accessibility service through an accessibility service API, and register real-time monitoring of interface elements; a classification module configured to use the accessibility service to obtain all accessible interface elements on the current screen, and identify and classify the interface elements according to the characteristic attributes of the elements; an automatic processing module configured to automatically generate a task script based on the classified interface elements, and simulate user interface interaction by executing the task script.

[0007] In an embodiment of the present invention, an accessibility service is initialized and started through an accessibility service API, and real-time monitoring of interface elements is registered; all accessible interface elements on the current screen are obtained by using the accessibility service, and the interface elements are identified and classified according to the characteristic attributes of the elements; based on the classified interface elements, a task script is automatically generated, and user interface interaction is simulated by executing the task script. Through the above solution, the technical problems that existing automation tools have compatibility problems in different system versions and are difficult to adapt to all systems are solved. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The illustrative embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation to the present invention. In the drawings:

[0009] Figure 1 is a flowchart of a method for implementing an automation robot based on an accessibility service according to an embodiment of the present invention;

[0010] Figure 2 is a flowchart of another method for implementing an automation robot based on an accessibility service according to an embodiment of the present invention;

[0011] Figure 3 is a flowchart of yet another method for implementing an automation robot based on an accessibility service according to an embodiment of the present invention;

[0012] Figure 4 is a schematic structural diagram of an apparatus for implementing an automation robot based on an accessibility service according to an embodiment of the present invention;

[0013] Figure 5 shows a schematic structural diagram of an electronic device suitable for implementing the embodiments of the present disclosure. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0014] In order to enable those skilled in the art to better understand the solution of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0015] It should be noted that the terms "first", "second", etc. in the description, claims and above-mentioned drawings of the present invention are used to distinguish similar objects and do not necessarily have to be used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances so that the embodiments of the present invention described herein can be implemented in an order different from those illustrated or described herein. In addition, the terms "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device comprising a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0016] According to an embodiment of the present invention, there is provided a method embodiment for implementing an automated robot based on an accessibility service. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. And although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that here.

[0017] Figure 1 is a flowchart of a method for implementing an automated robot based on an accessibility service according to an embodiment of the present invention, as Figure 1 shown, the method includes the following steps:

[0018] Step S102, initialize and start the accessibility service through the accessibility service API, and register a real-time listening for interface elements.

[0019] Android accessibility service is a system function mainly used to assist users in accessing elements on the device screen, especially for users with visual impairments. An automated robot simulates user operations through tools or scripts to complete tasks on an Android device.

[0020] In this embodiment, when initializing the accessibility service API, the corresponding service configuration parameters are dynamically loaded according to the device system version to ensure interface compatibility; after starting the service, a global listening for the hierarchical structure of interface elements is registered, and the creation, update or destruction events of interface elements are captured in real time through a callback function, and an event queue is established for asynchronous processing. In the service initialization stage, the system window management service is synchronously bound to obtain the screen rendering frame rate and layout refresh status, and the timing synchronization between the listening thread and the interface rendering is achieved.

[0021] In this embodiment, by dynamically loading system version adaptation parameters, the differences in the accessibility service interfaces between different Android versions are eliminated, the initialization failure caused by API changes is avoided, and the success rate of service startup is guaranteed. In addition, a timing synchronization mechanism is implemented based on the window management service, effectively solving the timing conflict problem between interface refresh and event listening, reducing the event loss rate, and ensuring the integrity and real-time nature of interface element listening.

[0022] Step S104: Use the accessibility service to obtain all accessible interface elements on the current screen, and identify and classify the interface elements according to the characteristic attributes of the elements.

[0023] Extract the characteristic attributes of the interface elements, where the characteristic attributes include at least one of the following: identifier, text description, and coordinate position; classify the interface elements into text controls, button controls, or input box controls according to the characteristic attributes.

[0024] In this embodiment, through the three-dimensional feature extraction of the comprehensive identifier hash value, text semantic vector, and normalized coordinates, visually similar controls (such as icon buttons and text labels) are effectively distinguished, improving the classification accuracy. In addition, based on the composite matching strategy of the identifier and relative coordinates, the problems of control ID changes or layout offsets in different Android system versions are avoided, enabling the classification model to maintain stability between system versions.

[0025] Step S106: Based on the classified interface elements, automatically generate a task script, and simulate user interface interaction by executing the task script.

[0026] According to the type of the classified interface elements, match the predefined operation templates; fill the coordinate positions or identifiers of the interface elements into the operation templates to generate executable script instructions as the task script. Through the above solution, the script generation efficiency and accuracy are significantly improved. At the same time, through the identifier priority matching strategy, the coordinate offset problem caused by screen resolution differences is avoided.

[0027] After simulating user interface interactions by executing the task script, for dynamically changing interface elements, the operation instructions of the task script are dynamically adjusted based on the real-time monitoring function of the accessibility service. For example, when it is detected that the position or attributes of an interface element have changed, the characteristic attributes of the dynamically changing interface element are re-acquired; based on the re-acquired characteristic attributes, the operation instructions of the task script are dynamically adjusted based on the real-time monitoring function of the accessibility service. In this embodiment, through the dual mechanisms of real-time monitoring and dynamic script adjustment, adaptive control of interface interactions is achieved. When it is detected that the attributes or positions of interface elements change, feature re-extraction and script instruction update are completed in a timely manner, effectively coping with dynamic scenarios such as pop-up window interference and network latency, reducing the interruption rate of the automation process, and at the same time reducing the resource consumption of global reloading through incremental script correction technology, significantly improving the robustness of task execution in complex scenarios.

[0028] Currently, the methods for executing automated tasks in the Android system usually rely on tools such as UIAutomator and Espresso. These tools simulate operations by capturing interface elements, achieving automated user interface interactions. Although these tools can complete automated tasks to a certain extent, there are still problems such as complex operations and poor adaptability. The Android accessibility service is a system function designed to provide a more friendly interaction experience for special groups (such as visually impaired users), and it can access and interact with all elements on the screen. By leveraging the accessibility service, more comprehensive operations and monitoring of the application interface can be achieved, which is applicable to different versions of Android devices and provides greater compatibility. However, there is currently no solution to organically combine the accessibility service with automated tasks to achieve comprehensive execution of Android automated tasks, resulting in: 1) Existing automated tools have compatibility issues in different Android versions and are difficult to adapt to all Android devices. 2) A relatively high technical threshold is required to write and maintain automated scripts, making it difficult for ordinary developers to master and use. 3) For dynamically changing interface elements, existing tools often cannot accurately locate them, resulting in unstable automation processes.

[0029] To solve the above problems, the present invention proposes an Android automation robot implemented based on the accessibility service, aiming to overcome the compatibility and operation complexity problems in the prior art and provide a more intelligent, user-friendly, and general automation solution. As Figure 2 shown, the UML process for implementing this automation robot includes the initialization of the accessibility service, the acquisition and recognition of elements, the generation and execution of task scripts, as well as the processing of dynamic elements and the feedback of task results. Specifically, it includes the following steps:

[0030] Step S202, initialize the accessibility service and register the monitoring of interface elements.

[0031] Start the accessibility service through the Android system's accessibility service API and register the listening for interface elements.

[0032] Step S204, Element acquisition and recognition.

[0033] Use the accessibility service to obtain all accessible elements on the current screen, including text, buttons, input boxes, etc. Identify and classify them based on the characteristic attributes of the elements (such as ID, text description).

[0034] Step S206, Task script generation and execution.

[0035] Based on the interface elements obtained by the accessibility service, automatically generate corresponding task scripts. These scripts include operations such as clicking, inputting, and swiping, which are used to simulate user interactions on the interface; by automatically executing the scripts, simulate user operations to verify the operability and responsiveness of the interface elements.

[0036] Step S208, Dynamic element processing.

[0037] For dynamically changing elements on the interface (such as asynchronously loaded content, animated elements), utilize the real-time listening function of the accessibility service to automatically adjust the operation steps in the script to ensure the stability of the task process.

[0038] Step S210, Task result feedback and optimization.

[0039] After the task is completed, collect the task execution results, including data such as operation success rate, interface response time, and abnormal behaviors. Based on these data, automatically analyze and generate a report to provide optimization suggestions.

[0040] The key technology of the present invention lies in using the Android accessibility service to obtain interface elements. In other embodiments, it is also possible to achieve access and operation of interface elements through other system-level services (such as assistive function services). In addition, image recognition technology can also be used in combination with the accessibility service to identify and operate on the interface to achieve similar automation functions.

[0041] This embodiment mainly adopts the following solutions: 1) Obtain interface elements using Android accessibility services: All elements on the screen are obtained in real time through accessibility services and operated on. 2) Automatic script generation: Task scripts are automatically generated based on the characteristics of interface elements without manual writing, reducing the usage threshold. 3) Dynamic processing mechanism: For dynamically changing interface elements, the listening ability of accessibility services is utilized to automatically adjust task operations, improving the stability and reliability of the process. Through the above solutions, this embodiment has the following beneficial effects: 1) Stronger compatibility: By using the built-in accessibility services of the Android system, it does not rely on specific tool versions and can adapt to various Android devices and system versions. 2) Higher usability: Task scripts are automatically generated, and users do not need to have programming skills, reducing the technical threshold and enabling more developers to easily use this tool for automated tasks. 3) Support for dynamic interfaces: Through real-time listening, it can handle dynamically changing interface elements and ensure the stable execution of task scripts.

[0042] The embodiment of the present application also provides an automated robot system with environment perception and dynamic response capabilities. This system adopts a four-layer collaborative architecture: The accessibility service proxy layer is responsible for interacting with the underlying Android system to capture screen element change events; the semantic element parsing engine performs multi-dimensional feature extraction and intelligent classification on interface elements; the intelligent script generator dynamically constructs an executable operation sequence based on the classification results; the adaptive execution controller responds to interface dynamic changes through a real-time feedback mechanism. Each module communicates asynchronously through an event bus and adopts a streaming data processing mode to ensure high throughput and low latency characteristics, forming a complete closed loop from environment perception to behavior decision-making.

[0043] Figure 3 is a flowchart of an implementation method of an automated robot with environment perception and dynamic response capabilities according to an embodiment of the present application, as Figure 3 shown, this method includes the following steps:

[0044] Step S302, initialization of the accessibility service and enhancement of listening.

[0045] During the initialization phase of the accessibility service, the system constructs a version feature fingerprint library that stores the accessibility service API difference parameters for each version from Android 4.4 to 14. By dynamically loading the configuration parameters corresponding to the device system version, it automatically adapts to the interface call specifications between different versions. For example, it handles the restriction policy for starting background services in versions above Android 9.0. After the service starts, it establishes a deep binding with the window management service and achieves hard synchronization between the event capture thread and the screen refresh cycle by monitoring the VSYNC vertical synchronization signal, ensuring that the element status scan is triggered immediately after each interface rendering is completed. For the event listening scenario, a dual-channel filtering mechanism is designed: the global channel captures macroscopic change events such as window switching, and the local channel focuses on control-level operations within the current active window. High-concurrency event processing is achieved through a circular buffer queue, and the priority scheduling algorithm is combined to ensure real-time response to key operation events (such as button clicks).

[0046] To improve the listening stability, a window animation state detection module is introduced. When system-level transition animations (such as page sliding, pop-up window expansion) are detected, event processing is automatically delayed until the animation ends, avoiding misjudgment caused by the instantaneous invisibility of interface elements.

[0047] Step S304, parse the semantic elements.

[0048] The element feature extraction adopts a multi-modal fusion strategy. The control identifier is hashed and encoded to generate a unique fingerprint to solve the problem of matching failure caused by dynamically generated IDs; the text description is processed by semantic vectorization, and the text content is mapped into a 768-dimensional feature vector through a lightweight natural language model to support semantic similarity calculation in cross-language scenarios; the physical coordinates are converted through a normalization algorithm to establish a relative coordinate system based on screen percentage, eliminating the positioning deviation caused by device resolution differences. For special controls (such as dynamically loaded list items), a layout relationship graph is constructed, and the functional attributes of the control are inferred by analyzing the hierarchical relationship between the parent and child containers and the positions of sibling nodes.

[0049] The classification model adopts a hybrid architecture of a rule engine and machine learning. The primary classifier makes judgments based on preset logic. For example, a control that is clickable and contains text is identified as a button; the secondary classifier introduces a gradient boosting decision tree model to make probability predictions based on 32-dimensional features (including text vector similarity, layout depth, adjacent control types, etc.). When the confidence level is lower than the threshold, a review mechanism is triggered. The model supports online incremental learning. When a new type of control (such as a collapsible menu) is detected, samples are automatically collected and the model is fine-tuned to ensure the continuous evolution of the classification system.

[0050] Specifically, first, a primary classifier is constructed. The primary classifier adopts a logical decision-making system based on control attributes and constructs multi-layer filtering rules to achieve fast classification. First, the operability attributes of the control are parsed: if the clickable and focusable flags are both true and the long-clickable is false, it is initially determined as an interactive control; further, the text attributes are checked. When the text field is non-empty and the length is between 2 and 20 characters, it is marked as a candidate button. For input controls, they are identified by combining the inputType attribute value (such as textEmailAddress, numberDecimal) with the isEditable state. If the control has both editable attributes and a specific input type, it is classified as an input box. In terms of layout features, the position weight of the control in the parent container is calculated: if the control is located at the bottom of the layout and the width ratio exceeds 60%, its priority as a bottom navigation button is enhanced.

[0051] Next, a secondary machine learning model is constructed. The secondary machine learning model inputs a 32-dimensional feature vector, covering the following key dimensions: 1) Text semantic features: Extract a 768-dimensional vector of the text description through a lightweight BERT model, reduce the dimension to 8 main components through PCA, and calculate the cosine similarity score with the standard template (such as the semantic correlation between "submit" and "confirm"). 2) Spatial relationship features: Include layout depth (the number of levels from the root node to the current control), index position among sibling nodes, and type distribution of adjacent controls (such as class statistics of the first two and the last two adjacent controls). 3) Visual presentation features: Calculate the aspect ratio of the control, the brightness value of the background color, and the ratio of the icon size to the text size. 4) Behavioral pattern features: Statistically calculate the click success rate and mis-touch rate of the same type of controls in the historical operation records. The model is trained using the XGBoost framework, setting early stopping to prevent overfitting, and optimizing hyperparameters through five-fold cross-validation. The final model contains 120 decision trees, with a maximum depth limit of 7 levels and a learning rate set to 0.05. In the prediction stage, the probability distribution of each control category is output. When the highest probability value is lower than the 0.85 threshold, it is determined as a low-confidence classification.

[0052] When a low-confidence sample triggers the review process, the system automatically intercepts the image of the screen area where the control is located and packs the feature data (including original attributes, predicted probability distribution, and context control snapshots) into the queue to be reviewed. The review interface presents the following information: the visual contour map of the control, the layout relationship diagram of adjacent elements, the machine prediction result, and alternative category suggestions. During the review, correction can be completed by checking the corrected label or adding comments, and the system records the correction result in the annotation database. To improve the review efficiency, an intelligent sorting algorithm is used to dynamically adjust the priority of the queue to be reviewed according to the control appearance frequency, business criticality, and historical correction records.

[0053] Establish a control feature evolution tracking system, and maintain a versioned feature template for each control category. When a persistent deviation is detected in the core features of a certain type of control (such as the standard size of a button, the placeholder text pattern of an input box), a feature change report is automatically generated. For example, if it is detected that the median width of the "OK button" gradually changes from 120dp to 140dp, the system will trigger an upgrade of the feature template version, while retaining the historical template for compatibility with the old version interface. For semantic features, build a synonym knowledge graph to dynamically expand the matching rules, such as establishing a cross-language mapping relationship between "Login" and "Sign in", and automatically adding the newly emerged text "Entrar" to the dictionary when it reaches a certain frequency.

[0054] When the machine learning service is unavailable, the system automatically switches to the degraded mode. The primary rule engine combines the K-Nearest Neighbors (KNN) algorithm to continue providing services, and uses a pre-built feature vector library for similarity matching. All degradation operations are marked with special tags, and after the service is restored, they are replayed to the machine learning pipeline for differential analysis. For control categories with persistent classification failures, an isolation mechanism is initiated to route them to a dedicated processing channel to avoid affecting the mainstream classification process. The system generates a health report every hour, monitors key metrics such as the model drift index and the feature distribution deviation degree, and triggers an automatic rollback process when the warning threshold is exceeded.

[0055] Step S306, intelligent script generation and dynamic correction.

[0056] The script generation engine has an operation template knowledge base built-in. Each template defines the target control features, execution actions, and exception handling strategies. A multi-dimensional matching algorithm is used to calculate the applicability weight of the template, taking into account factors such as the historical execution success rate, the current device adaptability, and the interface layout matching degree, and an initial script is generated through the optimal path selection. For coordinate-sensitive operations, using virtual coordinate mapping, the relative layout position is converted into an anti-jitter click instruction for the physical device, and by introducing a random position offset to simulate the characteristics of manual operations, the system's anti-automation detection mechanism is effectively avoided.

[0057] The dynamic adjustment module constructs an event-driven response system. When the real-time monitoring system detects changes in the attributes of interface elements (such as text updates) or layout offsets (coordinate changes exceeding 5 pixels), it triggers a script differential correction process: only locally updates the affected operation instructions and retains the verified and effective execution segments. To address the asynchronous element loading problem caused by network latency, a state prediction model is adopted, and the Kalman filter is used to estimate the time window when the control appears, and a waiting strategy is dynamically inserted.

[0058] The operation template system adopts a multi-layer structured design, including three core modules: target feature description, execution action definition, and exception handling strategy. The target features precisely define the control attributes through composite conditions, supporting regular expression matching of control identifiers, semantic model parsing of text content, and geometric constraints to limit the control position area. The execution action library predefines 20 standardized operation types, covering basic interactions such as clicking, swiping, and input, and supports parameterized configuration of refined control parameters such as click duration and retry strategy. The exception handling module adopts a hierarchical response mechanism, defining response strategies for 32 types of exception scenarios such as element loss and timeout without response, forming a multi-level fallback solution from simple retry to process jump.

[0059] The template matching process introduces a multi-dimensional weight calculation model, comprehensively evaluating four core indicators: control identifier similarity, text semantic relevance, layout position matching degree, and historical execution success rate. The identifier similarity quantifies the matching degree between the control ID and the template pattern through the edit distance algorithm. The text semantic relevance uses a lightweight BERT model to generate semantic vectors and calculates the cosine similarity score. The layout matching degree uses the Hausdorff distance algorithm to compare the geometric consistency between the actual position of the control and the template-defined area. The historical success rate dynamically counts the execution performance of each template in the same device environment. After the matching results are normalized, the optimal allocation is performed, constructing a bipartite graph model of controls and templates, and solving the maximum weight matching through the KM algorithm to effectively solve the multi-template competition problem. For the controls that fail to match successfully, an adaptive template generation process is started, recording the feature patterns into the new template candidate pool for subsequent manual review and model training.

[0060] In the script generation stage, a device-independent coordinate mapping technology is adopted to convert the absolute coordinates into normalized values based on the screen ratio, eliminating the influence of resolution differences. An anti-jitter mechanism is designed for touch operations, generating a sequence of click points that conform to the Gaussian distribution around the target coordinates, simulating the randomness of manual operations to avoid system anti-automation detection. When injecting control identifiers, a composite fingerprint system is constructed, fusing multi-dimensional features such as the resource ID hash value, truncated text content, layout hierarchy depth, and sibling node index to generate a unique identifier, and maintaining a hot control mapping table through the LRU cache strategy to improve the matching efficiency. The script instructions are written in a structured description language, encapsulating a triple of operation type, target parameter, and timing control, and embedding exception monitoring breakpoints to form an interruptible and recoverable execution flow.

[0061] When detecting dynamic changes in interface elements, an incremental script update process is triggered. Parse the original script to generate an abstract syntax tree, use the tree difference algorithm to locate the changed nodes, and generate a lightweight patch package to achieve local logic hot update and maintain task continuity.

[0062] Through the design concepts of templatization, parameterization, and adaptability, this application constructs a complete automated chain from element recognition to script execution, significantly reducing maintenance costs while improving development efficiency, and providing a highly robust solution for cross-platform and multi-scenario automated tasks.

[0063] This application also provides an implementation device for an automated robot based on accessibility services, as Figure 4 shown, including: an initialization module 42, configured to initialize and start the accessibility service through the accessibility service API and register real-time monitoring of interface elements; a classification module 44, configured to use the accessibility service to obtain all accessible interface elements on the current screen and identify and classify the interface elements according to the characteristic attributes of the elements; an automatic processing module 46, configured to automatically generate a task script based on the classified interface elements and simulate user interface interaction by executing the task script.

[0064] It should be noted that: for the implementation device of the automated robot provided in the above embodiment, only the above division of each functional module is used for illustration. In actual application, the above functions can be allocated to different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the implementation device of the automated robot provided in the above embodiment and the embodiment of the automated robot implementation method belong to the same concept, and the specific implementation process can be seen in the method embodiment, which will not be elaborated here.

[0065] Figure 5 The figure shows a schematic structural diagram of an electronic device suitable for implementing the embodiments of the present disclosure. It should be noted that Figure 5 the shown electronic device is only an example and should not bring any limitation to the functions and usage scope of the embodiments of the present disclosure.

[0066] As Figure 5 shown, the electronic device includes a central processing unit (CPU) 1001, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 1002 or the program loaded from the storage section 1008 into the random access memory (RAM) 1003. In the RAM 1003, various programs and data required for system operation are also stored. The CPU 1001, ROM 1002, and RAM 1003 are connected to each other through a bus 1004. The input / output (I / O) interface 1005 is also connected to the bus 1004.

[0067] The following components are connected to the I / O interface 1005: an input section 1006 including a keyboard, a mouse, etc.; an output section 1007 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc. and a speaker, etc.; a storage section 1008 including a hard disk, etc.; and a communication section 1009 including a network interface card such as a LAN card, a modem, etc. The communication section 1009 performs communication processing via a network such as the Internet. A drive 1010 is also connected to the I / O interface 1005 as required. A removable medium 1011 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc. is mounted on the drive 1010 as required so that a computer program read therefrom is installed into the storage section 1008 as required.

[0068] The above are only the preferred embodiments of the present application. It should be noted that for those of ordinary skill in the art, without departing from the principle of the present application, several improvements and modifications can be made, and these improvements and modifications should also be regarded as the protection scope of the present application.

Claims

1. An implementation method of an automated robot based on accessibility services, characterized in that, Including: Initialize and start an accessibility service through an accessibility service API, and register a real-time monitoring of interface elements; Use the accessibility service to obtain all accessible interface elements on the current screen, and identify and classify the interface elements according to the characteristic attributes of the elements; Based on the classified interface elements, automatically generate a task script, and simulate user interface interaction by executing the task script.

2. The method according to claim 1, wherein After simulating user interface interaction by executing the task script, the method further includes: for dynamically changing interface elements, dynamically adjust the operation instructions of the task script based on the real-time monitoring function of the accessibility service.

3. The method according to claim 1, wherein Identifying and classifying the interface elements according to the characteristic attributes of the elements includes: Extract the characteristic attributes of the interface elements, where the characteristic attributes include at least one of the following: identifier, text description, and coordinate position; Classify the interface elements into text controls, button controls, or input box controls according to the characteristic attributes.

4. The method according to claim 1, wherein Automatically generating a task script based on the classified interface elements includes: Match a predefined operation template according to the type of the classified interface elements; Fill the coordinate position or identifier of the interface element into the operation template to generate executable script instructions as the task script.

5. The method according to claim 2, characterized in that Dynamically adjusting the operation instructions of the task script based on the real-time monitoring function of the accessibility service for dynamically changing interface elements includes: When it is detected that the position or attribute of an interface element changes, re-obtain the characteristic attributes of the dynamically changing interface element; Based on the re-obtained characteristic attributes, dynamically adjust the operation instructions of the task script based on the real-time monitoring function of the accessibility service.

6. The method according to claim 1, wherein Initializing and starting an accessibility service through an accessibility service API includes: Configure the global monitoring parameters of the accessibility service, where the global monitoring parameters include at least one of the following, interface update frequency and element filtering rules; Based on the global monitoring parameters, register a service instance with the system through the accessibility service API and obtain access rights to the screen content.

7. An implementation device of an automated robot based on accessibility services, characterized in that, Including: An initialization module configured to initialize and start an accessibility service through an accessibility service API and register a real-time monitoring of interface elements; A classification module configured to use the accessibility service to obtain all accessible interface elements on the current screen and identify and classify the interface elements according to the characteristic attributes of the elements; An automatic processing module configured to automatically generate a task script based on the classified interface elements and simulate user interface interaction by executing the task script.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored program, wherein when the program runs, it controls the device where the computer-readable storage medium is located to execute the method according to any one of claims 1 to 6.

9. A computer device, characterized in that, Including: A memory and a processor, The memory stores a computer program; The processor is used to execute the computer program stored in the memory, and when the computer program runs, it causes the processor to execute the method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images