Training data acquisition method, system and equipment based on barrier-free service and medium

Abstract

The invention belongs to the technical field of artificial intelligence, and relates to a training data acquisition method, system and device based on barrier-free service and a medium, and the method comprises the steps: 1) determining a training task; 2) opening a floating window of the marking panel; 3) selecting an operation type of the training task and reducing a floating window of the labeling panel; 4) obtaining a page XML and a page screenshot of the current interface based on the barrier-free service; 5) performing operation by the labeling personnel according to the operation type, acquiring operation data by the interception layer, and performing simulation operation to acquire data generated in the operation process; (6) storing page XML, page screenshot and operation data and data generated in the operation process; 7) judging whether the training task is executed or not, and if so, ending; if not, returning to the step 3), and executing the next operation of the training task. The training data can be conveniently acquired, the acquisition efficiency of the training data is improved, manpower is saved, and the accuracy of the acquired training data is ensured.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence technology, and relates to a method, system, device and medium for obtaining training data, and in particular to a method, system, device and medium for obtaining training data based on barrier-free services. Background Art

[0002] Machine Learning is a branch of artificial intelligence (AI). Its core purpose is to enable computers to learn from data and make predictions or decisions based on the learned experience. In layman's terms, machine learning is to teach computers how to extract patterns and knowledge from data, and then use this knowledge to process new data and give answers or suggestions. To this end, training data is needed for machine learning.

[0003] By collecting data, adding labels and classification information to data samples, we can obtain training data. In the process of obtaining training data, we need to click, long press, slide, input text, call API, record the current page, press the keyboard, and record the data generated by the operation. The Android automated testing framework Espresso is a framework provided by Google for Android application UI testing. It can perform automated clicks, long presses, slides, and other operations and verifications on the user interface. However, although Espresso can automate the operation of the user interface, it can only automate the interface of the current application and cannot operate other apps. Espresso cannot meet the data operations required for some intelligent agent training.

[0004] Therefore, in view of the defects existing in the above-mentioned prior art, it is necessary to develop a new method for obtaining training data. Summary of the invention

[0005] In order to overcome the defects of the prior art, the present invention proposes a training data acquisition method, system, device and medium based on accessible services, which facilitate the acquisition of training data, improve the efficiency of acquiring training data, save manpower, and ensure the accuracy of the acquired training data.

[0006] In order to achieve the above object, the present invention provides the following technical solutions:

[0007] A method for acquiring training data based on barrier-free service, characterized by comprising the following steps:

[0008] 1) Determine the training tasks;

[0009] 2) Open the annotation panel floating window;

[0010] 3) Selecting the operation type of the training task on the annotation panel floating window and shrinking the annotation panel floating window;

[0011] 4) Get the page XML and page screenshots of the current interface based on the accessibility service;

[0012] 5) The annotator performs operations according to the operation type, and the interception layer obtains the data of the operation and performs simulation operations to obtain the data generated during the operation;

[0013] 6) Storing page XML, page screenshots, and the data of the operation obtained by the interception layer and the data generated during the operation;

[0014] 7) Determine whether the training task has been completed. If yes, end the process; if not, return to step 3) and perform the next step of the training task.

[0015] Preferably, the step 1) comprises:

[0016] 11) Determine the description of the training task;

[0017] 12) Determine the ID of the training task;

[0018] 23) Determine the storage address of the labeled data file for the training task.

[0019] Preferably, in step 3), shrinking the annotation panel floating window specifically includes: setting View.GONE of all buttons on the annotation panel floating window to be invisible and setting View.VISIBLE of < on the annotation panel floating window to be visible.

[0020] Preferably, the step 4) specifically includes:

[0021] 41) Take a screenshot of the current interface through the takeScreenshot() method of the Android Accessibility Service AccessibilityService and call back the ScreenshotResult object to the caller, and convert the screenshot into a Bitmap object through the ScreenshotResult object;

[0022] 42) The getRootInActiveWindow() method of the Android accessibility service AccessibilityService returns the AccessibilityNodeInfo object of the current interface, and all the information of the AccessibilityNodeInfo is converted into a string in XML format by parsing the properties of the AccessibilityNodeInfo object.

[0023] Preferably, in step 42), converting all the information of the accessibility service node information AccessibilityNodeInfo into a string in XML format specifically includes:

[0024] 421) Parse the node Node in the accessibility service node information AccessibilityServiceNodeInfo, and convert the parsing result into a variable character sequence StringBuilder;

[0025] 422) adjusting the structure of the variable character sequence StringBuilder, and performing parallel parsing on the child nodes ChildNod of the corresponding node Node;

[0026] 423) Obtain the parsing results of each child node ChildNod and assemble them into a variable character sequence StringBuilder, and adjust the assembled variable character sequence StringBuilder again to obtain a string in XML format.

[0027] Preferably, the interception layer in step 5) is created by adding a view View through Android's WindowManager and step 5) specifically includes:

[0028] 51) Monitor the onTouchEvent() event callback of the view View. The callback type is MotionEvent.ACTION_DOWN to get the x and y values ​​of the touch point from the upper left corner of the mobile phone screen when the annotator touches the mobile phone screen and presses it. The callback type is MotionEvent.ACTION_UP to get the x and y values ​​of the touch point from the upper left corner of the mobile phone screen when the annotator leaves the mobile phone screen;

[0029] 52) Store the x and y values ​​of the distance between the touch point and the upper left corner of the mobile phone screen when the annotator touches the screen and the x and y values ​​of the distance between the touch point and the upper left corner of the mobile phone screen when the annotator leaves the mobile phone screen, and perform simulation operations on the above two sets of x and y values ​​through the dispatchGesture() method of the AccessibilityServive to store the data generated during the operation.

[0030] Preferably, in step 5), the x, y values ​​of the touch point from the upper left corner of the mobile phone screen when the annotator touches the screen and the x, y values ​​of the touch point from the upper left corner of the mobile phone screen when the annotator leaves the mobile phone screen, as well as the data generated during the operation are stored in the trace.jsonl file.

[0031] In addition, the present invention also provides a training data acquisition system based on barrier-free service, which is characterized by comprising:

[0032] A task input module, which is used to input a training task and open a floating window of an annotation panel based on the input training task;

[0033] An operation selection module, which is used to select an operation type of a training task on the annotation panel floating window and reduce the annotation panel floating window;

[0034] The data acquisition module is used to obtain the page XML and page screenshots of the current interface based on the accessibility service;

[0035] An operation execution module is used for the annotation personnel to perform operations according to the operation type, and the interception layer obtains the data of the operation and performs simulation operations to obtain the data generated during the operation;

[0036] The data storage module is used to store page XML, page screenshots, and the data of the operation obtained by the interception layer and the data generated during the operation.

[0037] Moreover, the present invention also provides a training data acquisition device based on barrier-free service, characterized in that it includes:

[0038] one or more processors;

[0039] A memory for storing one or more programs;

[0040] When the one or more programs are executed by the one or more processors, the one or more processors implement the training data acquisition method based on accessibility service as described above.

[0041] Finally, the present invention provides a computer-readable storage medium having a computer program stored thereon, characterized in that when the program is executed by a processor, the steps of the training data acquisition method based on accessibility services as described above are implemented.

[0042] Compared with the prior art, the method, system, device and medium for obtaining training data based on accessibility services of the present invention have one or more of the following beneficial technical effects:

[0043] 1. The present invention facilitates manual collection of training data for machine learning.

[0044] 2. The present invention solves the problem that the manual sliding interface and the automatic sliding controls in the mobile phone app interface slide simultaneously, and the accessibility service AccessibilityService calls back the data of these two change events at the same time, making it difficult to distinguish the source of the data, thereby ensuring the accuracy of the collected training data.

[0045] 3. The present invention improves the convenience of obtaining training data.

[0046] 4. The present invention solves the problem of instability and randomness of manual operations by labelers, which leads to disordered data, ensures the efficiency of data acquisition and saves manpower. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 It is a flow chart of the method for acquiring training data based on barrier-free service of the present invention.

[0048] Figure 2 It is a flowchart of converting all information of the barrier-free service node information into a string in XML format in the present invention.

[0049] Figure 3 It is a schematic diagram of the structure of the training data acquisition system based on barrier-free service of the present invention. DETAILED DESCRIPTION

[0050] Before describing in detail any embodiment of the present invention, it should be understood that the present invention is not limited in its application to the construction and arrangement details of the components described below or illustrated in the following figures. The present invention can have other embodiments and can be practiced or carried out in various ways. In addition, it should be understood that the words and terms used here are for descriptive purposes and should not be considered restrictive. The use of "including" or "having" and its variations herein is intended to cover the items and their equivalents and additional items displayed below. Unless otherwise specified or limited, the terms "install", "connect", "support" and "couple" and their variations are widely used and cover direct installation and indirect installation, connection, support and connection. In addition, "connect" and "couple" are not limited to physical or mechanical connections or connections.

[0051] Furthermore, on the first aspect, in the disclosure of the present invention, the orientation or positional relationship indicated by terms such as "longitudinal", "transverse", "upper", "lower", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside" and "outside" are based on the orientation or positional relationship shown in the accompanying drawings, which are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operate in a specific orientation, and therefore the above terms cannot be understood as limitations on the present invention; on the second aspect, the term "one" should be understood as "at least one" or "one or more", that is, in one embodiment, the number of an element may be one, while in another embodiment, the number of the element may be multiple, and the term "one" cannot be understood as a limitation on the quantity.

[0052] Before introducing the specific content of the present invention, some technical terms involved in the present invention are briefly introduced to help those skilled in the art to better understand the present invention.

[0053] 1. Machine Learning: Machine Learning is a branch of the field of Artificial Intelligence (AI). Its core purpose is to enable computers to learn from data and make predictions or decisions based on the learned experience. In layman's terms, machine learning is to teach computers how to extract patterns and knowledge from data, and then use this knowledge to process new data and give answers or suggestions.

[0054] 2. Labeling: Labeling is the process of collecting data for machine learning and adding labels and classification information to data samples.

[0055] 3. AccessibilityService: It is a service provided by Android SDK to help special groups (people with visual, hearing or mobility impairments) use mobile phones more easily.

[0056] 4. Call api: It is the process of forwarding questions to the GLM-4 model based on the user's questions, and displaying the results of the GLM-4 model feedback to the user.

[0057] 5. XML: In Android development, XML is mainly used to define user interface structure and elements.

[0058] 6. Toast prompt: Toast is a small prompt box that pops up on the interface when using a mobile phone. It will display a short message and then disappear automatically. Its function is to provide users with some quick messages or feedback without interrupting the user's current operation.

[0059] Figure 1 FIG. 4 is a flowchart of a method for acquiring training data based on barrier-free service according to the present invention. Figure 1 As shown, the training data acquisition method based on barrier-free service of the present invention comprises the following steps:

[0060] 1. Determine the training tasks.

[0061] In the present invention, determining the training task includes:

[0062] 1. Determine the description of the training task.

[0063] The description of the training task is a simple description of the task to be performed in the mobile phone, so that technical personnel in this field can clearly understand the content or function of the training task, such as: "Make a video using Zhipu Qingyan".

[0064] 2. Determine the ID of the training task.

[0065] Each training task has an ID, which makes it easier to distinguish and retrieve the data of the training task.

[0066] 3. Determine the storage address of the labeled data file for the training task.

[0067] The storage address of the labeled data file of the training task can be a task document document, which is the file name of the data file generated by the labeling stored in the / Document / AiLabel / trace directory of the mobile phone, for example: zpqy. A new / Document / AiLabel / trace / zpqy folder will be created locally on the mobile phone. This is convenient for labelers to classify data of the same type of tasks on the mobile phone. The labeled data of the Zhipu Qingyan app are all placed in the zpqy directory.

[0068] In the present invention, the training task can be determined in the training task input module, and the task description, task ID and task document can be input in the training task input module. The training task input module is an Android activity page, which has three input boxes for inputting the information of the task to be annotated, that is, the task description, task ID and annotation data file storage directory can be input respectively.

[0069] 2. Open the annotation panel floating window.

[0070] After confirming the training task, you can open the annotation panel floating window. After entering the task description, task ID, and annotation data file storage directory, the start button on the page of the task input module is highlighted, and you can click the start button (the start button is an Android Button control. When the task information is not completed, by setting the Button's transparency alpha property, 0.2f shows an unclickable state, but it can be clicked. Clicking cannot start the task, and the annotator will be prompted to fill in the data. The task can only be started if it is filled correctly. When the annotator enters information in the input box, the content changes in the input box are monitored. When it is monitored that all three input boxes have data, the transparency of the start button is set, and 1f shows a clickable state) to open the annotation panel floating window.

[0071] The annotation panel floating window is a panel in the form of a floating window (created by adding a view through Android's WindowManager). It can be suspended on the upper layer of any interface of the mobile phone. When you open any app in the mobile phone, the annotation panel floating window will be on the upper layer of any app.

[0072] 3. Select the operation type of the training task in the annotation panel floating window and shrink the annotation panel floating window.

[0073] The annotation panel floating window is provided with operation types such as "click", "long press", "slide", "input", "Note", "INTERACT", "call GLM4-PAGE", "return" and the S button. The annotator determines the operation type of the first step operation according to the training task and selects the operation type on the annotation panel floating window, such as "slide".

[0074] After selecting the operation type, the annotation panel floating window will automatically shrink into a small sphere (by setting all buttons on the annotation panel floating window to invisible using view.setVisibility(View.GONE) and setting [<] iconsetVisibility(View.VISIBLE) to visible, the annotation panel floating window will naturally change its width and height due to the change in the size of the internal controls).

[0075] 4. Get the page XML and page screenshots of the current interface based on accessibility services.

[0076] In the present invention, the screenshot function takeScreenshot() method of the accessibility service AccessibilityService of Android is used to take a screenshot of the current interface of the mobile phone and call back the ScreenshotResult object to the caller. The screenshot is converted into a Bitmap object through the ScreenshotResult object, thereby obtaining a page screenshot.

[0077] At the same time, the getRootInActiveWindow() method of Android's accessibility service AccessibilityService returns the AccessibilityNodeInfo object of the current interface of the mobile phone. By parsing the properties of the AccessibilityNodeInfo object, all the information of the AccessibilityNodeInfo is converted into a string in XML format, thereby obtaining the page XML.

[0078] like Figure 2 As shown, in the present invention, converting all information of the accessibility service node information AccessibilityNodeInfo into a string in XML format includes the following steps:

[0079] 1. Parse the node Node in the accessibility service node information AccessibilityServiceNodeInfo, and convert the parsing result into a mutable character sequence StringBuilder.

[0080] In the present invention, the accessibility service node information AccessibilityServiceNodeInfo is passed into RecursiveTask (a subclass of ForkJoinPool, specifically used for recursive tasks with return values) to perform node Node parsing. RecursiveTask parses the passed-in node Node to obtain key fields, and then converts it into StringBuilder. In addition, when parsing the node Node, it is parsed from the root node RootNode first.

[0081] The RecursiveTask code logic is:

[0082] (1) Parameter passing. The parameters passed include: Node, which is the Node object to be parsed; index, which is the order of the current node Node in the parent node FatherNode. When assembling the results, it is assembled according to the index order to prevent the XML structure from being disordered; depth, which is the depth of the current node Node in the node tree NodeTree. When the format is indented, it is used to fill in the number of spaces.

[0083] (2) Method. The method is compute(), which is the core method of RecursiveTask. If the current task needs to be split, the task is split into more subtasks, and the fork() method is used to process these subtasks in parallel. After the parsing is completed, the join() method is called to obtain the result of the corresponding task. The node parsing logic is executed asynchronously in this method.

[0084] In the compute() method, the key fields of the current node Node are obtained first, and then the child node ChildNode of the current node Node is obtained. Then, according to the number of child nodes ChildNode, it is split into a fixed number of subtasks, and each child node ChildNode and the sequential index of the current node Node are passed to the new RecursiveTask, and fork() is called to execute the parsing task of each child node ChildNode, and the parsing result of the child node ChildNode is obtained through join().

[0085] In the present invention, the node Node in the accessibility service node information AccessibilityServiceNodeInfo is parsed, and the parsing result is converted into a variable character sequence StringBuilder, which specifically includes:

[0086] (1) Create an XmlSerializer object (the present invention uses the XmlSerializer object to ensure that the data complies with the XML specification), and set the StringWriter object and the startDocument attribute and the startTag attribute. The sample code for the specific settings is as follows:

[0087] valstringWriter=StringWriter()

[0088] valserializer=Xml.newSerializer()

[0089] serializer.setOutput(stringWriter)

[0090] serializer.startDocument("UTF-8",true)

[0091] serializer.startTag("","node").

[0092] (2) Obtain the key fields of the node Node in the accessibility service node information AccessibilityServiceNodeInfo.

[0093] The node itself carries some information fields, and the key fields of the node can be obtained directly through method calls. The following is an example of how the XmlSerializer object loads the key fields of the node:

[0094] serializer.attribute("","class",node.className.toString());

[0095] serializer.attribute("","text",node.text.toString()).

[0096] Among them, node.className can get the information of the className field of the node Node, and node.text can get the information of the text field of the node Node.

[0097] In the present invention, the key fields of the node Node include: className, contentDescription, text, viewIdResourceName, isClickable, isEnabled, isChecked, isFocused, isFocusable, isScrollable, isSelected, isLongClickable and boundsInScreen.

[0098] (3) Set the end tag endTag property and the end document endDocument property of the XmlSerialize object, insert the key field of the obtained node Node into the XmlSerializer object, and use the StringWriter object to convert it into a variable character sequence StringBuilder. The sample code for the specific settings is as follows:

[0099] serializer.endTag("","node")

[0100] serializer.endDocument()

[0101] valstringBuilder=StringBuilder()

[0102] stringBuilder.append(stringWriter.toString()).

[0103] Here, stringWriter is the StringWriter object set in step (1).

[0104] Thus, an exemplary StringBuilder is obtained as follows:

[0105] <? xml version='1.0'encoding='UTF-8'standalone='yes'? > <node NAF="true" index="0"text=""resource-id=""

[0106] class="android.view.ViewGroup" package="com.zhipu.agent"content-desc="" checkable="false" checked="false" clickable="false" enabledd="true" focusable="false" focused="false" scrollable="false"long-clickable="false" password="false" selected="false"

[0107] bounds="[0,148][1224,2776]" / >.

[0108] 2. Adjust the structure of the variable character sequence StringBuilder, and perform parallel parsing on the child nodes ChildNod of the corresponding node Node, which specifically includes:

[0109] (1) Remove the version, encoding, and standalone information of the XmlSerializer object in the mutable character sequence StringBuilder, and only keep the version, encoding, and standalone information of the outermost node Node to avoid duplication. The sample code is as follows:

[0110] StringBuilder.toString().replace("<?xml version='1.0'encoding='UTF-8'

[0111] standalone='yes'? >","").

[0112] (2) Determine whether the node Node corresponding to the variable character sequence StringBuilder contains a child node ChildNode. If the node Node corresponding to the variable character sequence StringBuilder contains a child node ChildNode, remove the end tag endTag of the node Node in the variable character sequence StringBuilder, otherwise a structural error will occur when assembling the child node ChildNode. The sample code is as follows:

[0113] StringBuilder.toString().replace(" / >",">").

[0114] (3) Obtain the key fields of each child node ChildNod in parallel and set a depth parameter for each child node ChildNod to record the depth of each child node ChildNod. Save the key fields and depth parameters of each child node ChildNod obtained into RecursiveTaskList.

[0115] The method for obtaining the key fields of each child node ChildNod is the same as the method for obtaining the key fields of the node Node in step 1, and will not be described for simplicity. However, the key fields of multiple child nodes ChildNod of a node Node can be obtained in parallel to improve parsing efficiency.

[0116] Of course, in order to facilitate subsequent assembly, the index of each child node ChildNod also needs to be saved in RecursiveTaskList.

[0117] 3. Obtain the parsing results of each child node ChildNod and assemble them into a variable character sequence StringBuilder, and adjust the structure of the assembled variable character sequence StringBuilder again to obtain a string in XML format, which specifically includes:

[0118] (1) Traverse the RecursiveTaskList and obtain the key fields and depth parameters of each child node ChildNod. Of course, if there are multiple child nodes ChildNod, it is also necessary to obtain the index of each child node ChildNod.

[0119] (2) The key fields of each child node ChildNod obtained are concatenated into a variable character sequence StringBuilder, and the number of indentation spaces is set according to the depth parameter to complete the format indentation. The sample code of the format indentation is as follows:

[0120] StringBuilder.append("\r\n").append("".repeat(depth)).

[0121] Among them, "\r\n" is a line break character, and depth is the depth of the current child node ChildNod.

[0122] The sample code for concatenating the key fields of each child node ChildNod obtained into a variable character sequence StringBuilder is as follows:

[0123] StringBuilder.append(childNodeStr).

[0124] Among them, childNodeStr is the result of parsing the child node ChildNode. If there are multiple child nodes ChildNode, childNodeStr needs to be assembled according to the index order to prevent the overall structure from being chaotic.

[0125] (3) After all the key fields of the child nodes ChildNod are concatenated into the variable character sequence StringBuilder, the end tag endTag of the variable character sequence StringBuilder is completed. The sample code is as follows:

[0126] StringBuilder.append("\r\n").append("".repeat(depth-1)).append("").

[0127] Thus, starting from the parsing of the root node RootNode, through the gradual parsing and splicing of its child node ChildNod and the child node ChildNod of the child node ChildNod, all the nodes Node in the accessibility service node information AccessibilityServiceNodeInfo of the page will be assembled into a StringBuilder to obtain a string in XML format.

[0128] 5. The labeling personnel perform operations according to the operation type, and the interception layer obtains the data of the operation and performs simulation operations to obtain the data generated during the operation.

[0129] The interception layer is created by adding a view View through Android's WindowManager, which intercepts the gestures of the annotator.

[0130] When the annotator selects operations such as "click", "long press", and "slide", the onTouchEvent() event callback of the interception layer view View is monitored. The callback type is MotionEvent.ACTION_DOWN, and the x and y values ​​of the distance between the touch point and the upper left corner of the mobile phone screen can be obtained when the annotator touches the mobile phone screen and presses it; the callback type is MotionEvent.ACTION_UP, and the x and y values ​​of the distance between the touch point and the upper left corner of the mobile phone screen can be obtained when the annotator leaves the mobile phone screen. The x and y values ​​of the distance between the touch point and the upper left corner of the mobile phone screen when the annotator touches the screen and presses it are stored, and the above two sets of x and y values ​​are simulated through the dispatchGesture() method of the AccessibilityServive to store the data generated during the operation.

[0131] In the present invention, the data of the gesture of the annotator intercepted by the interception layer is passed to the accessibility service AccessibilityService, which simulates the execution of the corresponding operation, thereby obtaining the specific node information NodeInfo triggered by the event through the onAccessibilityEvent() method of the accessibility service AccessibilityService. When annotating a "slide" event, since there may be an automatic scrolling control on the interface, if a scrolling event occurs at the same time, the accessibility service AccessibilityService will call back the data of the two slide events but cannot distinguish the sources. Through the area of ​​the slide event recorded by the interception layer, the corresponding slide event data can be found and stored in the multiple slide data called back by judging whether the control area is in the area of ​​the slide event.

[0132] 6. Store page XML, page screenshots, and the data of the operation obtained by the interception layer and the data generated during the operation.

[0133] In the present invention, the x and y values ​​of the touch point from the upper left corner of the mobile phone screen when the annotator touches the screen and the x and y values ​​of the touch point from the upper left corner of the mobile phone screen when the annotator leaves the mobile phone screen, as well as the data generated during the operation, can be stored in the trace.jsonl file.

[0134] The page XML and page screenshots can be stored in the specified directory of the mobile phone through the ContentResolver service, that is, under the document determined in step 1.

[0135] 7. Determine whether the training task has been completed. If yes, end the process; if not, return to step 3 and execute the next step of the training task.

[0136] Since a training task often has multiple steps, through the above steps 1 to 6, the training data of the first operation of the training task is obtained, and then, it can return to step 3 to obtain the training data of other operations of the training task until the training data is completed. Thus, the operation is executed step by step, and when each step is executed, the page screenshot and page XML and other information will be recorded and stored in trace.jsonl.

[0137] Figure 3 FIG. 2 shows a schematic diagram of the structure of the training data acquisition system based on barrier-free service of the present invention. Figure 3 As shown, the training data acquisition system based on barrier-free service of the present invention includes:

[0138] 1. Task input module.

[0139] The task input module is used to input a training task and open a floating window of the annotation panel based on the input training task;

[0140] 2. Operation selection module.

[0141] The operation selection module is used to select an operation type of a training task on the annotation panel floating window and shrink the annotation panel floating window.

[0142] 3. Data acquisition module.

[0143] The data acquisition module is used to obtain the page XML and page screenshots of the current interface based on the accessibility service.

[0144] 4. Operation execution module.

[0145] The operation execution module is used for the labeling personnel to perform operations according to the operation type, and the interception layer obtains the data of the operation and performs simulation operations to obtain the data generated during the operation.

[0146] 5. Data storage module.

[0147] The data storage module is used to store page XML, page screenshots, and the operation data obtained by the interception layer and the data generated during the operation.

[0148] In addition, the present invention also provides a training data acquisition device based on accessible services, which includes: one or more processors; a memory for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the training data acquisition method based on accessible services as described above.

[0149] Finally, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the training data acquisition method based on accessibility services as described above.

[0150] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention, rather than to limit the scope of protection of the present invention. Those skilled in the art can modify or replace the technical solution of the present invention according to the idea of ​​the present invention without departing from the essence and scope of the technical solution of the present invention.

Claims

1. A method for acquiring training data based on barrier-free service, characterized in that: The following steps are involved: 1) Determine the training tasks; 2) Open the annotation panel floating window; 3) Selecting the operation type of the training task on the annotation panel floating window and shrinking the annotation panel floating window; 4) Get the page XML and page screenshots of the current interface based on the accessibility service; 5) The annotator performs operations according to the operation type, and the interception layer obtains the data of the operation and performs simulation operations to obtain the data generated during the operation; 6) Storing page XML, page screenshots, and the data of the operation obtained by the interception layer and the data generated during the operation; 7) Determine whether the training task has been completed. If yes, end the process; if not, return to step 3) and perform the next step of the training task.

2. The method for acquiring training data based on barrier-free service according to claim 1, characterized in that: The step 1) comprises: 11) Determine the description of the training task; 12) Determine the ID of the training task; 23) Determine the storage address of the labeled data file for the training task.

3. The method for acquiring training data based on barrier-free service according to claim 2, characterized in that: In the step 3), shrinking the annotation panel floating window specifically includes: setting View.GONE of all buttons on the annotation panel floating window to be invisible and setting View.VISIBLE of < on the annotation panel floating window to be visible.

4. The method for acquiring training data based on barrier-free service according to claim 3, characterized in that: The step 4) specifically includes: 41) Take a screenshot of the current interface through the takeScreenshot() method of the Android Accessibility Service AccessibilityService and call back the ScreenshotResult object to the caller, and convert the screenshot into a Bitmap object through the ScreenshotResult object; 42) The getRootInActiveWindow() method of the Android accessibility service AccessibilityService returns the AccessibilityNodeInfo object of the current interface, and all the information of the AccessibilityNodeInfo is converted into a string in XML format by parsing the properties of the AccessibilityNodeInfo object.

5. The method for acquiring training data based on barrier-free service according to claim 4, characterized in that: The step 42) converts all the information of the accessibility service node information AccessibilityNodeInfo into a string in XML format, specifically including: 421) Parse the node Node in the accessibility service node information AccessibilityServiceNodeInfo, and convert the parsing result into a variable character sequence StringBuilder; 422) adjusting the structure of the variable character sequence StringBuilder, and performing parallel parsing on the child nodes ChildNod of the corresponding node Node; 423) Obtain the parsing results of each child node ChildNod and assemble them into a variable character sequence StringBuilder, and adjust the assembled variable character sequence StringBuilder again to obtain a string in XML format.

6. The method for acquiring training data based on barrier-free service according to any one of claims 1 to 5, characterized in that: The interception layer in step 5) is created by adding a view View through Android's WindowManager and step 5) specifically includes: 51) Monitor the onTouchEvent() event callback of the view View. The callback type is MotionEvent.ACTION_DOWN to get the x and y values ​​of the touch point from the upper left corner of the mobile phone screen when the annotator touches the mobile phone screen and presses it. The callback type is MotionEvent.ACTION_UP to get the x and y values ​​of the touch point from the upper left corner of the mobile phone screen when the annotator leaves the mobile phone screen; 52) Store the x and y values ​​of the distance between the touch point and the upper left corner of the mobile phone screen when the annotator touches the screen and the x and y values ​​of the distance between the touch point and the upper left corner of the mobile phone screen when the annotator leaves the mobile phone screen, and perform simulation operations on the above two sets of x and y values ​​through the dispatchGesture() method of the AccessibilityServive to store the data generated during the operation.

7. The method for acquiring training data based on barrier-free service according to claim 6, characterized in that: In step 5), the x and y values ​​of the distance between the touch point and the upper left corner of the mobile phone screen when the annotator touches the screen and the x and y values ​​of the distance between the touch point and the upper left corner of the mobile phone screen when the annotator leaves the mobile phone screen, as well as the data generated during the operation, are stored in the trace.jsonl file.

8. A training data acquisition system based on barrier-free service, characterized in that: include: A task input module, which is used to input a training task and open a floating window of an annotation panel based on the input training task; An operation selection module, which is used to select an operation type of a training task on the annotation panel floating window and reduce the annotation panel floating window; The data acquisition module is used to obtain the page XML and page screenshots of the current interface based on the accessibility service; An operation execution module is used for the annotation personnel to perform operations according to the operation type, and the interception layer obtains the data of the operation and performs simulation operations to obtain the data generated during the operation; The data storage module is used to store page XML, page screenshots, and the data of the operation obtained by the interception layer and the data generated during the operation.

9. A training data acquisition device based on barrier-free service, characterized in that: include: one or more processors; A memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the training data acquisition method based on accessibility service as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method for acquiring training data based on accessibility services as described in any one of claims 1 to 7 are implemented.

Images