Method for automatically operating Android smart phone

Abstract

The invention discloses a method for automatically operating an Android smart phone, which is characterized by comprising the following steps: S100: configuring and initializing a strategy file, including the following steps: S101: configuring the strategy file which comprises a uiautomator2 installation strategy, a mobile phone evidence obtaining installation strategy, a backup assistant installation strategy and an operation assistant strategy, the strategy comprises Android smart phone brand names and respective execution steps corresponding to the Android smart phone brand names; s102, initializing a running environment and loading the strategy file for configuring information and executing a strategy; s200, connecting the Android smart phone to a computer end; s300, the computer end starts an interaction process; s400, an auxiliary application for automatic operation is implanted into the Android smart phone terminal; s500, matching an automatic operation strategy in the strategy file according to the brand name of the Android smart phone; and S600, executing the automatic operation strategy to automatically operate the Android smart phone. According to the method, the acquisition time of the interface elements is optimized, and the strategy assembly efficiency is improved.

Description

Technical Field

[0001] The invention belongs to the field of mobile phone evidence collection and data recovery, and relates to a method for automatically operating a mobile phone, and in particular to a method for automatically operating an Android smart phone. Background Art

[0002] In the field of mobile phone forensics and data recovery, traditional methods usually rely on manual operations such as clicking on the interface to obtain the required data for forensics or data recovery. However, with the continuous enhancement of smartphone functions and the increase in data volume, manual operations have a series of problems: first, manual clicking may introduce misoperations, resulting in inaccurate or incomplete data; second, large-scale data extraction may be time-consuming and cumbersome, affecting the efficiency of forensics and data recovery; in addition, manual operations may be affected by subjective factors, resulting in irreproducible or inconsistent forensic results. In summary, the shortcomings and deficiencies of the existing technology are:

[0003] 1. Risk of misoperation: Manual operation is prone to misoperation, such as missed clicks and missed clicks, which may lead to inaccurate evidence collection results.

[0004] 2. High time cost: Large-scale data forensics requires a lot of manpower investment, which is time-consuming and inefficient.

[0005] 3. Subjective influence: Manual operation is affected by the operator’s personal experience and subjective factors, resulting in low reproducibility and consistency of evidence collection results.

[0006] 4. Not adaptable to complex scenarios: For complex mobile phone applications or interfaces, manual operations may not be able to accurately identify and operate them, making evidence collection more difficult.

[0007] 5. Human resource limitations: Relying on manual operations requires a large amount of human resource investment, which is costly and unstable, and it is difficult to meet the needs of large-scale data forensics. Summary of the invention

[0008] In view of the deficiencies of the prior art, the present invention provides a method for automatically operating an Android smartphone: the automatic click function of the smartphone is used in combination with a strategic control scheme to realize operations such as automatic clicks on the mobile phone interface. The method recognizes the positions of different elements of the mobile phone UI and formulates an execution path to realize automatic recognition and clicks on the mobile phone interface, thereby replacing the traditional manual operation method, and includes the following steps:

[0009] S100: Configuring a policy file and initializing it, including the following steps:

[0010] S101: Configure a policy file, the policy file includes uiautomator2 installation policy, mobile phone forensics installation policy, backup assistant installation policy, and operation assistant policy, the policy includes Android smartphone brand names and respective execution steps corresponding to each Android smartphone brand name; wherein,

[0011] The format of the policy file is json, and the Android smartphone brand names include oppo, vivo, Huawei, Samsung, Xiaomi, and redmi;

[0012] In the policy file, for mobile phones with unknown Android smartphone brand names, the default policies are used for operation, and the default policies include the default uiautomator2 installation policy, the default mobile phone forensics installation policy, the default backup assistant installation policy and the default operation assistant policy;

[0013] S102: Initializing the operating environment and loading the policy file to configure information and execute the policy;

[0014] S200: Connect Android smartphone to PC;

[0015] S300: The computer starts the interactive process;

[0016] S400: An auxiliary application for automatic operation is implanted in the Android smartphone, including the following steps:

[0017] S401: The computer uses an adb command to push the atx-agent installation package to the Android smartphone;

[0018] S402: The computer uses an adb command to install the atx-agent installation package of the Android smartphone;

[0019] S403: Determine whether the atx-agent installation package is successfully installed. If yes, execute step S404; otherwise, execute step S405;

[0020] S404: Use uiautomator2 to obtain the elements of the current interface and execute step S500;

[0021] S405: using uiautomator to obtain elements of the current interface;

[0022] S500: Automatically operate the policy in the matching policy file based on the brand name of the Android smartphone;

[0023] S600: Execute an automatic operation strategy to automatically click and operate the Android smartphone.

[0024] Preferably, step S200 includes the following steps:

[0025] S201: Use a USB data cable to connect the Android smartphone to the computer;

[0026] S202: The Android smartphone selects to enable the MTP data transmission mode;

[0027] S203: The Android smartphone enables developer options and USB debugging mode, including the following steps:

[0028] S2031: Enable developer mode in Developer Options;

[0029] S2032: Enable USB debugging mode in Developer Options;

[0030] S2033: Enable USB simulated click function in developer options.

[0031] Preferably, step S500 includes the following steps:

[0032] S501: The computer uses adb commands to obtain information about Android smartphones, including brand name and model;

[0033] S502: Match the policy file according to the acquired Android smartphone information: determine whether the brand name of the current Android smartphone exists in the policy file, if yes, execute step S503, otherwise, execute step S504;

[0034] S503: Select the policy corresponding to the current Android smartphone brand name for automatic operation, and execute step S600;

[0035] S504: Select a default strategy for automatic operation.

[0036] Preferably, step S600 includes the following steps:

[0037] S601: According to the automatic operation strategy in the matched strategy file, the execution steps corresponding to the current strategy are obtained and recorded as an execution step list;

[0038] S602: traverse the execution step list to determine whether the elements of the current interface match the steps in the execution step list. If yes, execute step S603; otherwise, execute step S604;

[0039] S603: According to the matched execution steps, the feature name, feature value offset, element type, and interface coordinates are matched in the current interface element and the click position of the current interface element is obtained. The computer uses an adb command to complete the click of the Android smartphone, and then executes step S606;

[0040] S604: Determine whether interactive text needs to be input, if yes, execute step S605, otherwise, execute step S606;

[0041] S605: The computer terminal uses adb input commands to complete the input of interactive text, and then executes step S606;

[0042] S606: Determine whether the element name of the current interface contains an end mark, if so, end the process, otherwise, execute step S602.

[0043] Preferably, step S603 includes the following steps:

[0044] S6031: Determine whether to click on the feature name, if yes, execute step S6032, otherwise, execute step S6033;

[0045] S6032: Obtain the click position according to the feature name, and execute step S6038;

[0046] S6033: Determine whether the click is based on the feature value offset. If yes, execute step S6034; otherwise, execute step S6035.

[0047] S6034: Obtain the click position according to the feature value offset, and execute step S6038;

[0048] S6035: Determine whether to click by element type, if yes, execute step S6036, otherwise, execute step S6037;

[0049] S6036: Obtain the click position according to the element type and execute step S6038;

[0050] S6037: Determine whether the click is based on the interface coordinates. If yes, execute step S6038; otherwise, execute step S604.

[0051] S6038: The computer uses an adb click command to complete the click on the Android smartphone, and executes step S606.

[0052] Preferably, the steps of installing the uiautomator2 strategy include:

[0053] A1: Default uiautomator2 installation strategy, including:

[0054] A11: Set the phone screen resolution to 1080*2160;

[0055] A12: The name of the matching mobile phone interface element is ATX interface, click to continue installation / complete;

[0056] A13: The name of the matching mobile interface element is the automator interface, and click to continue the installation / complete;

[0057] A14: The matching phone interface is the interface for entering the lock screen password. Enter the lock screen password.

[0058] A2: The uiautomator2 installation strategy for OPPO phones includes:

[0059] A21: Configure the phone screen resolution to 1080*2400;

[0060] A22: Determine whether a prompt for ignoring risk installation pops up. If so, execute the ignoring risk installation. Otherwise, execute the application installation protection interface provided by Smart Shield to continue the installation.

[0061] A3: The installation strategy of uiautomator2 for Huawei phones includes:

[0062] A31: A warning interface matching the name of a mobile phone interface element, which is clicked to install, wherein the warning interface includes: the application from unknown sources is likely to damage your device and personal data or you have learned that this application has not been tested;

[0063] A4: The UIAutomator2 installation strategy for vivo phones includes:

[0064] A41: Configure the phone screen resolution to 1080*2160;

[0065] A42: Cycle through the prompted operations in the installation interface elements, including: Continue Installation, Complete, Install;

[0066] A5: The UIAutomator2 installation strategy for Xiaomi phones includes:

[0067] A51: Configure the phone screen resolution to 1080*2160;

[0068] A52: Match the prompts in the installation interface elements, including the USB installation prompt, and click the install button to continue the installation.

[0069] Preferably, the steps of installing the mobile phone evidence collection strategy include:

[0070] B1: Default mobile phone forensics installation strategy, including:

[0071] B11: Configure the mobile phone screen resolution to 1080*2400;

[0072] B12: Match the names of the mobile phone interface elements: Always Allow, Continue Installation, Confirm, and click them respectively;

[0073] B13: Match the screen lock password input interface on the mobile phone interface and enter the screen lock password;

[0074] B2: Vivo mobile phone forensics installation strategy, including:

[0075] B21: Configure the mobile phone screen resolution to 1080*2400;

[0076] B22: Pause for 500 milliseconds and continue installation;

[0077] B3: OPPO mobile phone forensic installation strategy, including:

[0078] B31: Configure the mobile phone screen resolution to 1080*2400;

[0079] B32: Locate the click position of the installation button for mobile phone forensics: the installation button is offset 2020 pixels in the positive direction of the Y axis, and click to install.

[0080] Preferably, the step of installing the backup assistant strategy includes:

[0081] C1: Default Backup Assistant installation policy, including:

[0082] C11: Configure the mobile phone screen resolution to 1080*2160;

[0083] C12: Match the names of the mobile phone interface elements: Always Allow, Continue Installation, Confirm, and click them respectively;

[0084] C13: Match the screen lock password input interface on the mobile phone interface and enter the screen lock password;

[0085] C2: vivo mobile phone backup assistant installation strategy, including:

[0086] C21: Configure the phone screen resolution to 1080*2160;

[0087] C22: The name of the matching mobile phone interface element is "transfer", and click to install;

[0088] C3: OnePlus Mobile Backup Assistant installation strategy, including:

[0089] C31: Configure the phone screen resolution to 1240*2772;

[0090] C32: The type of the matched mobile phone interface element is com.android.packageinstaller:id / confirm_bottom_button_layout. The button element is found according to the element type and clicked to install.

[0091] Preferably, the assistants include OPPO Moving, Huawei Assistant, Xiaomi Assistant, vivo Transfer, and Samsung Switching Assistant, and the steps of running the assistant strategy include:

[0092] D1: Default run assistant strategy, including:

[0093] D11: Configure the mobile phone screen resolution to 1080*2400;

[0094] D12: Match the names of mobile interface elements to one or more of the following keywords and click them to continue the installation:

[0095] Next, Always allow, Allow, Always allow, Allow all installations, OK, Confirm, Always agree, Install, Continue installation, Agree and continue, Agree, Allow once, OK, Allow only while in use, Allow only while in use, Allow only when the app is in the foreground, Allow this time, Back up my data, I fully understand the risk, Continue installation, OK, Allow while using this app, Allow only while using this app;

[0096] D2: OnePlus phones run assistant strategies, including:

[0097] D21: Configure the mobile phone screen resolution to 1240*2772;

[0098] D22: Match the permission names of the mobile interface elements and click them respectively;

[0099] D23: When the name of the matching mobile phone interface element is Connect Now, click to install;

[0100] D3: Samsung mobile phone running assistant strategy, including:

[0101] D31: Configure the mobile phone screen resolution to 1080*2400;

[0102] D32: Match the permission names of the mobile phone interface elements in Samsung Switch Assistant and click them respectively;

[0103] D33: Click ApkDataMove to install;

[0104] D4: OPPO mobile phone operation assistant strategy, including:

[0105] D41: Configure the mobile phone screen resolution to 1080*2400;

[0106] D42: Match the permission names of the mobile phone interface elements in OPPO Moving Assistant and click them respectively;

[0107] D43: When the name of the matching mobile phone interface element is Connect Now, click to install;

[0108] D5: vivo mobile phone operation assistant strategy, including:

[0109] D51: Configure the phone screen resolution to 1080*2160;

[0110] D52: Match the permission names of the mobile phone interface elements in the vivo transfer assistant and click them respectively;

[0111] D53: When the name of the matching mobile phone interface element is backup, click to install;

[0112] D6: Xiaomi mobile phone operation assistant strategy, including:

[0113] D61: Configure the phone screen resolution to 1080*2400;

[0114] D62: Match the permission names of the mobile phone interface elements in Xiaomi Assistant and click them to install them respectively.

[0115] The beneficial effects of the present invention are: when manually performing operations such as clicking on the interface of an Android smartphone, the acquisition time of interface elements is optimized and the efficiency of strategy assembly is improved, thereby improving the accuracy, efficiency and reliability of data forensics. Specifically,

[0116] 1. Accuracy: Manual execution may have the risk of misoperation, such as accidentally touching the screen or clicking the wrong button. However, this application is based on precise program code execution to ensure the accuracy and consistency of the operation.

[0117] 2. Recognition accuracy: Based on the recognition and positioning of UI elements, this application can accurately find the target element and perform the corresponding operation. Compared with manual operation, this can avoid erroneous operations caused by misjudgment of UI elements.

[0118] 3. Flexibility: This application can operate according to specific strategies and conditions, for example, executing click actions according to specific UI element attributes or conditions, thereby achieving a more flexible and intelligent operation method. BRIEF DESCRIPTION OF THE DRAWINGS

[0119] Figure 1 is a flow chart of the present invention;

[0120] Figure 2 A flowchart of executing an automatic operation strategy to automatically click and operate an Android smartphone in an embodiment of the present invention;

[0121] Figure 3 This is a flow chart of executing an automatic operation strategy based on interface elements in an embodiment of the present invention. DETAILED DESCRIPTION

[0122] The following terms may be used in this application:

[0123] ATX (AutomatorX) is an open source automated testing tool that supports testing native applications, games, and web applications on iOS and Android platforms. Use Python to write test cases, and mix image recognition and control positioning technology to complete game automation. Attach a dedicated IDE to quickly write scripts

[0124] atx-agent: daemon, atx-agent is an agent program running on the mobile phone, which can perform mobile phone testing through the network

[0125] Minicap: is a tool for viewing the screen of an Android device in real time on a computer. It is a component of the open source project Smartphone Test Farm Jon (STF Jon), developed by the STF team. Minicap runs on the Android device and is responsible for capturing the device's screen video and sending it to the receiving end on the computer in real time through the Socket interface. The Minicap tool is divided into two parts: a dynamic link library (.so file) and a minicap executable file. It is intended to be used as a component in a larger program, such as in STF for remote control of Android devices.

[0126] JSON (JavaScript Object Notation) is a lightweight data exchange format. It is based on a subset of ECMAScript (the js specification developed by the European Computer Manufacturers Association) and uses a text format that is completely independent of programming languages ​​to store and represent data. The simplicity and clear hierarchical structure make JSON an ideal data exchange language. It is easy for people to read and write, and it is also easy for machines to parse and generate, and it effectively improves network transmission efficiency.

[0127] MTP (Media Transfer Protocol) data transmission mode.

[0128] ADB (Android Debug Bridge) Android Debug Bridge

[0129] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0130] This embodiment uses the components uiautomator2 and atx-agent to achieve the following beneficial effects:

[0131] 1. Optimize element acquisition time: Using uiautomator2 and atx-agent components, a more efficient screenshot solution is provided through a unified HTTP interface (GET / screenshot). This solution shields the differences between different Android phones and optimizes the element acquisition time. Compared with the traditional method, it reduces the steps of judging whether minicap is installed and available, minicap screenshots, and using the uiautomator2 interface to take screenshots, simplifies the operation process, improves the screenshot efficiency, and thus speeds up the acquisition of interface elements.

[0132] 2. Improve the efficiency of policy assembly: By introducing the atx-agent component, unified processing of different forms of Android phones is achieved. Whether using minicap screenshots, uiautomator2 interface screenshots, or screencap screenshots, atx-agent can help process and provide a unified HTTP interface, simplifying the policy assembly process and improving development efficiency and flexibility.

[0133] like Figure 1 As shown, the method of the present invention comprises the following steps:

[0134] S100: Configuring a policy file and initializing it, including the following steps:

[0135] S101: Configure a policy file, which includes uiautomator2 installation policy, mobile phone forensics installation policy, backup assistant installation policy, and operation assistant policy. The policy includes the brand name of Android smartphones and the execution steps corresponding to each Android smartphone brand name;

[0136] The format of the policy file is json. The Android smartphone brand names include oppo, vivo, Huawei, Samsung, Xiaomi, and Redmi.

[0137] In the policy file, for mobile phones with unknown Android smartphone brand names, the default policies are used for operation. The default policies include the default uiautomator2 installation policy, the default mobile phone forensics installation policy, the default backup assistant installation policy, and the default operation assistant policy.

[0138] The steps of uiautomator2 installation strategy include:

[0139] A1: Default uiautomator2 installation strategy, including:

[0140] A11: Set the phone screen resolution to 1080*2160;

[0141] A12: The name of the matching mobile phone interface element is ATX interface, click to continue installation / complete;

[0142] A13: The name of the matching mobile phone interface element is the automator interface. Click to continue the installation and complete.

[0143] A14: The matching phone interface is the interface for entering the lock screen password. Enter the lock screen password.

[0144] In this implementation, the default uiautomator2 installation strategy is as follows, where the Strategy node defines different strategies, the Models node defines different mobile phone brands (when Models is Default, it represents mobile phones with unknown Android smartphone brand names), and the Steps node defines different execution steps:

[0145] "Strategys":{

[0146] "InstallAutomator":{

[0147]

[0148]

[0149] For other known Android smartphone brands, their uiautomator2 installation strategies are similar. Due to space limitations, the json formatted strategy file in the embodiment is no longer cited for example.

[0150] A2: The uiautomator2 installation strategy for OPPO phones includes:

[0151] A21: Configure the phone screen resolution to 1080*2400;

[0152] A22: Determine whether a prompt for ignoring risk installation pops up. If so, execute the ignoring risk installation. Otherwise, execute the application installation protection interface provided by Smart Shield to continue the installation.

[0153] A3: The installation strategy of uiautomator2 for Huawei phones includes:

[0154] A31: A warning interface matching the name of a mobile phone interface element, which is clicked to install, wherein the warning interface includes: the application from unknown sources is likely to damage your device and personal data or you have learned that this application has not been tested;

[0155] A4: The UIAutomator2 installation strategy for vivo phones includes:

[0156] A41: Configure the phone screen resolution to 1080*2160;

[0157] A42: Cycle through the prompted operations in the installation interface elements, including: Continue Installation, Complete, Install;

[0158] A5: The UIAutomator2 installation strategy for Xiaomi phones includes:

[0159] A51: Configure the phone screen resolution to 1080*2160;

[0160] A52: Match the prompts in the installation interface elements, including the USB installation prompt, and click the install button to continue the installation.

[0161] The steps of the mobile forensics installation strategy include:

[0162] B1: Default mobile phone forensics installation strategy, including:

[0163] B11: Configure the mobile phone screen resolution to 1080*2400;

[0164] B12: Match the names of the mobile phone interface elements: Always Allow, Continue Installation, Confirm, and click them respectively;

[0165] B13: Match the screen lock password input interface on the mobile phone interface and enter the screen lock password;

[0166] B2: Vivo mobile phone forensics installation strategy, including:

[0167] B21: Configure the mobile phone screen resolution to 1080*2400;

[0168] B22: Pause for 500 milliseconds and continue installation;

[0169] B3: OPPO mobile phone forensic installation strategy, including:

[0170] B31: Configure the mobile phone screen resolution to 1080*2400;

[0171] B32: Locate the click position of the installation button for mobile phone forensics: the installation button is offset 2020 pixels in the positive direction of the Y axis, and click to install.

[0172] In this implementation, the OPPO mobile phone forensics installation strategy is as follows, where SPFSocket is the name of the mobile phone forensics software package. It should be understood that the forensics software package in this application is not limited to the forensics software package in this embodiment, and can also be other forensics software packages.

[0173]

[0174]

[0175]

[0176] For other known Android smartphone brands, the mobile phone forensics installation strategy is similar. Due to space limitations, the json format policy file in the embodiment is no longer cited for example.

[0177] The steps for installing the Backup Assistant strategy include:

[0178] C1: Default Backup Assistant installation policy, including:

[0179] C11: Configure the mobile phone screen resolution to 1080*2160;

[0180] C12: Match the names of the mobile phone interface elements: Always Allow, Continue Installation, Confirm, and click them respectively;

[0181] C13: Match the screen lock password input interface on the mobile phone interface and enter the screen lock password;

[0182] C2: vivo mobile phone backup assistant installation strategy, including:

[0183] C21: Configure the phone screen resolution to 1080*2160;

[0184] C22: The name of the matching mobile phone interface element is "transfer", and click to install;

[0185] In this implementation, the installation strategy of vivo mobile phone backup assistant is as follows:

[0186]

[0187]

[0188] For other known Android smartphone brands, the backup assistant installation strategy is similar. Due to space limitations, the json format policy file in the embodiment is no longer cited for example.

[0189] C3: OnePlus Mobile Backup Assistant installation strategy, including:

[0190] C31: Configure the phone screen resolution to 1240*2772;

[0191] C32: The type of the matched mobile phone interface element is com.android.packageinstaller:id / confirm_bottom_button_layout. The button element is found according to the element type and clicked to install.

[0192] The assistants include OPPO Moving, Huawei Assistant, Xiaomi Assistant, vivo Transfer, and Samsung Switching Assistant. The steps to run the assistant strategy include:

[0193] D1: Default run assistant strategy, including:

[0194] D11: Configure the mobile phone screen resolution to 1080*2400;

[0195] D12: Match the names of mobile interface elements to one or more of the following keywords and click them to continue the installation:

[0196] Next, Always allow, Allow, Always allow, Allow all installations, OK, Confirm, Always agree, Install, Continue installation, Agree and continue, Agree, Allow once, OK, Allow only while in use, Allow only while in use, Allow only when the app is in the foreground, Allow this time, Back up my data, I fully understand the risk, Continue installation, OK, Allow while using this app, Allow only while using this app;

[0197] D2: OnePlus phones run assistant strategies, including:

[0198] D21: Configure the mobile phone screen resolution to 1240*2772;

[0199] D22: Match the permission names of the mobile interface elements and click them respectively;

[0200] D23: When the name of the matching mobile phone interface element is Connect Now, click to install;

[0201] D3: Samsung mobile phone running assistant strategy, including:

[0202] D31: Configure the mobile phone screen resolution to 1080*2400;

[0203] D32: Match the permission names of the mobile phone interface elements in Samsung Switch Assistant and click them respectively;

[0204] D33: Click ApkDataMove to install;

[0205] In this implementation, the installation strategy of Samsung Mobile Phone Switch Assistant is as follows:

[0206]

[0207]

[0208]

[0209]

[0210]

[0211] For other mobile phones with known Android smartphone brand names, their mobile assistant installation strategies are similar. Due to space limitations, the json format policy file in the embodiment is no longer cited for example.

[0212] D4: OPPO mobile phone operation assistant strategy, including:

[0213] D41: Configure the mobile phone screen resolution to 1080*2400;

[0214] D42: Match the permission names of the mobile phone interface elements in OPPO Moving Assistant and click them respectively;

[0215] D43: When the name of the matching mobile phone interface element is Connect Now, click to install;

[0216] D5: vivo mobile phone operation assistant strategy, including:

[0217] D51: Configure the phone screen resolution to 1080*2160;

[0218] D52: Match the permission names of the mobile phone interface elements in the vivo transfer assistant and click them respectively;

[0219] D53: When the name of the matching mobile phone interface element is backup, click to install;

[0220] D6: Xiaomi mobile phone operation assistant strategy, including:

[0221] D61: Configure the phone screen resolution to 1080*2400;

[0222] D62: Match the permission names of the mobile phone interface elements in Xiaomi Assistant and click them to install them respectively.

[0223] S102: Initialize the operating environment and load the policy file to configure information and execute the policy;

[0224] S200: Connect Android smartphone to PC;

[0225] Step S200 includes the following steps:

[0226] S201: Use a USB data cable to connect the Android smartphone to the computer;

[0227] S202: The Android smartphone selects to enable the MTP data transmission mode;

[0228] S203: The Android smartphone enables developer options and USB debugging mode, including the following steps:

[0229] S2031: Enable developer mode in Developer Options;

[0230] S2032: Enable USB debugging mode in Developer Options;

[0231] S2033: Enable USB simulated click function in developer options.

[0232] S300: The computer starts the interactive process;

[0233] S400: An auxiliary application for automatic operation is implanted in the Android smartphone, including the following steps:

[0234] S401: The computer uses an adb command to push the atx-agent installation package to the Android smartphone;

[0235] S402: The computer uses an adb command to install the atx-agent installation package of the Android smartphone;

[0236] S403: Determine whether the atx-agent installation package is successfully installed. If yes, execute step S404; otherwise, execute step S405;

[0237] S404: Use uiautomator2 to obtain the elements of the current interface and execute step S500;

[0238] S405: using uiautomator to obtain elements of the current interface;

[0239] S500: Automatically operate the policy in the matching policy file based on the brand name of the Android smartphone;

[0240] Step S500 includes the following steps:

[0241] S501: The computer uses adb commands to obtain information about Android smartphones, including brand name and model;

[0242] S502: Matching the policy file according to the acquired Android smartphone information: determining whether the brand name of the current Android smartphone exists in the policy file, if yes, executing step S503, otherwise, executing step S504;

[0243] S503: Select the policy corresponding to the current Android smartphone brand name for automatic operation, and execute step S600;

[0244] S504: Select a default strategy for automatic operation.

[0245] S600: Execute an automatic operation strategy to automatically operate the Android smartphone.

[0246] Figure 2 A flowchart of executing an automatic operation strategy to automatically click and operate an Android smartphone in an embodiment of the present invention is shown. Figure 2 As shown, step S600 includes the following steps:

[0247] S601: According to the automatic operation strategy in the matched strategy file, the execution steps corresponding to the current strategy are obtained and recorded as an execution step list;

[0248] S602: traverse the execution step list to determine whether the elements of the current interface match the steps in the execution step list. If yes, execute step S603; otherwise, execute step S604;

[0249] S603: According to the matched execution steps, the feature name, feature value offset, element type, and interface coordinates are matched in the current interface element and the click position of the current interface element is obtained. The computer uses an adb command to complete the click of the Android smartphone, and then executes step S606;

[0250] Figure 3 A flow chart of executing an automatic operation strategy according to interface elements in an embodiment of the present invention is shown.

[0251] like Figure 3 As shown, step S603 includes the following steps:

[0252] S6031: Determine whether to click on the feature name, if yes, execute step S6032, otherwise, execute step S6033;

[0253] S6032: Obtain the click position according to the feature name, and execute step S6038;

[0254] S6033: Determine whether the click is based on the feature value offset. If yes, execute step S6034; otherwise, execute step S6035.

[0255] S6034: Obtain the click position according to the feature value offset, and execute step S6038;

[0256] S6035: Determine whether to click by element type, if yes, execute step S6036, otherwise, execute step S6037;

[0257] S6036: Obtain the click position according to the element type and execute step S6038;

[0258] S6037: Determine whether the click is based on the interface coordinates. If yes, execute step S6038; otherwise, execute step S604.

[0259] S6038: The computer uses an adb click command to complete the click on the Android smartphone, and executes step S606.

[0260] S604: Determine whether interactive text needs to be input, if yes, execute step S605, otherwise, execute step S606;

[0261] S605: The computer terminal uses adb input commands to complete the input of interactive text, and executes step S606;

[0262] S606: Determine whether the element name of the current interface contains an end mark. If so, end the process; otherwise, execute step S602.

[0263] In the json file format of the embodiments of the above-mentioned various strategies, QuitKeys is an end mark.

[0264] The method provided by the present invention solves the technical problem that there is no method for automatically operating an Android smartphone in the prior art.

[0265] It should be understood that the present invention is not limited to the above examples. For those skilled in the art, improvements or changes can be made based on the above description. All these improvements and changes should fall within the scope of protection of the claims attached to the present invention.

Claims

1. A method for automatically operating an Android smartphone, characterized in that The following steps are involved: S100: Configuring a policy file and initializing it, including the following steps: S101: Configure a policy file, the policy file includes uiautomator2 installation policy, mobile phone forensics installation policy, backup assistant installation policy, and operation assistant policy, the policy includes Android smartphone brand names and respective execution steps corresponding to each Android smartphone brand name; wherein, The format of the policy file is json, and the Android smartphone brand names include oppo, vivo, Huawei, Samsung, Xiaomi, and redmi; In the policy file, for mobile phones with unknown Android smartphone brand names, the default policies are used for operation, and the default policies include the default uiautomator2 installation policy, the default mobile phone forensics installation policy, the default backup assistant installation policy and the default operation assistant policy; S102: Initializing the operating environment and loading the policy file to configure information and execute the policy; S200: Connect Android smartphone to PC; S300: The computer starts the interactive process; S400: An auxiliary application for automatic operation is implanted in the Android smartphone, including the following steps: S401: The computer uses an adb command to push the atx-agent installation package to the Android smartphone; S402: The computer uses an adb command to install the atx-agent installation package of the Android smartphone; S403: Determine whether the atx-agent installation package is successfully installed. If yes, execute step S404; otherwise, execute step S405; S404: Use uiautomator2 to obtain the elements of the current interface and execute step S500; S405: Using uiautomator to obtain elements of the current interface; S500: Automatically operate policies in matching policy files based on the brand name of the Android smartphone; S600: Execute an automatic operation strategy to automatically operate the Android smartphone.

2. A method for automatically operating an Android smartphone according to claim 1, characterized in that: Step S200 includes the following steps: S201: Use a USB data cable to connect the Android smartphone to the computer; S202: The Android smartphone selects to enable the MTP data transmission mode; S203: The Android smartphone enables developer options and USB debugging mode, including the following steps: S2031: Enable developer mode in Developer Options; S2032: Enable USB debugging mode in Developer Options; S2033: Enable USB simulated click function in developer options.

3. A method for automatically operating an Android smartphone according to claim 1, characterized in that: Step S500 includes the following steps: S501: The computer uses adb commands to obtain information about Android smartphones, including brand name and model; S502: Match the policy file according to the acquired Android smartphone information: determine whether the brand name of the current Android smartphone exists in the policy file, if yes, execute step S503, otherwise, execute step S504; S503: Select the policy corresponding to the current Android smartphone brand name for automatic operation, and execute step S600; S504: Select a default strategy for automatic operation.

4. A method for automatically operating an Android smartphone according to claim 1, characterized in that: Step S600 includes the following steps: S601: According to the automatic operation strategy in the matched strategy file, the execution steps corresponding to the current strategy are obtained and recorded as an execution step list; S602: traverse the execution step list to determine whether the elements of the current interface match the steps in the execution step list. If yes, execute step S603; otherwise, execute step S604; S603: According to the matched execution steps, the feature name, feature value offset, element type, and interface coordinates are matched in the current interface element and the click position of the current interface element is obtained. The computer uses an adb command to complete the click of the Android smartphone, and then executes step S606; S604: Determine whether interactive text needs to be input, if yes, execute step S605, otherwise, execute step S606; S605: The computer terminal uses adb input commands to complete the input of interactive text, and executes step S606; S606: Determine whether the element name of the current interface contains an end mark. If so, end the process; otherwise, execute step S602.

5. A method for automatically operating an Android smartphone according to claim 4, characterized in that: Step S603 includes the following steps: S6031: Determine whether to click on the feature name, if yes, execute step S6032, otherwise, execute step S6033; S6032: Obtain the click position according to the feature name, and execute step S6038; S6033: Determine whether the click is based on the feature value offset. If yes, execute step S6034; otherwise, execute step S6035. S6034: Obtain the click position according to the feature value offset, and execute step S6038; S6035: Determine whether to click by element type, if yes, execute step S6036, otherwise, execute step S6037; S6036: Obtain the click position according to the element type and execute step S6038; S6037: Determine whether the click is based on the interface coordinates. If yes, execute step S6038; otherwise, execute step S604. S6038: The computer uses an adb click command to complete the click on the Android smartphone, and executes step S606.

6. A method for automatically operating an Android smartphone according to claim 1, characterized in that: The steps of the uiautomator2 installation strategy include: A1: Default uiautomator2 installation strategy, including: A11: Set the phone screen resolution to 1080*2160; A12: The name of the matching mobile phone interface element is ATX interface, click to continue installation / complete; A13: The name of the matching mobile interface element is the automator interface, and click to continue the installation / complete; A14: The matching phone interface is the interface for entering the lock screen password. Enter the lock screen password. A2: The uiautomator2 installation strategy for OPPO phones includes: A21: Configure the phone screen resolution to 1080*2400; A22: Determine whether a prompt for ignoring risk installation pops up. If so, execute the ignoring risk installation. Otherwise, execute the application installation protection interface provided by Smart Shield to continue the installation. A3: The installation strategy of uiautomator2 for Huawei phones includes: A31: A warning interface matching the name of a mobile phone interface element, which is clicked to install, wherein the warning interface includes: the application from unknown sources is likely to damage your device and personal data or you have learned that this application has not been tested; A4: The UIAutomator2 installation strategy for vivo phones includes: A41: Configure the phone screen resolution to 1080*2160; A42: Cycle through the prompted operations in the installation interface elements, including: Continue Installation, Complete, Install; A5: The UIAutomator2 installation strategy for Xiaomi phones includes: A51: Configure the phone screen resolution to 1080*2160; A52: Match the prompts in the installation interface elements, including the USB installation prompt, and click the install button to continue the installation.

7. A method for automatically operating an Android smartphone according to claim 1, characterized in that: The steps of the mobile phone forensics installation strategy include: B1: Default mobile phone forensics installation strategy, including: B11: Configure the mobile phone screen resolution to 1080*2400; B12: Match the names of the mobile phone interface elements: Always Allow, Continue Installation, Confirm, and click them respectively; B13: Match the screen lock password input interface on the mobile phone interface and enter the screen lock password; B2: Vivo mobile phone forensics installation strategy, including: B21: Configure the mobile phone screen resolution to 1080*2400; B22: Pause for 500 milliseconds and continue installation; B3: OPPO mobile phone forensic installation strategy, including: B31: Configure the mobile phone screen resolution to 1080*2400; B32: Locate the click position of the installation button for mobile phone forensics: the installation button is offset 2020 pixels in the positive direction of the Y axis, and click to install.

8. A method for automatically operating an Android smartphone according to claim 1, characterized in that: The steps of installing the backup assistant strategy include: C1: Default Backup Assistant installation policy, including: C11: Configure the mobile phone screen resolution to 1080*2160; C12: Match the names of the mobile phone interface elements: Always Allow, Continue Installation, Confirm, and click them respectively; C13: Match the screen lock password input interface on the mobile phone interface and enter the screen lock password; C2: vivo mobile phone backup assistant installation strategy, including: C21: Configure the phone screen resolution to 1080*2160; C22: The name of the matching mobile phone interface element is "transfer", and click to install; C3: OnePlus Mobile Backup Assistant installation strategy, including: C31: Configure the phone screen resolution to 1240*2772; C32: The type of the matched mobile phone interface element is com.android.packageinstaller:id / confirm_bottom_button_layout. The button element is found according to the element type and clicked to install.

9. A method for automatically operating an Android smartphone according to claim 1, characterized in that: in, The assistants include OPPO Moving, Huawei Assistant, Xiaomi Assistant, vivo Transfer, and Samsung Switching Assistant. The steps of running the assistant strategy include: D1: Default run assistant strategy, including: D11: Configure the mobile phone screen resolution to 1080*2400; D12: Match the names of mobile interface elements to one or more of the following keywords and click them to continue the installation: Next, Always allow, Allow, Always allow, Allow all installations, OK, Confirm, Always agree, Install, Continue installation, Agree and continue, Agree, Allow once, OK, Allow only while in use, Allow only while in use, Allow only when the app is in the foreground, Allow this time, Back up my data, I fully understand the risk, Continue installation, OK, Allow while using this app, Allow only while using this app; D2: OnePlus phones run assistant strategies, including: D21: Configure the mobile phone screen resolution to 1240*2772; D22: Match the permission names of the mobile interface elements and click them respectively; D23: When the name of the matching mobile phone interface element is Connect Now, click to install; D3: Samsung mobile phone running assistant strategy, including: D31: Configure the mobile phone screen resolution to 1080*2400; D32: Match the permission names of the mobile phone interface elements in Samsung Switch Assistant and click them respectively; D33: Click ApkDataMove to install; D4: OPPO mobile phone operation assistant strategy, including: D41: Configure the mobile phone screen resolution to 1080*2400; D42: Match the permission names of the mobile phone interface elements in OPPO Moving Assistant and click them respectively; D43: When the name of the matching mobile phone interface element is Connect Now, click to install; D5: vivo mobile phone operation assistant strategy, including: D51: Configure the phone screen resolution to 1080*2160; D52: Match the permission names of the mobile phone interface elements in the vivo transfer assistant and click them respectively; D53: When the name of the matching mobile phone interface element is backup, click to install; D6: Xiaomi mobile phone operation assistant strategy, including: D61: Configure the phone screen resolution to 1080*2400; D62: Match the permission names of the mobile phone interface elements in Xiaomi Assistant and click them to install them respectively.

Images