Cross-platform mobile terminal automatic testing method and device based on natural language driving

Abstract

The invention discloses a cross-platform mobile terminal automatic testing method and device based on natural language driving and a medium. The method comprises the following steps: analyzing a natural language instruction in a preset format file, and extracting an operation semantic feature and a target object description; matching an underlying control protocol instruction corresponding to the system type of the current tested equipment from a preset cross-platform instruction abstraction library according to the operation semantic features; acquiring a current interface image of the equipment, and retrieving a target coordinate area matched with the target object description by using a hybrid positioning mechanism; and finally, generating and executing an automatic test action. According to the method, the problem of inaccurate element positioning in a complex UI scene is solved through weighted fusion and intersection-union correlation of text and icon areas in a hybrid positioning mechanism; and meanwhile, the abstract library is utilized to shield the underlying protocol difference between the Android and the iOS, so that the cross-platform test driven by the natural language is realized, the script maintenance cost is remarkably reduced, and the test efficiency is improved.

Description

Technical Field

[0001] This invention relates to the field of mobile terminal testing technology, and in particular to a cross-platform automated testing method and apparatus for mobile terminals based on natural language driving. Background Technology

[0002] Existing automated testing frameworks (such as Appium) typically require testers to write scripts based on specific programming languages ​​(such as Python and Java). This not only requires testers to have a high level of programming skills, but also often requires maintaining two sets of scripts with significantly different logics when facing cross-platform testing of Android and iOS, resulting in extremely high maintenance costs.

[0003] Furthermore, traditional element location primarily relies on Resource IDs or XPath. However, in actual development, control attribute definitions vary across different platforms, and dynamically generated pages often cause IDs to become invalid, resulting in poor stability of test scripts. Although there is currently research introducing Optical Character Recognition (OCR) or image recognition, single location methods still struggle to achieve the accuracy required for industrial-grade automated testing when dealing with scenarios involving mixed text and images or ambiguous icon semantics.

[0004] Therefore, the existing technology still needs further improvement and enhancement. Summary of the Invention

[0005] This invention aims to solve the above-mentioned technical problems and provides a cross-platform mobile terminal automated testing method based on natural language driving, the method comprising: Parse the natural language instructions stored in the preset format file to extract the semantic features of the operation and the description of the target object; Based on the operational semantic features, the underlying control protocol instructions corresponding to the operating system type of the current device under test are matched from the preset cross-platform instruction abstraction library. The current interface image of the device under test is acquired, and a hybrid positioning mechanism is used to retrieve the target coordinate region in the interface image that matches the description of the target object; wherein, the hybrid positioning mechanism includes: performing optical character recognition on the interface image to extract the text semantic region; and performing image template matching on the interface image to extract the icon feature region; Based on the target object description, the text semantic region and the icon feature region are weighted and fused to determine the target coordinate region; Based on the underlying control protocol instructions and the target coordinate region, automated test actions are generated and executed for the device under test.

[0006] The following are preferred technical solutions of the present invention, but are not intended to limit the technical solutions provided by the present invention. The purpose and beneficial effects of the present invention can be better achieved and realized through the following preferred technical solutions.

[0007] As a preferred technical solution, the method, wherein the step of using a hybrid positioning mechanism to retrieve a target coordinate region in the interface image that matches the target object description includes: The semantic channel and the visual channel are launched in parallel; wherein, the semantic channel extracts a set of candidate text blocks through optical character recognition, and the visual channel extracts a set of candidate icon regions through template matching; Calculate the intersection-union ratio between the candidate text block and the candidate icon region; when the intersection-union ratio between any candidate text block and any candidate icon region exceeds a preset threshold, associate the candidate text block and the candidate icon region as the same composite logic component.

[0008] As a preferred technical solution, the method wherein determining the target coordinate region includes using a weighted scoring function: Calculate the comprehensive score for each candidate region; among them, This refers to the semantic relevance between keywords in the instruction and candidate text content. To score image similarity, The bias weight is the distance between the candidate region and the coordinates of the previous operation. These are the corresponding weighting coefficients.

[0009] As a preferred technical solution, the method wherein matching the underlying control protocol instructions corresponding to the operating system type of the current device under test from a preset cross-platform instruction abstraction library includes: identifying whether the operating system of the device under test is Android or iOS; if it is Android, mapping the operation semantic features to the corresponding uiautomator2 underlying instructions; if it is iOS, mapping the operation semantic features to the corresponding WebDriverAgent underlying instructions.

[0010] As a preferred technical solution, the method wherein parsing the natural language instructions stored in the preset format file includes: Identify action verbs, target nouns, and input parameters in instructions using natural language processing rules; The action verbs are mapped to a pre-set set of basic instructions or a set of encapsulated instructions; wherein the encapsulated instruction set is composed of multiple basic instructions combined in a logical sequence, and is used to perform continuous operations in a specific business scenario.

[0011] As a preferred technical solution, the method further includes, during the execution of automated test actions for the device under test: Continuously acquire continuous image frames of the device under test during the execution of a specific service; Calculate the frame similarity between adjacent image frames; When the frame similarity is higher than a preset threshold for a continuous preset time period, it is determined that the tested device has experienced interface lag, and the start time and duration of the lag are recorded.

[0012] As a preferred technical solution, in the method described, the result of the stuttering determination serves as a performance indicator feedback for automated testing, used to trigger preset exception handling actions or generate performance evaluation data in the test report.

[0013] As a preferred technical solution, the method further includes, prior to acquiring the current interface image of the device under test and retrieving it from the interface image using a hybrid positioning mechanism: In response to the interface capture command, it iterates through the semantic attributes and visual features of all interactive elements in the current interface. The interactive elements are cropped and semantically named to generate a visual element lookup table. The visual element lookup table assists in defining the target object description in the natural language instructions.

[0014] Secondly, a cross-platform automated testing device for mobile terminals based on natural language driving, comprising: The semantic parsing module is used to parse natural language instructions and extract operational semantic features and target object descriptions; The instruction mapping module is used to match the underlying control protocol instructions corresponding to the system type of the device under test based on the operation semantic features; The visual-assisted positioning module is used to determine the target coordinate area in the interface image of the device under test through a hybrid positioning mechanism of optical character recognition text recognition and image template matching. The execution control module is used to perform automated testing by combining the instruction mapping results with the target coordinate region.

[0015] Thirdly, a computer-readable storage medium having a computer program stored thereon, wherein the program, when executed by a processor, implements the method described above.

[0016] Beneficial Effects: Compared with existing technologies, this invention achieves complete decoupling between test logic and underlying implementation code by parsing natural language instructions in a preset format file and extracting operational semantic features. Testers do not need to master complex programming languages; they only need to describe action verbs, target nouns, and parameters using natural language to complete test case writing. Simultaneously, by using encapsulated instruction sets to combine basic operations into business scenario logic, the readability and reusability of test scripts are significantly improved. Utilizing a preset cross-platform instruction abstraction library, this invention can automatically match and map to the corresponding underlying control protocol instructions (such as uiautomator2 or WebDriverAgent) based on the operating system type of the device under test (such as Android or iOS). This mechanism enables the same set of natural language test scripts to be executed seamlessly across platforms, solving the problem of maintaining multiple sets of driver logic for different systems in traditional testing. Attached Figure Description

[0017] Figure 1 This is a schematic diagram of the cross-platform mobile terminal automated testing method based on natural language driving provided by the present invention. Figure 2 This is a schematic diagram of the execution flow of the automated testing tool in an embodiment of the present invention; Figure 3 This is a schematic diagram of the one-click interface information collection process in an embodiment of the present invention; Figure 4 This is a schematic diagram of the parallel processing logic of the hybrid positioning mechanism in an embodiment of the present invention; Figure 5 This is a flowchart of the interface stuttering determination logic based on frame similarity in an embodiment of the present invention; Figure 6 This is a schematic diagram of the structure of a cross-platform mobile terminal automated testing device driven by natural language in an embodiment of the present invention. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this invention clearer and more explicit, the invention is further described in detail below. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Experimental methods in the following embodiments, unless otherwise specified, are performed according to conventional methods and conditions, or as selected in the product specification. Furthermore, it should be understood that after reading the teachings of this invention, those skilled in the art can make various alterations or modifications to the invention, and these equivalent forms also fall within the scope defined by the appended claims.

[0019] like Figures 1 to 2 As shown, this invention provides a cross-platform mobile terminal automated testing method based on natural language driving, the method comprising: Step S101: Parse the natural language instructions stored in the preset format file and extract the semantic features of the operation and the description of the target object; Step S102: Based on the operation semantic features, match the underlying control protocol instructions corresponding to the operating system type of the current device under test from the preset cross-platform instruction abstraction library; Step S103: Obtain the current interface image of the device under test, and use a hybrid positioning mechanism to retrieve the target coordinate region in the interface image that matches the description of the target object; wherein, the hybrid positioning mechanism includes: performing optical character recognition on the interface image to extract the text semantic region; and performing image template matching on the interface image to extract the icon feature region; Step S104: Based on the target object description, perform weighted fusion on the text semantic region and the icon feature region to determine the target coordinate region; Step S105: Based on the underlying control protocol instructions and the target coordinate region, generate and execute automated test actions for the device under test.

[0020] In this embodiment, natural language instructions are stored in a preset format file (such as YAML or JSON). For example, the instruction is "Click the login button on the login page". The parsing process consists of three levels: First, the verb "click" is identified using NLP rules and mapped to the system's preset "CLICK" basic instruction. Second, the target noun "login button" is identified. For Android devices, the system automatically calls the corresponding interface of "uiautomator2"; for iOS devices, it maps to the "WebDriverAgent" instruction. This mapping logic is encapsulated in an abstract library and is transparent to front-end testers.

[0021] In this embodiment, after acquiring the interface image, the system does not simply perform recognition, but uses dual-channel parallel processing: for example, recognizing all text in the interface, such as "OK", "Exit", and "Login". It also recognizes icons with specific characteristics, such as the "cross icon" and the "search magnifying glass".

[0022] In this embodiment, as Figure 2 As shown, before starting the test, first check if ADB (Android Debug Bridge) is installed on the system. If not installed (N), you need to "actively install ADB"; if installed (Y), proceed directly to "launch the QT interface". After launching the QT interface, execute the following steps in sequence: Import a natural language table (.xlsx file) as the command source. Identify the current operating system as Android / iOS (clearly marked as Android in the figure).

[0023] Instruction Parsing and Execution Branches: After "Parsing Instructions," the process splits into two main branches: Basic Instruction Branch: Used for handling general operations. It checks if the instruction has sub-instructions. If not (N), it skips. If it exists (Y), it parses the instruction test type to determine whether it's a "system-level" operation or "interface element control." If it's "interface element control," it performs "ID / text / tag library" matching and then sends the instruction. If it's "system-level," it directly sends the instruction. Encapsulated Instruction Branch: Used for handling complex or specific application scenarios. It directly selects to parse basic test cases and then loops until the test execution ends. All instruction execution paths ultimately converge on "obtaining execution results and recording reports." After this step, the process returns to "Parsing Instructions," forming a loop until all test cases have been executed. The final output is "Generate a summary report."

[0024] By parsing test cases in .xlsx format, basic instructions and encapsulated instructions are distinguished. Different execution strategies are invoked based on the instruction type (such as low-level system operations or UI element control), and the process is continuously looped until completion. Finally, a test report is generated. The entire process demonstrates a complete closed loop from data input, logical judgment, instruction execution to result feedback.

[0025] In this embodiment, as Figure 3 As shown, the system provides a one-click interface information collection function, allowing users to automatically obtain the text and icon information of all interactive elements on the current screen through simple commands. The system automatically filters redundant attributes, retaining only key semantic information (such as button text and icon features), and presents it in a visual image cropping + ID naming format, making it easy for testers to quickly identify and reference, significantly reducing the difficulty of element identification and location.

[0026] In this embodiment, as Figure 4 As shown, a hybrid localization technology combining OCR text recognition and image template matching is employed. The system automatically identifies visible text content and icon features in the interface by acquiring a screenshot of the device screen, and uses this as the basis for element localization. Even in scenarios where control IDs are missing, dynamically changing, or obfuscated, it can still accurately complete click and input operations, significantly improving the stability of the test script and its cross-version reusability.

[0027] To further improve recognition accuracy, this invention introduces a scoring mechanism. This includes calculating the cosine similarity between instruction keywords (e.g., "confirm") and OCR-recognized text (e.g., "OK") based on semantic relevance using a word vector model; calculating image similarity by matching the features of the template image with the current screenshot area; and calculating distance bias, considering mobile operating habits where consecutive operations typically occur in close proximity. The Euclidean distance between the center of the current candidate area and the coordinates of the last click is calculated, with closer distances resulting in higher weight scores. This effectively avoids location drift when multiple similar elements exist on the page (e.g., multiple "delete" buttons in a list).

[0028] For example, in the search interface of an e-commerce app, there is a search button consisting of a magnifying glass icon and the word "search", and there are multiple related terms containing the word "search" in the list at the bottom of the interface.

[0029] The system parses the natural language command "Click the search button". OCR identifies the coordinate set A of all "search" text blocks within the interface. Image template matching retrieves a set B of candidate regions similar to the preset "search icon". The system calculates the intersection-union ratio (IoU) between the text blocks in set A and the icon regions in set B. When an IoU between a "search" text and the "magnifying glass icon" is detected (exceeding a preset threshold of 0.5), the system associates them as the same "composite logical component".

[0030] Using a weighted scoring function: Calculate the overall score for each candidate region separately; among which, This refers to the semantic relevance between keywords in the instruction and the candidate text content. To score image similarity, The bias weight is the distance between the candidate region and the coordinates of the previous operation. These are the corresponding weight coefficients. Among them, since the keyword for the instruction is "search"... The highest score was achieved because multiple similar texts existed, making it possible to accidentally click on the text based solely on semantics. The icon features were identified; combined with fdist (bias weight), if the component is located at the top of the screen in the regular search position, its bias score relative to the previous center operation position meets expectations; ultimately, the composite component obtains the highest score, the system determines it as the target coordinate area and executes the click action.

[0031] In one implementation of the present invention, the tester wrote a business-level natural language instruction: "Execute 'Cancel Account' encapsulation operation", which is required to be executed synchronously on test machines on both Android and iOS platforms.

[0032] Analysis revealed that "Cancel Account" belongs to a preset "encapsulated instruction set", which is composed of three basic instruction logic sequences: "Enter Settings", "Security Center", and "Submit Cancellation".

[0033] Platform identification and mapping: If the device system is identified as Android, the instruction mapping module maps the semantics of "click settings" to uiautomator2 underlying instructions such as device(text="settings").click(). If the device is identified as iOS, the same semantic features are mapped to webDriverAgent underlying instructions such as session.tap(element).

[0034] Testers do not need to be aware of the differences in underlying protocols; they can achieve cross-platform business continuity operations through a set of natural language descriptions, reducing script maintenance costs.

[0035] like Figure 5 As shown, this invention, when performing the automated test task of "opening the high-definition video playback page," incorporates a live stream stuttering detection function within the system. By continuously comparing the similarity of consecutive frame images, it monitors whether the video freezes. When an image frame does not change significantly for an extended period, it is determined that stuttering has occurred, and the start time and duration of the stutter are automatically recorded. This function can achieve automated monitoring and evaluation of the playback smoothness of live streaming apps (iOS / Android) without relying on the internal logs of the application under test.

[0036] For example, the test begins by monitoring video playback. First, screenshot A is captured as the current baseline frame. Then, screenshot B is captured as the subsequent frame. The subsequent frame is compared with the baseline frame. The algorithm compares the similarity differences to determine whether there is a significant change between the two frames (i.e., whether stuttering occurs).

[0037] If "stuttering occurs during testing" (i.e., the similarity difference does not reach the expected threshold or there is no change in consecutive frames), the following branch is triggered: "Record current time" marks the start time of the lag. "Detect lag duration within 20 seconds" continuously monitors lag over the next 20 seconds. Finally, "Return results to engineer," including the time of the lag and its duration. If "no lag occurs within the set time," directly "Return results to engineer," reporting normal operation.

[0038] After "getting screenshot B", if no stuttering is detected, the process will return to "getting screenshot A", forming a continuous monitoring loop to ensure uninterrupted monitoring of the entire video playback process.

[0039] This embodiment periodically captures video frames and calculates their similarity. When the difference between frames is too small, it is determined to be a stutter. The time and duration of the stutter are recorded, and the detection results are finally fed back to the engineer for video playback quality assessment and problem localization.

[0040] like Figure 6 As shown, the present invention also provides a cross-platform mobile terminal automated testing device based on natural language driving, comprising: a semantic parsing module 100, used to parse natural language instructions and extract operational semantic features and target object descriptions; an instruction mapping module 200, used to match underlying control protocol instructions corresponding to the system type of the device under test according to the operational semantic features; a visual-assisted positioning module 300, used to determine the target coordinate region in the interface image of the device under test through a hybrid positioning mechanism of optical character recognition text recognition and image template matching; and an execution control module 400, used to perform automated testing by combining the instruction mapping results and the target coordinate region.

[0041] Based on the same inventive concept, the present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described above.

[0042] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A cross-platform automated testing method for mobile terminals based on natural language, characterized in that, The method includes: Parse the natural language instructions stored in the preset format file to extract the semantic features of the operation and the description of the target object; Based on the operational semantic features, the underlying control protocol instructions corresponding to the operating system type of the current device under test are matched from the preset cross-platform instruction abstraction library. The current interface image of the device under test is acquired, and a hybrid positioning mechanism is used to retrieve the target coordinate region in the interface image that matches the description of the target object; wherein, the hybrid positioning mechanism includes: performing optical character recognition on the interface image to extract the text semantic region; and performing image template matching on the interface image to extract the icon feature region; Based on the target object description, the text semantic region and the icon feature region are weighted and fused to determine the target coordinate region; Based on the underlying control protocol instructions and the target coordinate region, automated test actions are generated and executed for the device under test.

2. The method according to claim 1, characterized in that, The step of using a hybrid positioning mechanism to retrieve target coordinate regions in the interface image that match the description of the target object includes: The semantic channel and the visual channel are launched in parallel; wherein, the semantic channel extracts a set of candidate text blocks through optical character recognition, and the visual channel extracts a set of candidate icon regions through template matching; Calculate the intersection-union ratio between the candidate text block and the candidate icon region; when the intersection-union ratio between any candidate text block and any candidate icon region exceeds a preset threshold, associate the candidate text block and the candidate icon region as the same composite logic component.

3. The method according to claim 2, characterized in that, Determining the target coordinate region includes using a weighted scoring function: Calculate the overall score for each candidate region separately; among which, This refers to the semantic relevance between keywords in the instruction and the candidate text content. To score image similarity, The bias weight is the distance between the candidate region and the coordinates of the previous operation. These are the corresponding weighting coefficients.

4. The method according to claim 1, characterized in that, The step of matching the underlying control protocol instructions corresponding to the operating system type of the current device under test from the preset cross-platform instruction abstraction library includes: identifying whether the operating system of the device under test is Android or iOS; if it is Android, mapping the operation semantic features to the corresponding uiautomator2 underlying instructions; if it is iOS, mapping the operation semantic features to the corresponding WebDriverAgent underlying instructions.

5. The method according to claim 1, characterized in that, The parsing of natural language instructions stored in the preset format file includes: Identify action verbs, target nouns, and input parameters in instructions using natural language processing rules; The action verbs are mapped to a pre-set set of basic instructions or a set of encapsulated instructions; wherein the encapsulated instruction set is composed of multiple basic instructions combined in a logical sequence, and is used to perform continuous operations in a specific business scenario.

6. The method according to claim 1, characterized in that, During the execution of automated test actions for the device under test, the method further includes: Continuously acquire continuous image frames of the device under test during the execution of a specific service; Calculate the frame similarity between adjacent image frames; When the frame similarity is higher than a preset threshold for a continuous preset time period, it is determined that the tested device has experienced interface lag, and the start time and duration of the lag are recorded.

7. The method according to claim 6, characterized in that, The result of the stuttering determination serves as a performance indicator feedback for automated testing, which is used to trigger preset exception handling actions or generate performance evaluation data in the test report.

8. The method according to claim 1, characterized in that, Before acquiring the current interface image of the device under test and searching in the interface image using a hybrid positioning mechanism, the process also includes: In response to the interface capture command, it iterates through the semantic attributes and visual features of all interactive elements in the current interface. The interactive elements are cropped and semantically named to generate a visual element lookup table. The visual element lookup table assists in defining the target object description in the natural language instructions.

9. A cross-platform automated testing device for mobile terminals based on natural language, characterized in that, include: The semantic parsing module is used to parse natural language instructions and extract operational semantic features and target object descriptions; The instruction mapping module is used to match the underlying control protocol instructions corresponding to the system type of the device under test based on the operation semantic features; The visual-assisted positioning module is used to determine the target coordinate area in the interface image of the device under test through a hybrid positioning mechanism of optical character recognition text recognition and image template matching. The execution control module is used to perform automated testing by combining the instruction mapping results with the target coordinate region.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method described in any one of claims 1-8.

Images