Automated Sandbox Testing Methods, Equipment, and Computer Storage Media Based on Computer Vision and Machine Learning

Abstract

This invention provides an automated sandbox testing method based on computer vision and machine learning, comprising: collecting operation recording data of a sandbox terminal, and saving screen layout information files and operation marker sequence files of the recording data according to timestamps; dividing the recording data into operation slices of several scenes, extracting frames from each slice, preprocessing each extracted frame, and extracting feature elements from the preprocessed images using feature extraction algorithms and screen layout information files; inputting the extracted feature elements into a computer vision network model for training to construct an operable element classification model, which identifies operable elements in the recording data; inputting the operable elements into a machine learning model for self-supervised learning training to construct a business scene recognition model; and outputting prediction results based on the collected recording data for automated testing. This invention provides a comprehensive and reliable automation tool.

Description

Technical Field

[0001] The invention belongs to the field of automated testing technology, and specifically relates to a sandbox automated testing method, device and computer storage medium based on computer vision and machine learning. Background Technology

[0002] In existing technologies, rule-based automated testing tools face a series of challenges in processing applications on smartphone devices. Traditional automated testing tools typically rely on parsing and matching rules of interface XML elements to perform depth-first or breadth-first traversal operations. However, this approach struggles to develop universal rules to cover all possible scenarios when dealing with diverse mobile applications, especially those that intersect with multiple business scenarios, such as office, social networking, finance, and shopping applications, or even more complex situations like live-streaming e-commerce. Traditional methods also suffer from inaccurate interface change recognition; for example, it's difficult to distinguish whether a change in control focus is due to a newly popped-up dialog box or an advertisement page, and in cases with extremely long ListViews, page scrolling can cause a complete update of the main interface. Therefore, these methods often lack universal adaptability to numerous application scenarios, have limited coverage, and offer poor interface recognition performance.

[0003] In extreme cases, when the interface layout remains unchanged while the main display element is replaced, traditional methods will completely fail. The most typical scenario is game applications. Mobile games typically integrate third-party frameworks, operating entirely independently of the phone's interface layout. All interface transitions within the game occur within this third-party framework; the outermost system-based interface layout remains unchanged. In this situation, traditional application behaviors automatically trigger a complete system malfunction. Summary of the Invention

[0004] The purpose of this invention is to address the problems in the prior art by proposing a sandbox automated testing method, device, and computer storage medium based on computer vision and machine learning. This provides a comprehensive and reliable automated tool for application security compliance testing and software development testing, and is of practical significance for the complex and ever-changing application scenarios in smartphone devices.

[0005] To achieve the above objectives, the present invention adopts the following technical solution.

[0006] The aforementioned automated sandbox testing method based on computer vision and machine learning includes collecting operation video recordings of the sandbox terminal and saving the screen layout information file and operation marker sequence file of the video recordings according to timestamps.

[0007] The video data is divided into several scene operation slices, the slices are extracted, each extracted frame image is preprocessed, and feature elements in the preprocessed image are extracted using a feature extraction algorithm and the screen layout information file.

[0008] The extracted feature elements are input into a computer vision network model for training to construct an operable element classification model. The operable element classification model identifies operable elements in the video data. The operable elements are input into a machine learning model for self-supervised learning training to construct a business scenario recognition model. The business scenario recognition model outputs prediction results based on the collected video data for automated testing.

[0009] Furthermore, the video recording data includes user operation behavior on the terminal, interface interaction, and application scenario transitions.

[0010] Furthermore, the operation marker sequence file records user interaction events during terminal use, and each event includes screen coordinates. The screen coordinates of each event are saved one by one according to the timestamp.

[0011] Furthermore, the step of dividing the video recording data into several scene operation slices includes, during the operation of each terminal, identifying the return to the application homepage as a marker indicating the end of a scene operation, and dividing the video recording data into different operation scenes based on the identification marker.

[0012] Furthermore, the step of extracting frames from the slice includes:

[0013] After extracting frames from the slices, the interface changes are determined based on image similarity. The identified independent interfaces are retained, resulting in an ordered set of images representing a video slice of an operation scene. The corresponding screen layout information file is obtained based on the timestamps of the retained frames.

[0014] Furthermore, the step of inputting the extracted feature elements into a machine learning model for training to construct an operable element classification model includes,

[0015] The extracted feature elements are manually labeled, and the labeled feature elements are divided into a feature element training set and a feature element test set. The feature element training set is input into the machine learning model for model training, and the feature element test set is input into the trained machine learning model for model validation, thus obtaining an operable element classification model.

[0016] Furthermore, the step of inputting the operable elements into a self-supervised learning model for self-supervised learning training to construct a business scenario recognition model includes,

[0017] After obtaining the image frame sequence representing the scene and the information of operable elements in the image, a single complete scene is regarded as a single sentence, and the interface images and interface operation objects arranged in an orderly manner in a single scene are regarded as words. A machine learning model is used for semantic training.

[0018] Furthermore, it also includes,

[0019] Collect new operation recording data from the sandbox terminal and optimize the business scenario recognition model using a feedback mechanism. Adjust the parameters of the business scenario recognition model by comparing and analyzing the prediction results with actual user behavior.

[0020] To achieve the above objectives, the present invention also provides an electronic device, including a memory and a processor, wherein the memory stores a program that runs on the processor, and the processor executes the steps of the sandbox automated testing method based on computer vision and machine learning as described above when running the program.

[0021] To achieve the above objectives, the present invention also provides a computer-readable storage medium having computer instructions stored thereon, which, when executed, perform the steps of the sandbox automated testing method based on computer vision and machine learning as described above.

[0022] Beneficial effects

[0023] This invention proposes a sandbox automated testing method, device, and computer storage medium based on computer vision and machine learning, which has the following beneficial effects:

[0024] (1) Comprehensiveness and accuracy: Combining computer vision and machine learning technologies, it can comprehensively evaluate the security, performance and reliability of software programs in smartphone devices, and accurately identify potential security vulnerabilities and abnormal behaviors.

[0025] (2) Automation and efficiency: The tool can automatically execute test tasks, reduce manual intervention, and improve testing efficiency. It no longer relies on predefined rules, but instead uses intelligent triggering by simulating user behavior.

[0026] (3) Real-time monitoring and feedback: Data is collected in real time through the sandbox environment, which can promptly detect and report abnormal behavior, enabling problems to be resolved quickly.

[0027] (4) Continuous optimization: The machine learning model learns and improves continuously to adapt to the ever-changing application scenarios and improve the accuracy and robustness of the test.

[0028] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention. Attached Figure Description

[0029] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used together with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0030] Figure 1 This is a flowchart of the sandbox automated testing method based on computer vision and machine learning according to the present invention;

[0031] Figure 2 This is a schematic diagram of interface element feature extraction in the sandbox automated testing based on computer vision and machine learning according to the present invention.

[0032] Figure 3 This is a schematic diagram of the training of the interface operable element recognition model in the sandbox automated testing based on computer vision and machine learning according to the present invention.

[0033] Figure 4 This is a schematic diagram illustrating the identification and classification of operable elements in the sandbox automated testing interface based on computer vision and machine learning according to the present invention.

[0034] Figure 5 This is a schematic diagram illustrating the construction of a business scenario recognition model in the sandbox automated testing interface based on computer vision and machine learning according to the present invention. Detailed Implementation

[0035] The preferred embodiments of the present invention will be described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0036] Example 1

[0037] Figure 1 The flowchart of the sandbox automated testing method based on computer vision and machine learning according to the present invention will be referenced below. Figure 1 The present invention provides a detailed description of the sandbox automated testing method based on computer vision and machine learning.

[0038] In step 101, the operation recording data of the sandbox terminal is collected, and the screen layout information file and operation mark sequence file of the recording data are saved according to the timestamp.

[0039] Preferably, video recordings of app usage on smartphone devices are collected, such as at least 1000 different apps or 1000 video recordings. These recordings document user actions on the device, including screenshots and user interaction events. The video data covers user actions with the apps, interface interactions, and changes in application scenarios.

[0040] Preferably, during the recording process, an XML file containing screen layout information and a corresponding operation mark sequence file are saved by timestamp (per second).

[0041] Preferably, the operation marker sequence records user events such as clicks, swipes, and text inputs during APP use. Each event includes screen coordinates and is saved one by one according to timestamps.

[0042] In this embodiment, the sandbox refers to the hardware and software environment required for the app to run. It can be an actual Android phone with a specific ROM burned into it, a customized development board, or even a customized emulator compatible with the ARM instruction set. The sandbox is required to be able to collect and retain comprehensive data during the app's operation, such as screenshots and video recordings.

[0043] In step 102, the video recording data is divided into several scene operation slices, frames are extracted from the slices, each extracted frame image is preprocessed, and feature elements in the preprocessed image are extracted using a feature extraction algorithm and the screen layout information file.

[0044] Preferably, the video recording is divided into different operation scenarios. During the operation of each app, "returning to the application homepage" is identified as the end of an operation scenario. Based on this rule, the application operation recording is divided into operation slices of several scenarios.

[0045] Preferably, after extracting frames from the video slices, interface changes are determined based on image similarity, retaining all identifiable independent interfaces. This results in an ordered set of images representing a video slice of an operation scene, and the corresponding layout XML file is found based on the timestamps of the retained frames.

[0046] Preferably, each frame of the image is preprocessed, including denoising, enhancement, edge detection and other preprocessing operations, in preparation for the subsequent feature extraction steps.

[0047] Preferably, such as Figure 2 As shown, various feature extraction algorithms, such as edge detection, corner detection, and color histograms, are used, combined with layout information from the XML file, to extract key feature elements from the processed image, such as the position, size, and color of interface elements. These features help the system identify various operable objects on the interface.

[0048] In step 103, the extracted feature elements are input into a computer vision network model for training to construct an operable element classification model. The operable element classification model identifies operable elements in the video data. The operable elements are input into a machine learning model for self-supervised learning training to construct a business scenario recognition model. The business scenario recognition model outputs prediction results based on the collected video data for automated testing.

[0049] Preferably, such as Figure 3 As shown, the operable element recognition model is based on a supervised learning model. First, key feature elements extracted from the interface elements need to be manually labeled. These labeled key feature elements are then divided into training and test sets. The training set is input into the Faster R-CNN model for training, and the test set is input into the trained Faster R-CNN model for testing, resulting in the operable element recognition model. Figure 4 As shown, the operable element recognition model automatically identifies operable object information on the interface to be detected, including elements that users can interact with, such as buttons, text boxes, sliders, etc., and classifies and labels the identified operable objects according to their attributes.

[0050] Preferably, the steps for constructing a business scenario recognition model can be specifically executed as follows: After obtaining the image sequence representing the scenario and the information of operable objects in each image, each complete scenario is regarded as a sentence, and the interface images and interface operable objects arranged in an orderly manner in each scenario are regarded as words; the APP running the scenario intelligent recognition model trained using the Transformer architecture is used for semantic training. In this process, the system does not need to pre-label the data, but automatically discovers the patterns and structures in the data by observing the data. The patterns include the classification of scenarios for interfaces and interface sequences with similar specific features; the structures include the summary and understanding of the behavior trigger sequences that may exist in a specific scenario, as well as the inherent correlation between the scenarios and interface switching caused by the behavior triggers.

[0051] In this embodiment, the business scenario identification model is implemented based on an unsupervised learning model, such as... Figure 5 As shown, the business scenario recognition model automatically analyzes the app's usage recordings to identify different business scenarios. It can identify patterns between user behavior and interface interactions, thereby automatically segmenting the recordings and labeling them with different business scenarios. For example, when a user moves from the login screen to the main page and then to the settings page, the system can accurately identify these scenario transitions.

[0052] In this embodiment, as Figure 5As shown, frames labeled with operable object information are input into a Transformer encoder according to their frame feature sequences. Simultaneously, the frame feature sequences and the positional encodings of the operable objects are input into the Transformer encoder for model training, resulting in a business scenario recognition model. The output is a feature representation of the frame sequence, used for business scenario prediction, predicting the business scenario and behavior for the next frame, and performing the corresponding operation. It also includes applying the same operation to similar identified scenarios. The frame feature sequence includes records of interface element click behaviors. Positional encoding maps behaviors to operable objects, finding the corresponding operable object by analyzing the position of the interface element click behavior within the entire interface and its position in the frame feature sequence.

[0053] Preferably, new APP usage video data is continuously collected, and a feedback mechanism is used to optimize the unsupervised learning model. By comparing and analyzing the data with actual user behavior, the system adjusts the model parameters to improve the accuracy and stability of identifying business scenarios and operable objects.

[0054] This invention can identify the business scenarios and operable objects of an app in real time, providing data support for subsequent automated testing. When abnormal behavior or security vulnerabilities are detected, the system can report them promptly to help developers fix the problems. Through these steps, the system can automatically identify the operable objects and business scenarios of an app using unsupervised learning technology, providing a strong foundation for automated testing.

[0055] This invention proposes a sandbox-based automated testing tool based on computer vision and machine learning. It combines APP operation data, interface layout data, and screenshot / screen recording data, using machine vision technology to self-learn interface changes and operable elements. APP operation behaviors are segmented based on business scenarios, using this as training corpus to construct a self-supervised learning-based machine learning model. An attention mechanism is used to analyze and learn the inherent business logic and contextual relationships of the application scenario, enabling fully automated testing of reproduced scenarios. This tool is used for automated testing and security assessment of software programs in smartphone devices. Combining computer vision technology and machine learning algorithms, it can simulate user behavior and execute software programs in a controlled environment to detect potential security vulnerabilities or abnormal behaviors. During testing, the software under test runs in a customized sandbox, allowing for real-time collection and detection of sample software behavior and network data. Computer vision technology is used to capture the interactive interface of the software program, identify interface changes and transitions, and analyze interface elements, parsing clickable, swipeable, zoomable, and inputtable control objects. Machine learning algorithms are used to identify the application's business scenarios, understand and learn the inherent relationships between interface changes during continuous operation, and then execute logical behavior triggers.

[0056] It will be understood by those skilled in the art that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A sandbox automated testing method based on computer vision and machine learning, characterized in that, include: Collect operation recording data of the sandbox terminal, and save the screen layout information file and operation mark sequence file of the recording data according to the timestamp; The video data is divided into several scene operation slices, the slices are extracted, each extracted frame image is preprocessed, and feature elements in the preprocessed image are extracted using a feature extraction algorithm and the screen layout information file. The extracted feature elements are input into a computer vision network model for training to construct an operable element classification model. The operable element classification model identifies operable elements in the video data. The operable elements are input into a machine learning model for self-supervised learning training to construct a business scenario recognition model. The business scenario recognition model outputs prediction results based on the collected video data for automated testing. The operation of dividing the video recording data into several scene slices includes, during the operation of each terminal, identifying the return to the application homepage as a sign that a scene operation has ended, and dividing the video recording data into different operation scenes according to the identification sign; The step of extracting frames from the slice includes: After extracting frames from the slices, the interface changes are determined based on image similarity. The identified independent interfaces are retained, resulting in an ordered set of images representing a video slice of an operation scene. The corresponding screen layout information file is obtained based on the timestamps of the retained frames. The step of inputting the extracted feature elements into a machine learning model for training to construct an operable element classification model includes, The extracted feature elements are manually labeled, and the labeled feature elements are divided into a feature element training set and a feature element test set. The feature element training set is input into the machine learning model for model training, and the feature element test set is input into the trained machine learning model for model validation, thus obtaining an operable element classification model. The step of inputting the operable elements into a self-supervised learning model for self-supervised learning training to construct a business scenario recognition model includes, After obtaining the image frame sequence representing the scene and the information of operable elements in the image, a single complete scene is regarded as a single sentence, and the interface images and interface operation objects arranged in an orderly manner in a single scene are regarded as words. A machine learning model is used for semantic training. It also includes, Collect new operation recording data from the sandbox terminal and optimize the business scenario recognition model using a feedback mechanism. Adjust the parameters of the business scenario recognition model by comparing and analyzing the prediction results with actual user behavior.

2. The sandbox automated testing method based on computer vision and machine learning according to claim 1, characterized in that, The recorded data includes user actions on the terminal, interface interactions, and application scenario transitions.

3. The sandbox automated testing method based on computer vision and machine learning according to claim 1, characterized in that, The operation marker sequence file records user interaction events during terminal use. Each event includes screen coordinates, and the screen coordinates of each event are saved one by one according to the timestamp.

4. An electronic device, characterized in that, It includes a memory and a processor, wherein the memory stores a program that runs on the processor, and the processor executes the sandbox automated testing method based on computer vision and machine learning as described in any one of claims 1-3 when running the program.

5. A computer-readable storage medium storing computer instructions thereon, characterized in that, The computer instructions execute the sandbox automated testing method based on computer vision and machine learning as described in any one of claims 1-3.

Images