Application program testing method, device and system and storage medium

Abstract

The invention relates to the technical field of automatic testing, and discloses an application program testing method, device and system and a storage medium. According to the method, a screenshot of a to-be-tested component in an application interface is obtained, and first visual feature extraction is conducted on the screenshot; performing feature extraction and coordinate recognition on all components in the current frame image of the terminal equipment, and outputting a second visual feature and coordinate mapping set; matching the first visual feature with the second visual feature, and determining a first coordinate corresponding to the first visual feature; extracting an image identifier corresponding to the first visual feature from the element component module, and then obtaining an associated test case from the test case module; according to the method, the to-be-tested component is tested based on the associated test case and the first coordinate corresponding to the first visual feature, intelligent positioning and interaction operation of the user interface component element are achieved through the visual feature recognition technology, dependence on a traditional element positioning expression is not needed, and the development complexity of the test case is effectively reduced.

Description

Technical Field

[0001] This invention relates to the field of automated testing technology, and specifically to an application testing method, apparatus, system, and storage medium. Background Technology

[0002] Traditional automated testing tools have structural defects in system adaptability and multimodal interface recognition accuracy. In particular, in complex scenarios such as text controls, the proportion has exceeded 58%, and the false recognition rate of traditional testing tools is as high as 42%. This directly leads to a sharp drop of 37% in the pass rate of automated testing of core business processes. Moreover, with the surge in demand for cross-operating system application (APP) testing, there is an urgent need for a brand-new APP testing solution.

[0003] The core technologies of current mainstream APP automated testing tools include the script-driven mode based on element location: test scripts are written based on element attributes such as control ID and XPath. However, there are the following problems: the naming rules of controls on different operating systems are very different, and scripts must be developed separately, resulting in high adaptation costs. Furthermore, when the APP interface is frequently updated, the element location logic needs to be manually adjusted, which is time-consuming and prone to errors. Summary of the Invention

[0004] In view of this, the present invention provides an application testing method, apparatus, system and storage medium to solve the problems of difficult driver adaptation and high maintenance cost of script-driven mode based on element location.

[0005] In a first aspect, the present invention provides an application testing method applied to an application testing system. The application testing system includes an element component module and a test case module. The element component module stores visual feature images and associated image identifiers corresponding to each component, and the test case module stores test cases generated based on the image identifiers. The method includes: acquiring a screenshot of a region of the component to be tested in the application interface, and extracting features from the screenshot to output a first visual feature; acquiring interface video stream data of the terminal device where the application is located, and extracting the current frame image; performing feature extraction and coordinate recognition on all components in the current frame image, and outputting a second visual feature and a coordinate mapping set corresponding to the second visual feature; matching the first visual feature with the second visual feature, and determining a first coordinate corresponding to the first visual feature from the coordinate mapping set based on the matching result; extracting the image identifier corresponding to the first visual feature from the element component module, and obtaining associated test cases from the test case module based on the image identifier; and testing the component to be tested based on the associated test cases and the first coordinate corresponding to the first visual feature to generate test results.

[0006] The application testing method provided in this embodiment is applied to an application testing system. It includes an element component module and a test case module. The element component module stores visual feature images and associated image identifiers for each component. The test case module stores test cases generated based on the image identifiers, eliminating the need for traditional element location expressions, effectively reducing the complexity of test case development and improving test case writing efficiency. The method involves acquiring a screenshot of the area of ​​the component under test in the application interface, extracting features from the screenshot, and outputting the first visual feature; acquiring the interface video stream data of the terminal device where the application is located, and extracting the current frame image; and performing feature extraction and coordinate recognition on all components in the current frame image. Output the second visual feature and the coordinate mapping set corresponding to the second visual feature; match the first visual feature with the second visual feature, and determine the first coordinate corresponding to the first visual feature from the coordinate mapping set according to the matching result; extract the image identifier corresponding to the first visual feature from the element component module, and obtain the associated test case from the test case module based on the image identifier; test the component under test based on the associated test case and the first coordinate corresponding to the first visual feature, generate test results, realize intelligent positioning and interactive operation of user interface component elements through visual feature recognition technology, execute automated operation instructions based on coordinate information, and realize seamless connection between automated test process and test business logic.

[0007] In one optional implementation, the first visual feature includes image visual features and / or text visual features. Feature extraction is performed on a screenshot of the area containing the component to be tested to output the first visual feature, including: determining, based on a preset component type determination rule, that the component to be tested in the screenshot is a text component and / or an image component; using a preset text recognition engine, extracting text elements from the text component in the screenshot and outputting text visual features; using a preset image feature recognition engine, extracting image visual elements from the image component in the screenshot and outputting image visual features; and integrating the text visual features and / or image visual features to output the first visual feature.

[0008] This invention designs a text recognition engine to extract, arrange, and locate text elements, significantly improving the accuracy of image text recognition. Considering that components include not only text content but also image features, the image feature recognition engine can also be used to recognize patterns of non-text user interface (UI) elements. This dual-engine collaborative operation can cover the recognition needs of interface elements across all application scenarios.

[0009] In one optional implementation, the image identifier is an image name, and the element component module is configured through the following steps: constructing a tree-like directory structure in the element component module, wherein each directory level corresponds to the navigation node of each application page in the application; storing the visual feature images corresponding to all components contained in the current application page in the corresponding hierarchical directory, wherein the visual feature images include text visual feature images and image visual feature images; naming the stored visual feature images using naming rules associated with the component's functional semantics to obtain the element component module.

[0010] This invention designs a tree-structured directory component managed by page navigation nodes, which can directly locate target components based on the directory hierarchy, avoiding full database scanning and improving component matching efficiency.

[0011] In an optional implementation, the method further includes: monitoring updates and changes to components in the application; and modifying the corresponding visual feature image in the element component module when a change is detected in the component.

[0012] When a change is detected in a component, this invention can modify the visual feature image in the element component module in real time to provide a real-time updated component visual image. This can flexibly respond to the dynamic change requirements of complex component elements. Furthermore, by accurately locating the component storage path and replacing it in a targeted manner, it avoids invalid resource occupation and achieves refined management of component visual resources.

[0013] In one optional implementation, modifying the corresponding visual feature image in the element component module when a component change is detected includes: when a change in the visual features of the current component is detected, performing feature extraction on the current component, outputting a third visual feature image, and replacing the visual feature image corresponding to the current component in the element component module with the third visual feature image; or, when a current component is detected to be deleted, deleting the visual feature image corresponding to the current component in the element component module; or, when a new component is detected, performing feature extraction on the new component, outputting a fourth visual feature image, and adding the fourth visual feature image to the corresponding hierarchical directory in the element component module.

[0014] This invention employs operations of replacing, deleting, and adding visual feature images for the three states of component updates, deletions, and additions, respectively, to ensure that the visual feature images stored in the element component module are completely consistent with the component states during actual application runtime, thus guaranteeing the real-time synchronization of visual resources.

[0015] In one optional implementation, the testing system further includes a data storage module that stores test data required for each test case. The step of testing the component under test based on the associated test cases and the first coordinates corresponding to the first visual feature includes: executing test instructions in the test cases sequentially; obtaining corresponding test data from the data storage module when a data request marker is detected in the test instruction; and injecting the test data into the component under test for testing based on the first coordinates.

[0016] This invention designs the test case module and data storage module to exist independently, so that when test data needs to be updated, only the test data module needs to be adjusted, without modifying the test cases themselves. Furthermore, when the test case logic is optimized, it will not affect the test data storage.

[0017] In an optional implementation, the method further includes: capturing debugging information during the test process when executing the test script corresponding to the test case; obtaining network request data from the running log of the terminal device where the application is located; recording the entire operation trajectory of the application test process; setting a unified timestamp for the debugging information, network request data, and the entire operation trajectory, so as to query the data information during the application test process based on the timestamp and generate a test status report.

[0018] This invention acquires log data during the testing process and associates it with a unified timestamp to form a complete chain of evidence. When an anomaly occurs during testing, the operation, network request, and system status at a certain moment can be quickly located based on the timestamp, and the root cause of the problem can be accurately identified.

[0019] Secondly, the present invention provides an application testing apparatus applied to an application testing system. The application testing system includes an element component module and a test case module. The element component module stores visual feature images and associated image identifiers corresponding to each component. The test case module stores test cases generated based on the image identifiers. The apparatus includes: a feature extraction module for acquiring a screenshot of a region of the component to be tested in the application interface, extracting features from the screenshot, and outputting a first visual feature; an image capture module for acquiring the interface video stream data of the terminal device where the application is located, and extracting the current frame image; and an image feature extraction module for processing the current frame image. The system performs feature extraction and coordinate recognition on all components in the image, outputting a second visual feature and a set of coordinate mappings corresponding to the second visual feature; a visual feature matching module is used to match the first visual feature with the second visual feature, and determine the first coordinate corresponding to the first visual feature from the set of coordinate mappings based on the matching result; a test case acquisition module is used to extract the image identifier corresponding to the first visual feature from the element component module, and obtain the associated test case from the test case module based on the image identifier; a component testing module is used to test the component under test based on the associated test case and the first coordinate corresponding to the first visual feature, and generate test results.

[0020] Thirdly, the present invention provides an application testing system, which includes a controller, a terminal device where the application resides, an element component module, and a test case module. The element component module stores visual feature images and associated image identifiers corresponding to each component. The test case module stores test cases generated based on the image identifiers. The controller includes a memory and a processor, which are communicatively connected. The memory stores computer instructions, and the processor executes the computer instructions to perform the application testing method described in the first aspect or any corresponding embodiment.

[0021] In one optional implementation, the application testing system includes multiple terminal devices, and the controller establishes communication connections with each of the terminal devices via debug bridge commands.

[0022] Fourthly, the present invention provides a computer-readable storage medium storing computer instructions for causing a computer to execute the application testing method described in the first aspect or any corresponding embodiment thereof.

[0023] The present invention has the following technical effects:

[0024] The application testing method provided in this embodiment is applied to an application testing system. It includes an element component module and a test case module. The element component module stores visual feature images and associated image identifiers for each component. The test case module stores test cases generated based on the image identifiers, eliminating the need for traditional element location expressions, effectively reducing the complexity of test case development and improving test case writing efficiency. The method involves acquiring a screenshot of the area of ​​the component under test in the application interface, extracting features from the screenshot, and outputting the first visual feature; acquiring the interface video stream data of the terminal device where the application is located, and extracting the current frame image; and performing feature extraction and coordinate recognition on all components in the current frame image. Output the second visual feature and the coordinate mapping set corresponding to the second visual feature; match the first visual feature with the second visual feature, and determine the first coordinate corresponding to the first visual feature from the coordinate mapping set according to the matching result; extract the image identifier corresponding to the first visual feature from the element component module, and obtain the associated test case from the test case module based on the image identifier; test the component under test based on the associated test case and the first coordinate corresponding to the first visual feature, generate test results, realize intelligent positioning and interactive operation of user interface component elements through visual feature recognition technology, execute automated operation instructions based on coordinate information, and realize seamless connection between automated test process and test business logic. Attached Figure Description

[0025] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0026] Figure 1 This is a structural example diagram of an application testing system according to an embodiment of the present invention;

[0027] Figure 2 This is a flowchart illustrating an application testing method according to an embodiment of the present invention;

[0028] Figure 3 This is a flowchart illustrating another application testing method according to an embodiment of the present invention;

[0029] Figure 4 This is an example diagram of the testing process for a text component according to an embodiment of the present invention;

[0030] Figure 5 This is a schematic diagram of the structure of an application testing system according to an embodiment of the present invention;

[0031] Figure 6 This is a structural block diagram of an application testing apparatus according to an embodiment of the present invention;

[0032] Figure 7 This is a schematic diagram of the hardware structure of the controller according to an embodiment of the present invention. Detailed Implementation

[0033] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0034] Current automated testing tools for mobile apps (such as testing frameworks like Airtest Project Airtest) are based on a no-code model using image recognition. While they reduce coding requirements by matching and locating elements through screenshots, they have the following drawbacks: Low pure text recognition rate: The accuracy rate for recognizing text-based buttons and links is less than 60%, requiring the use of automation framework plugins (PocoUI Automation Framework, Poco) to supplement the code, which contradicts the original intention of "zero code"; Resolution sensitivity: Differences in screen sizes across different devices increase the failure rate of image matching, requiring manual adjustment of screenshot or scaling parameters.

[0035] According to an embodiment of the present invention, an application testing method embodiment is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0036] This embodiment provides an application testing method applied to an application testing system. The application testing system includes an element component module and a test case module. The element component module stores visual feature images and associated image identifiers corresponding to each component, and the test case module stores test cases generated based on the image identifiers.

[0037] like Figure 1As shown, the element component module designed in this embodiment of the invention is used to implement multimodal management of component elements in an application interface. It includes a component image library, which stores visual feature images and associated image identifiers corresponding to each component. The visual features are text semantics, icon outline proportions, textures, etc., rather than low-level features that depend on specific pixel size or resolution. The visual feature images can be images with text features. This element component module can provide a text element screenshot import interface (such as capturing the image of the "Login" button) and support expanding the component image library through image acquisition. The visual feature images can also be images with non-text features. By directly and flexibly importing the visual feature images into the component image library, the image library can be expanded without code configuration or complex operations, which meets the usability requirements of the "no-code" mode and facilitates updating the component image library based on visual features. The element component module also stores image identifiers associated with each visual feature image. These identifiers are uniquely associated with the component elements. The form of the image identifiers is not limited and can be unique image IDs, image names, hash values, etc., which are only examples.

[0038] The test case module designed in this embodiment stores test cases generated based on image identifiers. Each test case contains an "image identifier" field, which corresponds to a unique image identifier of a visual feature image. Test cases associate elements through image identifiers, eliminating the need to write positioning code. Even if the style or position of an element changes, as long as the image identifier remains unchanged, the test cases can still execute normally. That is, when the visual feature corresponding to the image identifier changes in the element component module (e.g., the positioning changes), the test cases do not need to be modified. They can be associated with the changed visual feature simply through the image identifier. By maintaining and updating the preset element image library, the decoupling management of script logic and interface elements is achieved, solving the technical problem of the complexity of maintaining dynamic interface elements.

[0039] Figure 2 This is a flowchart of an application testing method according to an embodiment of the present invention, such as... Figure 2 As shown, the process includes the following steps:

[0040] Step S201: Obtain a screenshot of the region of the component to be tested in the application interface, extract features from the region screenshot, and output the first visual features.

[0041] This invention does not limit the method for determining the component to be tested in the APP interface. It can be that the tester uses a visual tool (such as an interface selection tool) to select the component to be tested (such as the "account and password" input box, application icon, etc.) on the APP interface, and the system can automatically capture a screenshot of the component to be tested in the selected area through an image acquisition device. Alternatively, the system can automatically scan the current interface according to preset rules (such as identifying all button, input box and other component types in the interface), mark potential components to be tested, and capture their area screenshots in batches for subsequent testing. This is just an example.

[0042] This invention embodiment can perform screenshot preprocessing on the region screenshot, such as cropping invalid pixels at the edges of the screenshot, retaining the core area of ​​the component to be tested, or converting it to a unified image format. Then, feature extraction can be performed on the preprocessed region screenshot to output the first visual feature. In this case, a multimodal feature fusion strategy can be used for feature extraction. Text content features are extracted from the region screenshot to identify the text content in the screenshot. At the same time, non-text content features are extracted from the region screenshot to extract the outline lines, texture features, etc. of the component to be tested. Finally, the text content and non-text features are integrated into a unified first visual feature. This is just an example.

[0043] Step S202: Obtain the interface video stream data of the terminal device where the application is located, and extract the current frame image.

[0044] This invention allows the deployment of a real-time screen stream capture tool, such as a mini-capture tool, on the terminal device where the application resides. Minicap can be used to acquire real-time interface video stream data from the terminal device and capture the current frame image of the video stream.

[0045] Step S203: Perform feature extraction and coordinate recognition on all components in the current frame image, and output the second visual feature and the coordinate mapping set corresponding to the second visual feature.

[0046] In this embodiment of the invention, all independent components can be identified and separated from the current frame image first. Then, feature extraction and coordinate recognition are performed on all components in the current frame image. The feature extraction method can refer to the multimodal feature fusion strategy described above, and will not be repeated here. The coordinate recognition method can be to take the upper left corner of the current frame image as the origin, calculate the center coordinates (as the target point for subsequent simulation operations) and boundary coordinates (used to record the complete boundary box and determine the positional relationship of the components) based on the component bounding box. Then, the second visual features of all components and the corresponding coordinate information can be associated to form a coordinate mapping set, which is only an example.

[0047] Step S204: Match the first visual feature with the second visual feature, and determine the first coordinate corresponding to the first visual feature from the coordinate mapping set based on the matching result.

[0048] In this embodiment of the invention, a first visual feature can be matched with a second visual feature in the coordinate mapping set to obtain a matching result, that is, a second visual feature that is successfully matched with the first visual feature. Then, the first coordinates corresponding to the first visual feature can be determined from the coordinate mapping set based on the matching result.

[0049] Step S205: Extract the image identifier corresponding to the first visual feature from the element component module, and obtain the associated test case from the test case module based on the image identifier.

[0050] This invention compares a first visual feature with all visual feature images in the element component module to find the visual feature image with the highest similarity. The image identifier corresponding to the most similar visual feature image is used as the image identifier associated with the first visual feature. If the first visual feature matches multiple similar visual feature images (such as different styles of "login" buttons), they can be sorted by similarity and returned. The system can automatically select the highest matching item or prompt manual confirmation. Then, all test cases containing the image identifier can be filtered out. Further filtering can be performed based on the test scenario and priority to obtain associated test cases. As an example, the image identifier is used as an intermediate key to decouple test cases from component features. Even if the component style changes, as long as the image identifier remains unchanged, the test cases can still be associated. Moreover, no manual writing of association logic is required throughout the process. It is completed automatically through visual feature matching and image identifier acquisition, which meets the requirements of no-code mode.

[0051] Step S206: Based on the associated test cases and the first coordinates corresponding to the first visual feature, test the component under test and generate test results.

[0052] In this embodiment of the invention, the associated test cases can first be parsed into executable instructions, including operation type, operation parameters, target coordinates, and expected results. Then, based on the parsed test case steps and the first coordinates, automated simulation operations can be executed through Android Debug Bridge (ADB) instructions. After the simulation operation is executed, the actual results can be compared with the expected results. For example, in visual feature verification, the interface image after the operation is captured, the visual features of the expected verification object are extracted, and matched with the features in the current frame image to determine whether the visual features of the verification object exist and whether the position meets the expectations. As an example, the verification results can finally be integrated into a standardized test report.

[0053] The application testing method provided in this embodiment is applied to an application testing system. It includes an element component module and a test case module. The element component module stores visual feature images and associated image identifiers for each component. The test case module stores test cases generated based on the image identifiers, eliminating the need for traditional element location expressions, effectively reducing the complexity of test case development and improving test case writing efficiency. The method involves acquiring a screenshot of the area of ​​the component under test in the application interface, extracting features from the screenshot, and outputting the first visual feature; acquiring the interface video stream data of the terminal device where the application is located, and extracting the current frame image; and performing feature extraction and coordinate recognition on all components in the current frame image. Output the second visual feature and the coordinate mapping set corresponding to the second visual feature; match the first visual feature with the second visual feature, and determine the first coordinate corresponding to the first visual feature from the coordinate mapping set according to the matching result; extract the image identifier corresponding to the first visual feature from the element component module, and obtain the associated test case from the test case module based on the image identifier; test the component under test based on the associated test case and the first coordinate corresponding to the first visual feature, generate test results, realize intelligent positioning and interactive operation of user interface component elements through visual feature recognition technology, execute automated operation instructions based on coordinate information, and realize seamless connection between automated test process and test business logic.

[0054] This embodiment provides an application testing method applied to an application testing system. The application testing system includes an element component module and a test case module. The element component module stores visual feature images and associated image identifiers corresponding to each component, while the test case module stores test cases generated based on the image identifiers. Figure 3 This is a flowchart of an application testing method according to an embodiment of the present invention, such as... Figure 3 As shown, the process includes the following steps:

[0055] Step S301: Obtain a screenshot of the area of ​​the component to be tested in the application interface, extract features from the area screenshot, and output the first visual features.

[0056] Specifically, the first visual feature includes image visual features and / or text visual features. Based on preset component type determination rules, the component to be tested in the screenshot is determined to be a text component and / or an image component. Using a preset text recognition engine, text elements are extracted from the text component in the screenshot, and text visual features are output. Using a preset image feature recognition engine, image visual elements are extracted from the image component in the screenshot, and image visual features are output. The text visual features and / or image visual features are integrated to output the first visual feature.

[0057] Existing pixel-based feature matching techniques for text recognition suffer from insufficient semantic understanding, leading to inadequate text morphological resilience. For instance, in tests of text controls containing standard fonts like Song and Hei, subtle differences in font rendering engines (significant deviations in anti-aliasing levels and font weight parameters) or a lower-than-ideal contrast between the background color and text can cause a drop in the false recognition rate of text recognition technology, potentially reducing the pass rate of test cases for critical business processes. Therefore, this invention designs a text recognition engine to extract, arrange, and locate text elements, significantly improving the accuracy of image text recognition. Furthermore, considering that components include not only text content but also image features, an image feature recognition engine can also be used to recognize patterns in non-text user interface (UI) elements. This dual-engine collaborative operation can cover the recognition needs of all interface elements across the entire application scenario.

[0058] This invention embodiment can pre-determine whether a component to be tested in a screenshot is a text component and / or an image component based on a preset component type determination rule. The preset component type determination rule is not limited. For example, if a pixel distribution analysis is performed on the screenshot, and a continuous edge contour is detected that contains high-frequency texture features, then it is determined that the component includes image features and is an image component. If a regular character contour is detected that conforms to a text arrangement distribution model, then it is determined that the component includes text features and is a text component. If the component to be tested includes both image features and text features, then the component to be tested is determined to be a text-image component. This is just an example.

[0059] like Figure 1 As shown, the application testing system also includes an image recognition module, specifically an image feature recognition engine and a text recognition engine (Optical Character Recognition, OCR). In this embodiment, the preset text recognition engine can extract text elements from text components in a screenshot. The preset text recognition engine can utilize deep learning algorithms (text recognition models) to extract, arrange, and locate text elements, initialize the text recognition model, load pre-trained text detection and recognition parameters, and input the screenshot into the text recognition model to obtain text elements. Simultaneously, the preset image feature recognition engine can extract visual elements from image components in the screenshot and output visual image features. The image feature recognition engine can utilize convolutional neural networks to achieve pattern recognition of non-text UI elements.

[0060] In specific embodiments, such as Figure 4As shown, the text component testing method includes: establishing an ADB communication link, establishing a control connection with the terminal device under test, acquiring a screenshot of the region of the text component under test through an image acquisition device, initializing a text recognition model, loading pre-trained text detection and recognition parameters, inputting the screenshot of the region of the text component under test into the text recognition model, and outputting the first text visual feature; deploying and implementing the screen stream acquisition tool minicap on the terminal device, acquiring real-time video stream data of the terminal device through minicap, capturing the current frame image of the video stream, performing text recognition processing on the current frame image, generating a second text visual feature and a corresponding coordinate mapping set, matching the first text visual feature with the second text visual feature, determining the first coordinate corresponding to the first text visual feature, obtaining the corresponding image identifier from the element component module based on the first visual feature, obtaining the corresponding test case based on the image identifier, and sending a test operation event to the terminal device through ADB commands based on the test case, with the event coordinate being the first coordinate. For detailed explanation, please refer to the above embodiment, which will not be repeated here.

[0061] Step S302: Obtain the interface video stream data of the terminal device where the application is located, and extract the current frame image. For details, please refer to [link to relevant documentation]. Figure 2 Step S202 of the illustrated embodiment will not be described again here.

[0062] Step S303: Perform feature extraction and coordinate recognition on all components in the current frame image, and output the second visual feature and the coordinate mapping set corresponding to the second visual feature. The method of feature extraction on the components can be found in the above embodiment, and will not be repeated here.

[0063] Step S304: Match the first visual feature with the second visual feature, and determine the first coordinate corresponding to the first visual feature from the coordinate mapping set based on the matching result. See details below. Figure 2 Step S204 of the illustrated embodiment will not be described again here.

[0064] Step S305: Extract the image identifier corresponding to the first visual feature from the element component module, and obtain the associated test case from the test case module based on the image identifier.

[0065] Specifically, image identification is image naming; the element component module is configured through the following steps: a tree-like directory structure is built in the element component module, where each directory level corresponds to the navigation node of each application page in the application; the visual feature images corresponding to all components contained in the current application page are stored in the corresponding hierarchical directory, and the visual feature images include text visual feature images and image visual feature images; the stored visual feature images are named using naming rules associated with the semantics of component functions to obtain the element component module.

[0066] like Figure 1 As shown, the application testing system designed in this embodiment of the invention also includes an element component module, which is used to implement multimodal management of application interface elements. A tree-shaped directory structure can be constructed in the component image library in the element component module. Each directory level corresponds to the navigation node of each application page in the application, which is used to organize the image data of page elements at each level of the application. Then, the visual feature images (including text visual feature images and image visual feature images) corresponding to all components contained in the current application page can be stored in the directory corresponding to the navigation node of the application page. The stored visual feature images can be named using naming rules associated with the semantics of component functions, thereby generating the configured element component module.

[0067] This invention designs a tree-structured directory component managed by page navigation nodes, which can directly locate target components based on the directory hierarchy, avoiding full database scanning and improving component matching efficiency.

[0068] In one alternative implementation, the system monitors updates to components within the application; when a change is detected in a component, the corresponding visual feature image in the element component module is modified.

[0069] This invention can monitor attribute change events of each component in an application in real time, including at least the component's size, text, text content, and visual style. When an attribute change event is detected, the visual feature changes of the component before and after the change can be compared. For components with visual feature changes, their current visual feature image can be captured, and the storage path of the component can be located in the element component module. Finally, the captured new visual feature image is used to replace the element component module. This is just an example.

[0070] When a change is detected in a component, this invention can modify the visual feature image in the element component module in real time to provide a real-time updated component visual image. This can flexibly respond to the dynamic change requirements of complex component elements. Furthermore, by accurately locating the component storage path and replacing it in a targeted manner, it avoids invalid resource occupation and achieves refined management of component visual resources.

[0071] Furthermore, when a change in the visual features of the current component is detected, features are extracted from the current component, a third visual feature image is output, and the third visual feature image replaces the visual feature image corresponding to the current component in the element component module; or, when the current component is detected to be deleted, the visual feature image corresponding to the current component in the element component module is deleted; or, when a new component is detected, features are extracted from the new component, a fourth visual feature image is output, and the fourth visual feature image is added to the corresponding hierarchical directory in the element component module.

[0072] In this embodiment of the invention, when a change in the visual features of the current component is detected, features of the current component can be extracted, a third visual feature image can be output, and the third visual feature image can replace the visual feature image corresponding to the current component in the element component module. Alternatively, when the current component is detected to be deleted, the current component can be located in the tree directory, and its visual feature image can be deleted while its associated semantic name is deleted simultaneously. Alternatively, when a new component is detected, features of the new component can be extracted to obtain a fourth visual feature image, and the page hierarchy of the new component can be determined. Then, the naming rules associated with the component's functional semantics can be called to generate an image name for the visual feature image of the new component, and the fourth visual feature image and the name can be stored together under the page hierarchy. This is only an example.

[0073] This invention employs operations of replacing, deleting, and adding visual feature images for the three states of component updates, deletions, and additions, respectively, to ensure that the visual feature images stored in the element component module are completely consistent with the component states during actual application runtime, thus guaranteeing the real-time synchronization of visual resources.

[0074] Step S306: Based on the associated test cases and the first coordinates corresponding to the first visual feature, test the component under test and generate test results.

[0075] like Figure 1 As shown, the application testing system designed in this embodiment of the invention also includes a general business module, which includes a business logic layer for constructing atomic operation sequences of test cases, determining the validity of test results through an assertion verification mechanism, and generating test results containing pass / fail statuses.

[0076] Specifically, such as Figure 1 As shown, the testing system also includes a data storage module, which stores the test data required by each test case. This module is separate from the test case module, meaning the test data can be accessed independently of the business module. This allows testers to freely select and combine data to generate test scripts according to actual needs. Step S306 includes:

[0077] Step S3061: Execute the test instructions in the test cases in sequence.

[0078] Step S3062: When a test instruction including a data request flag is detected, the corresponding test data is obtained from the data storage module.

[0079] Step S3063: Based on the first coordinate, inject the test data into the component under test for testing.

[0080] This invention can execute test operations according to the preset order of instructions in the test cases. When the current test instruction contains a data request marker, a data acquisition operation can be triggered to obtain the test dataset (such as account password) associated with the data request marker from the data storage module. Then, based on the first coordinates corresponding to the first visual feature, the specific component to be tested (such as an input box) is located, and the test dataset is injected into the located component. This coordinate-based precise injection ensures targeted testing of specific components, improving testing efficiency and accuracy. Subsequently, the business logic of the component is triggered, and the output results of the component are monitored to complete the test verification of the component's function.

[0081] This invention designs the test case module and data storage module to exist independently, so that when test data needs to be updated, only the test data module needs to be adjusted, without modifying the test cases themselves. Furthermore, when the test case logic is optimized, it will not affect the test data storage.

[0082] In one optional implementation, when executing the test script corresponding to the test case, the debugging information during the test process is captured; the network logcat request data in the running log of the terminal device where the application is located is obtained; the entire operation trajectory of the application test process is recorded; and a unified timestamp is set for the debugging information, network request data and the entire operation trajectory, so as to query the data information of the application test process based on the timestamp and generate a test status report.

[0083] like Figure 1 As shown, the application testing system designed in this embodiment of the invention also includes a general business module, which includes a log monitoring service unit. During the execution of the test script corresponding to the test case, the system can capture debugging information during the test process through a script execution log recorder, obtain network request data in the terminal device's operation log in real time through a device operation log collector, and record the entire process of the application testing process through a screen recording unit using video encoding technology. Finally, the system can align and associate the obtained operation process screenshot sequence, terminal device screen recording file, device operation log data packet, and script execution log record, etc., through timestamps to form a traceable test process reproduction system. This facilitates subsequent querying of the evidence chain files in the application testing process based on timestamps, and generating test cases, data, or system test status reports. This is just an example.

[0084] This invention acquires log data during the testing process and associates it with a unified timestamp to form a complete chain of evidence. When an anomaly occurs during testing, the operation, network request, and system status at a certain moment can be quickly located based on the timestamp, and the root cause of the problem can be accurately identified.

[0085] In an optional implementation, the general business module designed in this embodiment of the invention further includes a device cluster management unit, which supports parallel access and status monitoring of multiple terminal devices. These terminal devices can establish communication connections with the controller in the system via the ADB protocol. Furthermore, the component image library has cross-platform compatibility. Image template libraries can be established for terminal devices operating different operating systems. Specifically, the system acquires the region to be identified from the current screenshot of the terminal device under test, extracts the visual features of this region, and calculates the similarity between these visual features and a benchmark model in the standardized image template library. When the similarity exceeds a dynamic adaptation threshold, a successful match is determined. This visual feature image matching mechanism ensures the stable execution of automated test scripts on heterogeneous terminal devices, eliminating the script adaptation development requirements caused by device differences.

[0086] The technical effects of this invention are reflected in significantly improving the accuracy of image text recognition (reaching over 98.6%), reducing the complexity of test script development by over 60%, achieving 100% cross-platform test case reuse rate, and effectively solving the technical bottlenecks of dynamic element location failure and high multi-device adaptation costs in traditional automated testing.

[0087] This embodiment also provides an application testing system, such as Figure 5 As shown, the application testing system includes a controller 51, a terminal device 52 where the application resides, an element component module 53, and a test case module 54. The element component module stores visual feature images and associated image identifiers corresponding to each component. The test case module stores test cases generated based on the image identifiers. The controller includes a memory and a processor, which are interconnected. The memory stores computer instructions, and the processor executes the application testing method described in the above embodiment by executing the computer instructions.

[0088] Furthermore, the application testing system includes multiple terminal devices. The controller establishes communication connections with each of the multiple terminal devices through the debug bridge ADB command. For details, please refer to the above embodiment, which will not be repeated here.

[0089] This embodiment also provides an application testing apparatus for implementing the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the apparatus described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0090] This embodiment provides an application testing device applied to an application testing system. The application testing system includes an element component module and a test case module. The element component module stores visual feature images and associated image identifiers corresponding to each component, and the test case module stores test cases generated based on the image identifiers, such as... Figure 6 As shown, the system includes: a feature extraction module 601, used to acquire a screenshot of the area of ​​the component under test in the application interface, extract features from the screenshot, and output a first visual feature; an image capture module 602, used to acquire the interface video stream data of the terminal device where the application is located, and extract the current frame image; an image feature extraction module 603, used to extract features and recognize coordinates of all components in the current frame image, and output a second visual feature and a set of coordinate mappings corresponding to the second visual feature; a visual feature matching module 604, used to match the first visual feature with the second visual feature, and determine the first coordinate corresponding to the first visual feature from the set of coordinate mappings based on the matching result; a test case acquisition module 605, used to extract the image identifier corresponding to the first visual feature from the element component module, and obtain the associated test case from the test case module based on the image identifier; and a component testing module 606, used to test the component under test based on the associated test case and the first coordinate corresponding to the first visual feature, and generate test results.

[0091] In some optional implementations, the first visual feature includes image visual features and / or text visual features. The feature extraction module 601 includes: a component determination unit, used to determine whether the component to be tested in the area screenshot is a text component and / or an image component based on a preset component type determination rule; a text feature recognition unit, used to extract text elements from the text component in the area screenshot using a preset text recognition engine, and output text visual features; an image feature recognition unit, used to extract image visual elements from the image component in the area screenshot using a preset image feature recognition engine, and output image visual features; and a visual feature output unit, used to integrate the text visual features and / or image visual features, and output the first visual feature.

[0092] In some optional implementations, image identification is image naming, and the element component module is configured through the following steps: a tree-like directory structure is constructed in the element component module, wherein each directory level corresponds to the navigation node of each application page in the application; visual feature images corresponding to all components contained in the current application page are stored in the corresponding hierarchical directory, the visual feature images including text visual feature images and image visual feature images; the stored visual feature images are named using naming rules associated with the component's functional semantics, thus obtaining the element component module.

[0093] In some optional implementations, the application testing apparatus further includes: a component change monitoring module for monitoring updates and changes of components in the application; and a feature image modification module for modifying the corresponding visual feature image in the element component module when a change is detected in a component.

[0094] In some optional implementations, the feature image modification module includes: a feature modification unit, configured to extract features from the current component and output a third visual feature image when a change in the visual features of the current component is detected, and replace the visual feature image corresponding to the current component in the element component module with the third visual feature image; or, a feature image deletion unit, configured to delete the visual feature image corresponding to the current component in the element component module when a deletion of the current component is detected; or, a feature image addition module, configured to extract features from the addition component and output a fourth visual feature image when a addition of the component is detected, and add the fourth visual feature image to the corresponding hierarchical directory in the element component module.

[0095] In some optional implementations, the test system further includes a data storage module that stores the test data required for each test case. The component test module 606 includes: a test instruction execution unit for executing test instructions in the test cases in sequence; a test data acquisition unit for acquiring the corresponding test data from the data storage module when a test instruction includes a data request flag; and a test data injection unit for injecting the test data into the component under test for testing based on a first coordinate.

[0096] In some optional implementations, the application testing apparatus further includes: a debugging information acquisition module, used to capture debugging information during the testing process when executing the test script corresponding to the test case; a request data acquisition module, used to acquire network request data in the running log of the terminal device where the application is located; an operation trajectory recording module, used to record the entire operation trajectory of the application testing process; and an evidence chain generation module, used to set a unified timestamp for the debugging information, network request data, and the entire operation trajectory, so as to query the data information during the application testing process based on the timestamp and generate a test status report.

[0097] Further functional descriptions of the above modules and units are the same as those in the corresponding embodiments described above, and will not be repeated here.

[0098] In this embodiment, the application testing device is presented in the form of a functional unit. Here, a unit refers to an ASIC (Application Specific Integrated Circuit), a processor and memory that execute one or more software or fixed programs, and / or other devices that can provide the above functions.

[0099] This invention also provides a controller 51 in an application testing system, having the above-described features. Figure 6 The application testing device shown.

[0100] Please see Figure 7 , Figure 7 This is a schematic diagram of the structure of controller 51 in an application testing system provided by an optional embodiment of the present invention, as shown below. Figure 7 As shown, the controller includes one or more processors 10, memory 20, and interfaces for connecting the components, including high-speed interfaces and low-speed interfaces. The components communicate with each other via different buses and can be mounted on a common motherboard or otherwise as required. The processors can process instructions executed within the controller, including instructions stored in or on memory to display graphical information of a GUI on external input / output devices (such as display devices coupled to the interfaces). In some alternative implementations, multiple processors and / or multiple buses can be used with multiple memories and multiple memory modules, if desired. Similarly, multiple controllers can be connected, each providing some of the necessary operations (e.g., as a server array, a group of blade servers, or a multiprocessor system). Figure 7 Take a processor 10 as an example.

[0101] Processor 10 may be a central processing unit, a network processor, or a combination thereof. Processor 10 may further include a hardware chip. The hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The programmable logic device may be a complex programmable logic device (CAMP), a field-programmable gate array (FPGA), a general-purpose array logic (GPA), or any combination thereof.

[0102] The memory 20 stores instructions executable by at least one processor 10 to cause at least one processor 10 to perform the method shown in the above embodiments.

[0103] The memory 20 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the controller. Furthermore, the memory 20 may include high-speed random access memory and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some alternative embodiments, the memory 20 may optionally include memory remotely located relative to the processor 10, and these remote memories may be connected to the controller via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0104] The memory 20 may include volatile memory, such as random access memory; the memory may also include non-volatile memory, such as flash memory, hard disk or solid-state drive; the memory 20 may also include a combination of the above types of memory.

[0105] The controller also includes an input device 30 and an output device 40. The processor 10, memory 20, input device 30, and output device 40 can be connected via a bus or other means. Figure 7 Taking the example of a connection between China and Israel via a bus.

[0106] Input device 30 can receive input numerical or character information, and generate key signal inputs related to user settings and function control of the controller, such as a touch screen, keypad, mouse, trackpad, touchpad, joystick, one or more mouse buttons, trackball, joystick, etc. Output device 40 may include display devices, auxiliary lighting devices (e.g., LEDs), and haptic feedback devices (e.g., vibration motors). The aforementioned display devices include, but are not limited to, liquid crystal displays, light-emitting diodes, displays, and plasma displays. In some alternative embodiments, the display device may be a touch screen.

[0107] This invention also provides a computer-readable storage medium. The methods described above according to embodiments of the invention can be implemented in hardware or firmware, or implemented as computer code that can be recorded on a storage medium, or implemented as computer code downloaded via a network and originally stored on a remote storage medium or a non-transitory machine-readable storage medium and then stored on a local storage medium. Thus, the methods described herein can be processed by software stored on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. The storage medium can be a magnetic disk, optical disk, read-only memory, random access memory, flash memory, hard disk, or solid-state drive, etc.; further, the storage medium can also include combinations of the above types of memory. It is understood that computers, processors, microprocessor controllers, or programmable hardware include storage components capable of storing or receiving software or computer code, which, when accessed and executed by the computer, processor, or hardware, implements the methods shown in the above embodiments.

[0108] A portion of this invention can be applied as a computer program product, such as computer program instructions, which, when executed by a computer, can invoke or provide the methods and / or technical solutions according to the invention through the operation of the computer. Those skilled in the art will understand that the forms in which computer program instructions exist in a computer-readable medium include, but are not limited to, source files, executable files, installation package files, etc. Correspondingly, the ways in which computer program instructions are executed by a computer include, but are not limited to: the computer directly executing the instructions, or the computer compiling the instructions and then executing the corresponding compiled program, or the computer reading and executing the instructions, or the computer reading and installing the instructions and then executing the corresponding installed program. Here, the computer-readable medium can be any available computer-readable storage medium or communication medium accessible to a computer.

[0109] Although embodiments of the invention have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the invention, and such modifications and variations all fall within the scope defined by the appended claims.

Claims

1. An application testing method, characterized in that, An application testing system is used, comprising an element component module and a test case module. The element component module stores visual feature images and associated image identifiers corresponding to each component, and the test case module stores test cases generated based on the image identifiers. The method includes: Obtain a screenshot of the area of ​​the component to be tested in the application interface, extract features from the screenshot, and output the first visual features; Obtain the video stream data of the terminal device where the application is located, and extract the current frame image; Perform feature extraction and coordinate recognition on all components in the current frame image, and output a second visual feature and a set of coordinate mappings corresponding to the second visual feature; The first visual feature is matched with the second visual feature, and the first coordinate corresponding to the first visual feature is determined from the coordinate mapping set based on the matching result; Extract the image identifier corresponding to the first visual feature from the element component module, and obtain the associated test case from the test case module based on the image identifier; Based on the associated test cases and the first coordinates corresponding to the first visual feature, the component under test is tested, and test results are generated.

2. The method according to claim 1, characterized in that, The first visual feature includes image visual features and / or text visual features. Feature extraction is performed on a screenshot of the region of the component under test, and the first visual feature is output, including: Based on the preset component type determination rules, it is determined that the component to be tested in the screenshot of the region is a text component and / or an image component. Using a preset text recognition engine, text elements are extracted from the text components in the screenshot of the region, and the visual features of the text are output. Using a preset image feature recognition engine, the visual elements of the image components in the screenshot of the region are extracted, and the visual features of the image are output. The text visual features and / or image visual features are integrated to output the first visual feature.

3. The method according to claim 1, characterized in that, The image identifier is the image name, and the element component module is configured through the following steps: Build a tree-like directory structure in the element component module, where each directory level corresponds to the navigation node of each application page in the application; The visual feature images corresponding to all components contained in the current application page are stored in the corresponding hierarchical directory. The visual feature images include text visual feature images and image visual feature images. The stored visual feature images are named using naming rules associated with component functional semantics to obtain the element component modules.

4. The method according to claim 3, characterized in that, The method further includes: Monitor updates and changes to components within the application; When a change is detected in a component, the corresponding visual feature image in the element component module is modified.

5. The method according to claim 4, characterized in that, The step of modifying the corresponding visual feature image in the element component module when a change in a component is detected includes: When a change in the visual features of the current component is detected, feature extraction is performed on the current component, a third visual feature image is output, and the third visual feature image replaces the visual feature image corresponding to the current component in the element component module; or, When the deletion of the current component is detected, the visual feature image corresponding to the current component in the element component module is deleted, or... When a new component is detected, features are extracted from the new component, a fourth visual feature image is output, and the fourth visual feature image is added to the corresponding hierarchical directory in the element component module.

6. The method according to claim 1, characterized in that, The testing system further includes a data storage module, which stores the test data required for each test case. The testing of the component under test based on the associated test cases and the first coordinates corresponding to the first visual feature includes: Execute the test instructions in the test cases sequentially; When the test instruction includes a data request flag, the corresponding test data is obtained from the data storage module; Based on the first coordinate, the test data is injected into the component under test for testing.

7. The method according to claim 1, characterized in that, The method further includes: When executing the test script corresponding to the test case, capture debugging information during the test process; Obtain network request data from the runtime logs of the terminal device where the application is located; Record the entire operation process of application testing; A unified timestamp is set for the debugging information, network request data, and the entire process operation trajectory, so as to query the data information during the application testing process based on the timestamp and generate a test status report.

8. An application testing device, characterized in that, An application testing system is used, comprising an element component module and a test case module. The element component module stores visual feature images and associated image identifiers corresponding to each component, and the test case module stores test cases generated based on the image identifiers. The device includes: The feature extraction module is used to obtain a screenshot of the region of the component to be tested in the application interface, extract features from the screenshot, and output the first visual features. The image capture module is used to acquire the interface video stream data of the terminal device where the application is located, and extract the current frame image; The image feature extraction module is used to extract features and recognize coordinates of all components in the current frame image, and output a second visual feature and a set of coordinate mappings corresponding to the second visual feature; A visual feature matching module is used to match the first visual feature with the second visual feature, and determine the first coordinate corresponding to the first visual feature from the coordinate mapping set according to the matching result; The test case acquisition module is used to extract the image identifier corresponding to the first visual feature from the element component module, and obtain the associated test case from the test case module based on the image identifier; The component testing module is used to test the component under test based on the associated test cases and the first coordinates corresponding to the first visual feature, and generate test results.

9. An application testing system, characterized in that, The application testing system includes a controller, a terminal device where the application resides, an element component module, and a test case module. The element component module stores visual feature images and associated image identifiers corresponding to each component. The test case module stores test cases generated based on the image identifiers. The controller includes a memory and a processor, which are interconnected. The memory stores computer instructions, and the processor executes the computer instructions to perform the application testing method according to any one of claims 1 to 7.

10. The system according to claim 9, characterized in that, The application testing system includes multiple terminal devices, and the controller establishes communication connections with each of the terminal devices through debug bridge commands.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing the computer to execute the application testing method according to any one of claims 1 to 7.

Images