Multi-dimensional perceptual analysis cross-platform automatic testing system and multi-dimensional perceptual analysis cross-platform automatic testing method

Abstract

The invention relates to the technical field of software testing and automatic control, and discloses a cross-platform automatic testing system and method for multi-dimensional perceptual analysis. A cross-platform execution layer obtains running interface data of running of a tested application; the multi-dimensional perception recognition analysis layer executes corresponding multi-dimensional perception analysis operation on the operation interface data according to the multi-dimensional perception analysis model, and generates a candidate element set corresponding to the operation interface data; the comprehensive target positioning layer determines a target element set corresponding to the operation interface data according to a preset multi-dimensional feature similarity comprehensive calculation model and the candidate element set; and the cross-platform execution layer executes an application test operation matched with the tested application according to the target element set and a pre-analyzed test instruction set. Therefore, the flexibility and robustness of the cross-platform automatic test can be improved in a multi-modal manner.

Description

Technical Field

[0001] This invention relates to the fields of software testing and automated control technology, and in particular to a cross-platform automated testing system and method for multi-dimensional perception and analysis. Background Technology

[0002] With the rapid development of mobile internet and IoT technologies, various applications need to run stably on multiple hardware devices and operating systems. To ensure the consistency, stability, and user experience of applications across different platform environments, automated testing technology has become an indispensable part of the software development lifecycle.

[0003] Traditional automated testing methods, such as those based on control tree recognition (e.g., UI Automator for Android, XCUITest for iOS) or image recognition (e.g., SikuliX), face significant challenges in practical cross-platform testing. Control tree-based methods heavily rely on platform-specific accessibility interfaces to obtain UI element attributes (e.g., ID, text, coordinates). However, in cross-platform scenarios, UI elements of the same application often have different internal identifiers on different platforms. Furthermore, for games, applications heavily using custom controls, or partially hybrid applications, control tree information may be incomplete or unavailable, making test scripts unusable across platforms and resulting in high maintenance costs. Methods based on fixed coordinates or static image template matching struggle to adapt to different screen resolutions, aspect ratios, and dynamically changing UI content, exhibiting poor robustness.

[0004] Therefore, it is particularly important to propose a technical solution to improve the flexibility and robustness of cross-platform automated testing. Summary of the Invention

[0005] This invention provides a cross-platform automated testing system and method for multi-dimensional perception analysis, which can improve the flexibility and robustness of cross-platform automated testing.

[0006] To address the aforementioned technical problems, the first aspect of this invention discloses a cross-platform automated testing system for multi-dimensional perception analysis, the system comprising: A cross-platform execution layer used to obtain runtime interface data of the application under test; The multi-dimensional perception and recognition analysis layer is used to perform corresponding multi-dimensional perception and analysis operations on the running interface data according to the multi-dimensional perception and analysis model, and generate a candidate element set corresponding to the running interface data. The multi-dimensional perception and analysis operations include at least two of the following: text recognition and analysis, semantic understanding and analysis, icon recognition and analysis, interface layout recognition and analysis, and action intent prediction and analysis. The comprehensive target localization layer is used to determine the target element set corresponding to the running interface data based on the preset multi-dimensional feature similarity comprehensive calculation model and the candidate element set; The cross-platform execution layer is also used to execute application test operations adapted to the application under test based on the target element set and the pre-parsed test instruction set.

[0007] As an optional implementation, in the first aspect of the present invention, the multi-dimensional perception analysis model includes at least a multi-dimensional feature recognition analysis model, a multi-dimensional semantic understanding analysis model, and an operation intent prediction analysis model. The multi-dimensional perception analysis model is matched with the operation. The multi-dimensional perception recognition analysis layer performs a corresponding multi-dimensional perception analysis operation on the running interface data according to the multi-dimensional perception analysis model. The specific method for generating the candidate element set corresponding to the running interface data includes: Based on the multi-dimensional feature recognition and analysis model, the interface data is subjected to interface element recognition and image denoising to obtain preliminary element recognition results. Based on the multi-dimensional semantic understanding analysis model, multimodal semantic understanding operations are performed on the preliminary element recognition results to integrate icon detection, layout analysis, and semantic partitioning features to generate a semantically enhanced element set; Based on the operation intent prediction and analysis model and the action intent instructions in the pre-parsed test instruction set, the semantic enhancement element set is subjected to intent matching and confidence calculation to generate a candidate element set corresponding to the running interface data. Each candidate element in the candidate element set includes its corresponding element attribute, element position coordinates and confidence score.

[0008] As an optional implementation, in the first aspect of the present invention, the specific method by which the comprehensive target localization layer determines the target element set corresponding to the running interface data based on a preset multi-dimensional feature similarity comprehensive calculation model and the candidate element set includes: For each candidate element in the candidate element set, similarity information between the candidate element and the target element description data in the pre-parsed test instruction set is extracted across multiple feature dimensions. The feature dimensions include at least text features, local image features, and semantic features. The similarity information on the text features is used to represent the degree of matching between the candidate element and the target element description data in the text content dimension. The similarity information on the local image features is used to represent the degree of matching between the candidate element and the target element description data in the element visual appearance dimension. The similarity information on the semantic features is used to represent the degree of semantic association between the candidate element and the target element description data in the interface context dimension. All the extracted similarity information is input into a preset multi-dimensional feature similarity comprehensive calculation model for weighted fusion calculation. The weighted fusion calculation dynamically adjusts the weight allocation of each feature dimension based on the interface context to generate a comprehensive similarity score for each element. Based on the comprehensive similarity score and the preset confidence threshold, the target element set corresponding to the running interface data is determined.

[0009] As an optional implementation, in the first aspect of the present invention, the specific method by which the cross-platform execution layer executes application testing operations adapted to the application under test based on the target element set and the pre-parsed test instruction set includes: The element position coordinates in the target element set are converted into device-independent normalized coordinate representations, which are then adapted to different screen resolutions and irregular screen display characteristics through safe area correction processing. Based on the operation types and parameters defined in the pre-parsed test instruction set, a platform adaptation component matching the running platform of the application under test is invoked to map the normalized coordinate representation into executable operation instructions for a specific platform corresponding to the application under test, so as to execute application test operations adapted to the application under test. The operation types include at least one of click, swipe, long press, double click, drag, zoom, text input, and gesture operation.

[0010] As an optional implementation, in the first aspect of the invention, the system further includes: The assertion verification layer is used to capture multi-dimensional feedback data in real time during the execution of the application test operation. The multi-dimensional feedback data includes at least one of the following: interface response status data, operation trajectory sequence, network request log data, system event log data, and performance indicator data. The assertion verification layer is also used to compare and verify the multi-dimensional feedback data with the preset assertion conditions in the test instruction set to obtain multi-dimensional verification results. The assertion verification layer is also used to generate a traceable test report of the application test operation based on the multi-dimensional verification results.

[0011] As an optional implementation, in the first aspect of the present invention, the assertion verification layer compares and verifies the multi-dimensional feedback data with the preset assertion conditions in the test instruction set to obtain the multi-dimensional verification result in the following specific ways: Cross-dimensional correlation analysis is performed on the multi-dimensional feedback data to obtain inter-dimensional correlation rules. The inter-dimensional correlation rules are used to represent the statistical dependence or temporal relationship between different dimension feedback data. The inter-dimensional correlation rules include at least one of causal relationship rules, temporal constraint rules, and statistical correlation rules. Based on the inter-dimensional association rules and the preset assertion conditions, an adaptive verification condition set is generated. The adaptive verification condition set is used to dynamically adjust the strictness or priority of verification. The adaptive verification condition set includes at least one of dynamic threshold conditions, verification priority rules, and condition combination rules. Based on the adaptive verification condition set, the multi-dimensional feedback data is subjected to hierarchical verification to generate multi-dimensional verification results.

[0012] As an optional implementation, in the first aspect of the invention, the system further includes: An adaptive closed-loop layer is used to generate a test adaptation strategy based on the multi-dimensional verification results and the test instruction set. The test adaptation strategy represents an adaptive adjustment rule for the test execution process. The test adaptation strategy is obtained by analyzing the correlation between verification anomaly events in the multi-dimensional verification results and the test instruction set. The test adaptation strategy includes at least one of test step skipping conditions, retry triggering conditions, and execution order optimization rules. The adaptive closed-loop layer is also used to dynamically reconstruct the test instruction set based on the test adaptation strategy to generate an adaptive test instruction sequence. The dynamic reconstruction is achieved by modifying the execution logic, parameters, or order of the test instructions. The adaptive closed-loop layer is also used to re-execute the application test operation according to the adaptive test instruction sequence. The application test operation adapts to the dynamic behavior changes of the application under test through the closed-loop control of the adaptive test instruction sequence and real-time test feedback.

[0013] A second aspect of this invention discloses a cross-platform automated testing method for multi-dimensional perception analysis, the method comprising: Obtain the running interface data of the application under test; Based on the multi-dimensional perception analysis model, the corresponding multi-dimensional perception analysis operation is performed on the running interface data to generate a candidate element set corresponding to the running interface data. The multi-dimensional perception analysis operation includes at least two of the following: text recognition and analysis, semantic understanding and analysis, icon recognition and analysis, interface layout recognition and analysis, and action intent prediction and analysis. Based on the preset multi-dimensional feature similarity comprehensive calculation model and the candidate element set, the target element set corresponding to the running interface data is determined; Based on the target element set and the pre-parsed test instruction set, execute application test operations adapted to the application under test.

[0014] As an optional implementation, in a second aspect of the present invention, the multi-dimensional perception analysis model includes at least a multi-dimensional feature recognition analysis model, a multi-dimensional semantic understanding analysis model, and an operation intent prediction analysis model. The multi-dimensional perception analysis model is matched with the operation. The step of performing a corresponding multi-dimensional perception analysis operation on the running interface data according to the multi-dimensional perception analysis model to generate a candidate element set corresponding to the running interface data includes: Based on the multi-dimensional feature recognition and analysis model, the interface data is subjected to interface element recognition and image denoising to obtain preliminary element recognition results. Based on the multi-dimensional semantic understanding analysis model, multimodal semantic understanding operations are performed on the preliminary element recognition results to integrate icon detection, layout analysis, and semantic partitioning features to generate a semantically enhanced element set; Based on the operation intent prediction and analysis model and the action intent instructions in the pre-parsed test instruction set, the semantic enhancement element set is subjected to intent matching and confidence calculation to generate a candidate element set corresponding to the running interface data. Each candidate element in the candidate element set includes its corresponding element attribute, element position coordinates and confidence score.

[0015] As an optional implementation, in a second aspect of the present invention, determining the target element set corresponding to the running interface data based on a preset multi-dimensional feature similarity comprehensive calculation model and the candidate element set includes: For each candidate element in the candidate element set, similarity information between the candidate element and the target element description data in the pre-parsed test instruction set is extracted across multiple feature dimensions. The feature dimensions include at least text features, local image features, and semantic features. The similarity information on the text features is used to represent the degree of matching between the candidate element and the target element description data in the text content dimension. The similarity information on the local image features is used to represent the degree of matching between the candidate element and the target element description data in the element visual appearance dimension. The similarity information on the semantic features is used to represent the degree of semantic association between the candidate element and the target element description data in the interface context dimension. All the extracted similarity information is input into a preset multi-dimensional feature similarity comprehensive calculation model for weighted fusion calculation. The weighted fusion calculation dynamically adjusts the weight allocation of each feature dimension based on the interface context to generate a comprehensive similarity score for each element. Based on the comprehensive similarity score and the preset confidence threshold, the target element set corresponding to the running interface data is determined.

[0016] As an optional implementation, in a second aspect of the invention, the step of performing application testing operations adapted to the application under test based on the target element set and the pre-parsed test instruction set includes: The element position coordinates in the target element set are converted into device-independent normalized coordinate representations, which are then adapted to different screen resolutions and irregular screen display characteristics through safe area correction processing. Based on the operation types and parameters defined in the pre-parsed test instruction set, a platform adaptation component matching the running platform of the application under test is invoked to map the normalized coordinate representation into executable operation instructions for a specific platform corresponding to the application under test, so as to execute application test operations adapted to the application under test. The operation types include at least one of click, swipe, long press, double click, drag, zoom, text input, and gesture operation.

[0017] As an optional implementation, in a second aspect of the invention, the method further includes: During the execution of the application testing operation, multi-dimensional feedback data is captured in real time. The multi-dimensional feedback data includes at least one of the following: interface response status data, operation trajectory sequence, network request log data, system event log data, and performance indicator data. The multi-dimensional feedback data is compared and verified with the preset assertion conditions in the test instruction set to obtain the multi-dimensional verification results. Based on the multi-dimensional verification results, a traceable test report of the application testing operation is generated.

[0018] As an optional implementation, in a second aspect of the present invention, the step of comparing and verifying the multi-dimensional feedback data with preset assertion conditions in the test instruction set to obtain multi-dimensional verification results includes: Cross-dimensional correlation analysis is performed on the multi-dimensional feedback data to obtain inter-dimensional correlation rules. The inter-dimensional correlation rules are used to represent the statistical dependence or temporal relationship between different dimension feedback data. The inter-dimensional correlation rules include at least one of causal relationship rules, temporal constraint rules, and statistical correlation rules. Based on the inter-dimensional association rules and the preset assertion conditions, an adaptive verification condition set is generated. The adaptive verification condition set is used to dynamically adjust the strictness or priority of verification. The adaptive verification condition set includes at least one of dynamic threshold conditions, verification priority rules, and condition combination rules. Based on the adaptive verification condition set, the multi-dimensional feedback data is subjected to hierarchical verification to generate multi-dimensional verification results.

[0019] As an optional implementation, in a second aspect of the invention, the method further includes: Based on the multi-dimensional verification results and the test instruction set, a test adaptation strategy is generated. The test adaptation strategy is used to represent adaptive adjustment rules for the test execution process. The test adaptation strategy is obtained by analyzing the correlation between verification anomaly events in the multi-dimensional verification results and the test instruction set. The test adaptation strategy includes at least one of test step skipping conditions, retry triggering conditions, and execution order optimization rules. Based on the test adaptation strategy, the test instruction set is dynamically reconstructed to generate an adaptive test instruction sequence. The dynamic reconstruction is achieved by modifying the execution logic, parameters, or order of the test instructions. According to the adaptive test instruction sequence, the application test operation is re-executed. The application test operation adapts to the dynamic behavior changes of the application under test through closed-loop control of the adaptive test instruction sequence and real-time test feedback.

[0020] A third aspect of this invention discloses another cross-platform automated testing system for multi-dimensional perception analysis, the system comprising: Memory containing executable program code; A processor coupled to the memory; The processor calls the executable program code stored in the memory to execute the cross-platform automated testing method for multi-dimensional perception analysis disclosed in the second aspect of the present invention.

[0021] The fourth aspect of the present invention discloses a computer storage medium storing computer instructions, which, when invoked, are used to execute the cross-platform automated testing method for multi-dimensional perception analysis disclosed in the second aspect of the present invention.

[0022] Compared with the prior art, the embodiments of the present invention have the following beneficial effects: In this embodiment of the invention, the cross-platform execution layer acquires the running interface data of the application under test; the multi-dimensional perception and analysis layer performs corresponding multi-dimensional perception and analysis operations on the running interface data according to the multi-dimensional perception and analysis model, generating a candidate element set corresponding to the running interface data. The multi-dimensional perception and analysis operations include at least two of the following: text recognition and analysis, semantic understanding and analysis, icon recognition and analysis, interface layout recognition and analysis, and action intent prediction and analysis; the comprehensive target localization layer determines the target element set corresponding to the running interface data according to the preset multi-dimensional feature similarity comprehensive calculation model and the candidate element set; the cross-platform execution layer executes application testing operations adapted to the application under test according to the target element set and the pre-parsed test instruction set. It is evident that implementing this invention, by introducing "multi-dimensional perception and analysis operations" (including at least two operations such as text, semantics, icons, layout, and action intent), can overcome the limitations of traditional single-reliance on control tree or image recognition technologies, improve the depth and comprehensiveness of understanding the running interface of the application under test, and thus facilitate accurate identification of interface elements even in scenarios where control tree information is missing or unreliable, such as games and heavily customized UIs, thereby enhancing the universality of the testing system for different types of applications. By leveraging the synergy of "multi-dimensional perception analysis" and "pre-defined multi-dimensional feature similarity comprehensive calculation model," the accuracy and robustness of target element localization can be improved. This helps reduce the probability of element recognition failure due to minor interface changes (such as theme changes, resolution adaptation, and text internationalization), thereby reducing the maintenance cost of test scripts and achieving a higher degree of test automation. The unified handling of interface acquisition and test execution through the "cross-platform execution layer" reduces the coupling between test logic and specific platform-level interfaces. This facilitates the reuse of test cases for the same application business on different operating systems (such as Android and iOS), significantly improving the efficiency of cross-platform testing and achieving true cross-platform automated testing. Multimodal approaches enhance the flexibility and robustness of cross-platform automated testing. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 This is a schematic diagram of the structure of a cross-platform automated testing system for multi-dimensional perception and analysis disclosed in an embodiment of the present invention; Figure 2 This is a schematic diagram of the structure of another cross-platform automated testing system for multi-dimensional perception and analysis disclosed in an embodiment of the present invention; Figure 3 This is a flowchart illustrating a cross-platform automated testing method for multi-dimensional perception analysis disclosed in an embodiment of the present invention. Figure 4 This is a schematic diagram of the structure of another cross-platform automated testing system for multi-dimensional perception and analysis disclosed in an embodiment of the present invention. Detailed Implementation

[0025] To enable those skilled in the art to better understand the present invention, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0026] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this invention are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or end that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or ends.

[0027] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0028] This invention discloses a cross-platform automated testing system and method based on multi-dimensional perception analysis. By introducing "multi-dimensional perception analysis operations" (including at least two operations such as text, semantics, icons, layout, and action intent), it overcomes the limitations of traditional single-reliance on control trees or image recognition technologies, improving the depth and comprehensiveness of understanding the interface of the application under test. This allows for accurate identification of interface elements even in scenarios where control tree information is missing or unreliable, such as games or heavily customized UIs, thereby enhancing the universality of the testing system across different application types. Through the synergistic effect of "multi-dimensional perception analysis" and "preset multi-dimensional feature similarity comprehensive calculation model," the accuracy and robustness of target element location are improved. This helps reduce the probability of element recognition failure due to minor interface changes (such as theme changes, resolution adaptation, and text internationalization), thereby reducing the maintenance cost of test scripts and achieving a higher degree of test automation. By unifying the processing of interface acquisition and test execution through a "cross-platform execution layer," the coupling between test logic and the underlying interfaces of specific platforms can be reduced. This facilitates the reuse of test cases for the same application business on different operating systems (such as Android and iOS), thereby significantly improving the efficiency of cross-platform testing and achieving true cross-platform automated testing. Multimodal approaches enhance the flexibility and robustness of cross-platform automated testing. These will be explained in detail below.

[0029] Example 1 Please see Figure 1 , Figure 1 This is a schematic diagram of the structure of a cross-platform automated testing system for multi-dimensional perception and analysis disclosed in an embodiment of the present invention. Figure 1 The described multi-dimensional perception and analysis cross-platform automated testing system can be applied to system platforms such as Android, iOS, Windows, Linux, and HarmonyOS, and also to intelligent devices related to these platforms. These intelligent devices include, but are not limited to, one or more of mobile terminal devices, switch devices, cloud devices, edge computing devices, relay devices, base station devices, urban management devices, and intelligent connected devices. This invention does not limit the scope of these applications. Furthermore, this invention can be applied to regression testing, smoke testing, stability inspection, internationalization verification, online inspection and black-box acceptance; and end-to-end automated verification scenarios in industries such as finance, government and enterprise, e-commerce, automotive, IoT, and education. This invention does not specifically limit the scope of these applications. Figure 1 As shown, this cross-platform automated testing system for multi-dimensional perception and analysis may include: Cross-platform execution layer 101 is used to obtain the runtime interface data of the application under test; In this embodiment of the invention, optionally, for the cross-platform execution layer to acquire runtime interface data: this layer acquires the user interface of the application under test in real time by calling the native screen capture interface or graphics interface (such as Android's MediaProjection, iOS's CGWindowListCreateImage, and Windows' GDI or DirectX) provided by each operating system (such as Android, iOS, Windows, Linux, etc.). The runtime interface data is usually a screenshot in bitmap format. To further improve the accuracy of subsequent analysis, the screen resolution, density, and safe area information of irregular screens (such as notch screens and punch-hole screens) can be acquired simultaneously.

[0030] Optionally, before the cross-platform execution layer obtains the runtime interface data, a script description layer can be set up, or the cross-platform execution layer can uniformly parse user-written test scripts that are not limited to adapting to the system platform. This allows for the definition of a unified test script language (such as JSON / YAML), defining actions (such as tap, long_press, double_tap, type, swipe, drag, pinch, wait, assert, etc.), targets (such as name, id, etc. to confirm the target to be tested), and parameters (such as runtime environment, timeout, etc.). It supports variables, conditions, loops, parallelism, timeout and retry strategies, and supports data-driven and test case orchestration. Alternatively, testers can write test cases within a pre-defined framework, and the script parser can parse the test cases to generate a unified test script, and then generate the corresponding platform-executable test code, i.e., a set of test instructions.

[0031] The multi-dimensional perception and recognition analysis layer 102 is used to perform corresponding multi-dimensional perception and analysis operations on the running interface data according to the multi-dimensional perception and analysis model, and generate a candidate element set corresponding to the running interface data. The multi-dimensional perception and analysis operations include at least two of the following: text recognition and analysis, semantic understanding and analysis, icon recognition and analysis, interface layout recognition and analysis, and action intent prediction and analysis. In this embodiment of the invention, optionally, the analysis operation is performed on the multi-dimensional perception and recognition analysis layer: after receiving a screenshot, this layer starts multiple analysis engines in parallel. For example: Text recognition and analysis engine: Uses OCR technology to recognize all text blocks in the image and records the bounding box coordinates and recognized text content of each text block.

[0032] Icon recognition and analysis engine: Uses pre-trained icon detection models (such as deep learning-based object detection networks) to identify common UI icons (such as "Search", "Settings", "Favorites", etc.) and outputs the icon type and its location.

[0033] Interface layout recognition and analysis engine: Through image processing technology (such as edge detection and contour analysis) and layout reasoning algorithms, it identifies the structural areas of the interface, such as the top navigation bar, bottom tab bar, side drawer menu, list items, etc., and understands the relative positional relationship of each element.

[0034] The results of these analytical operations are aggregated and used together to understand the interface.

[0035] Optionally, AI engines (such as YOLOv8, Detectron2, OpenCV DNN, etc.) can be used to perform element recognition (such as buttons, switches, input boxes, speech recognition, etc.) on the current screen or UI layer. Further, denoising can be performed (such as adaptive brightness / contrast, gamma / sharpening, morphological denoising, etc., to improve text readability). Further, multimodal semantic understanding can be performed: fusing icon detection, layout analysis (such as semantic partitioning of title area, content area, bottom bar / navigation bar, etc.) or LLM semantic understanding to form a candidate element set and confidence score. Further, adaptive normalization can be performed: for example, converting coordinates to device-independent normalized coordinates. Further, the final recognition result can be output (such as text content, element coordinates, confidence score, language tags, etc.).

[0036] In this embodiment of the invention, as an optional implementation, the aforementioned multi-dimensional perception analysis model includes at least a multi-dimensional feature recognition analysis model, a multi-dimensional semantic understanding analysis model, and an operation intent prediction analysis model. The multi-dimensional perception analysis model is matched with the operation. The multi-dimensional perception recognition analysis layer 102 performs corresponding multi-dimensional perception analysis operations on the running interface data according to the multi-dimensional perception analysis model. The specific method for generating the candidate element set corresponding to the running interface data includes: Based on the multi-dimensional feature recognition and analysis model, interface element recognition and image denoising are performed on the running interface data to obtain preliminary element recognition results. Based on the multi-dimensional semantic understanding analysis model, multi-modal semantic understanding operations are performed on the preliminary element recognition results to integrate icon detection, layout analysis and semantic partitioning features to generate a semantically enhanced element set; Based on the operation intent prediction and analysis model and the action intent instructions in the pre-parsed test instruction set, the semantic enhancement element set is subjected to intent matching and confidence calculation to generate a candidate element set corresponding to the running interface data. Each candidate element in the candidate element set includes its corresponding element attribute, element position coordinates and confidence score.

[0037] In this embodiment of the invention, optionally, the multi-dimensional feature recognition and analysis model described above is processed as follows: This stage mainly involves the extraction and preprocessing of low-level features. For example, image noise reduction processing is performed on the acquired screenshots to reduce the impact of screen noise and compression artifacts on recognition. Subsequently, computer vision methods such as contour detection and connected component analysis are used to initially segment regions that may be independent UI elements (such as buttons, input boxes, and images), and their positions and basic visual attributes are recorded to form preliminary element recognition results.

[0038] Optionally, the aforementioned multi-dimensional semantic understanding analysis model can be fused: this stage aims to enhance the understanding of the interface's meaning. It receives the preliminary results from the previous step and performs multimodal fusion. For example, it associates the "shopping cart" icon identified by the icon detection model, the "isolated area at the bottom of the screen" identified by layout analysis, and the "￥99.9" text identified by OCR. Through semantic understanding, it may infer that this is a "bottom shopping cart bar," where "￥99.9" is the price and the "shopping cart" icon is clickable. This fusion generates a set of semantically enhanced elements, each containing not only its own attributes but also its contextual relationship with surrounding elements.

[0039] Optionally, the above-mentioned operation intent prediction and analysis model can be further subjected to intent matching: this stage matches the intent of the test instruction with the interface elements. For example, the test instruction is "click the login button". The model will analyze that the "action intent" of the instruction is "click", and the "semantic intent" of the target object is "the button used for login"; in the set of semantically enhanced elements, it will find the element that best matches the intent. A button with the text "login" or a "human" icon will be assigned a high confidence score; while a "register" button will be assigned a lower score; finally, a set of candidate elements with confidence scores will be output to prepare for subsequent precise positioning.

[0040] As can be seen, implementing this optional embodiment, through a progressive analysis process of "multi-dimensional feature recognition analysis model – multi-dimensional semantic understanding analysis model – operation intent prediction analysis model," can improve the hierarchical and structured nature of information extracted from the original interface data. This facilitates the gradual filtering of image noise, the fusion of multi-modal features, and the introduction of test intent, resulting in a candidate element set that not only contains geometric attributes but is also rich in semantic and intent information, achieving a more accurate semantic description of interface elements. By fusing icon, layout, and semantic partitioning features through "multi-modal semantic understanding operation," the ability to recognize the overall structure of the interface and the functions of elements can be improved. This facilitates the correct identification of visually separated but semantically related element combinations (such as label icons and the text below them), thereby reducing element misidentification and omission, and enhancing the reliability of subsequent steps. Through intent matching and confidence calculation using the "operation intent prediction analysis model," the correlation between candidate elements and the expected target of the test instruction can be improved. This facilitates the selection of the one that best matches the current operation intent from multiple similar elements, providing a quantitative basis for subsequent accurate positioning and reducing the possibility of misoperation.

[0041] The comprehensive target localization layer 103 is used to determine the target element set corresponding to the running interface data based on the preset multi-dimensional feature similarity comprehensive calculation model and the candidate element set; In this embodiment of the invention, optionally, the target element set is determined for the aforementioned comprehensive target localization layer: this layer receives a candidate element set from the upper layer. Its core is calculating the matching degree between each candidate element and the target element described in the test instruction. The matching degree calculation is not a single process but integrates multiple features. For example, the test instruction requires finding a "save" button. The system will simultaneously calculate the similarity of candidate elements in the following aspects: Text similarity: Whether the text content of the candidate element is "save", "save", etc.

[0042] Visual similarity: Whether the icon of the candidate element is visually similar to the preset "save" icon (such as the disk icon).

[0043] Semantic / Positional Similarity: Whether the candidate element is located in a reasonable contextual position, such as the top right corner or bottom toolbar of an editing page, which conforms to the common layout semantics of a "Save" button. Finally, a weighted fusion algorithm is used to combine multiple similarity scores into a comprehensive score, and one or more elements with scores higher than a threshold are selected as the target element set.

[0044] In this embodiment of the invention, as another optional implementation, the specific method by which the comprehensive target localization layer 103 determines the target element set corresponding to the running interface data based on the preset multi-dimensional feature similarity comprehensive calculation model and the candidate element set includes: For each candidate element in the candidate element set, extract the similarity information between the candidate element and the target element description data in the pre-parsed test instruction set across multiple feature dimensions. The feature dimensions include at least text features, local image features, and semantic features. The similarity information on text features is used to represent the degree of matching between the candidate element and the target element description data in the text content dimension. The similarity information on local image features is used to represent the degree of matching between the candidate element and the target element description data in the element visual appearance dimension. The similarity information on semantic features is used to represent the degree of semantic association between the candidate element and the target element description data in the interface context dimension. All extracted similarity information is input into a preset multi-dimensional feature similarity comprehensive calculation model for weighted fusion calculation. The weighted fusion calculation dynamically adjusts the weight allocation of each feature dimension based on the interface context to generate a comprehensive similarity score for each element. Based on the comprehensive similarity score and the preset confidence threshold, the set of target elements corresponding to the running interface data is determined.

[0045] In this embodiment of the invention, optionally, the extraction of multi-feature dimension similarity information described above is as follows: Text feature similarity: The degree of matching between the candidate element's text and the target description text can be calculated using text matching algorithms (such as edit distance and word vector cosine similarity). For example, if the description is "username" and the candidate text is "user name", the similarity is high.

[0046] Local image feature similarity: This can be calculated using image feature extraction algorithms (such as traditional feature points like ORB and SIFT, or image features extracted through deep learning) to determine the similarity between the visual appearance of a candidate element region and the target description image (if provided) or a preset template. For example, comparing the similarity between a candidate icon and a "search" icon template.

[0047] Semantic feature similarity: The semantic matching degree with the target description can be calculated by analyzing the role of an element in the interface layout (such as whether it is a clickable item or whether it is inside a form) and its semantic relationship with the surrounding text (such as an input box next to the text "password:", then its semantics are a password input box).

[0048] Weighted fusion calculation: A pre-defined fusion model (which could be a simple linear weighted model or a more complex decision model) assigns appropriate weights to each feature dimension. The weight assignment is dynamic, for example: When the target description explicitly states that it is an "image button", the weight of local image features will be automatically increased.

[0049] When the interface context is displayed within an explicit "login" form, the weight of semantic features increases because, in this context, the "button" element is highly likely to be a "login button".

[0050] By weighted summation, a comprehensive similarity score is obtained for each candidate element.

[0051] Determine the target element set: Set a confidence threshold (e.g., 0.8). All candidate elements with a comprehensive similarity score exceeding this threshold are selected into the target element set. If multiple elements have high scores, the one with the highest score, or all of them, can be selected as candidate targets for further processing.

[0052] As can be seen, implementing this optional embodiment can improve the comprehensiveness and fault tolerance of element matching judgment by extracting and comprehensively calculating similarity information from multiple dimensions such as "text features, local image features, and semantic features." This is beneficial for ensuring the correctness of the overall result by relying on strong matching of other feature dimensions when the matching of a certain feature dimension is poor (such as blurred text or distorted icons), thereby improving the system's adaptability to complex and dynamic interfaces. By "dynamically adjusting the weight allocation of each feature dimension based on the interface context," the context awareness and intelligence level of the similarity calculation model can be improved. This is beneficial for highlighting key features in different test scenarios (such as increasing image weight when searching for icon buttons and increasing semantic weight when filling out forms), thus making the comprehensive similarity score more reflective of the true matching degree in the current context and improving positioning accuracy. Through "weighted fusion calculation" and "confidence threshold" filtering, the fuzzy multi-dimensional matching problem can be transformed into a clear quantitative decision problem, reducing the subjectivity and arbitrariness of determining the target element set. This is beneficial for achieving stable and repeatable element positioning, thus laying a solid foundation for the accurate execution of automated operations.

[0053] The cross-platform execution layer 101 is also used to execute application test operations adapted to the application under test based on the target element set and the pre-parsed test instruction set.

[0054] In this embodiment of the invention, optionally, the test operation is performed on the aforementioned cross-platform execution layer: the layer obtains the position coordinates of the target element (e.g., screen coordinates (x, y)). Then, it converts these coordinates into normalized coordinates that are independent of the specific device's screen resolution (e.g., percentage coordinates relative to the screen width and height). Next, it calls the corresponding platform's input simulation interface (such as the click method of Instrumentation or UIAutomator in Android, the tap method of XCUITest in iOS, and the SendInput function in Windows) to execute the operation specified by the test instruction, such as clicking or inputting text.

[0055] As can be seen, implementing the embodiments of the present invention can overcome the limitations of traditional single-reliance on control tree or image recognition technology by introducing "multi-dimensional perception analysis operations" (including at least two operations such as text, semantics, icons, layout, and action intent), thereby improving the depth and comprehensiveness of understanding the interface of the application under test. This is beneficial for accurately identifying interface elements even in scenarios where control tree information is missing or unreliable, such as games and heavily customized UIs, thus enhancing the universality of the testing system for different types of applications. Through the synergistic effect of "multi-dimensional perception analysis" and "preset multi-dimensional feature similarity comprehensive calculation model", the accuracy and robustness of target element positioning can be improved, thereby reducing the probability of element recognition failure caused by slight changes in the interface (such as theme change, resolution adaptation, and text internationalization), thus reducing the maintenance cost of test scripts and achieving a higher degree of test automation. By unifying the processing of interface acquisition and test execution through the "cross-platform execution layer," the coupling between test logic and the underlying interface of a specific platform can be reduced. This facilitates the reuse of test cases for the same application business on different operating systems (such as Android and iOS), thereby significantly improving the efficiency of cross-platform testing and achieving true cross-platform automated testing. Multimodal approaches enhance the flexibility and robustness of cross-platform automated testing.

[0056] In this embodiment of the invention, as an optional implementation, the cross-platform execution layer 101 executes application testing operations adapted to the application under test based on the target element set and the pre-parsed test instruction set in the following specific ways: The element position coordinates in the target element set are converted into device-independent normalized coordinate representations. The normalized coordinate representations are then adapted to different screen resolutions and irregular screen display characteristics through safe area correction. Based on the operation types and parameters defined in the pre-parsed test instruction set, the platform adaptation component that matches the running platform of the application under test is called to map the normalized coordinate representation into executable operation instructions for the specific platform corresponding to the application under test, so as to execute application test operations adapted to the application under test. The operation types include at least one of click, swipe, long press, double click, drag, zoom, text input, and gesture operation.

[0057] In this embodiment of the invention, optionally, for the above coordinate transformation and safe area correction: the obtained target element coordinates are absolute coordinates relative to the current device screen (e.g., (100, 200) at a 1920*1080 resolution). First, they are converted to device-independent normalized coordinates, for example, to relative coordinates (width percentage, height percentage): (100 / 1920 ≈ 0.052, 200 / 1080 ≈ 0.185). Next, safe area correction is performed: the device's irregular screen information (e.g., iPhone's Safe Area Insets) is queried to ensure that the converted coordinate point is not in unclickable areas such as the notch, camera bezel, or gesture bar. If the original coordinates fall in an unsafe area, they are shifted to the nearest safe point.

[0058] Alternatively, for the above mapping to platform instructions: the platform adaptation component generates native instructions based on the operation type and normalized coordinates of the test instructions.

[0059] For example, on the Android platform, a "click" operation might ultimately be mapped to an ADB command implementation of input tap xy.

[0060] For "text input" operations, the corresponding setText() method or input text command will be called.

[0061] For the "swipe" operation, the normalized coordinates of the start and end points need to be mapped into a series of touchdown, touch move, and touch up events.

[0062] As can be seen, implementing this optional embodiment can reduce the dependence of test operations on the screen resolution of specific devices by converting the element position coordinates into a "device-independent normalized coordinate representation." This allows the same test instruction to be executed on devices of different sizes and resolutions without modification, thereby significantly improving the portability and execution efficiency of test cases. Through "safety area correction processing to adapt to different screen resolutions and irregular screen display characteristics," the success rate of test operations on devices with irregular screen shapes such as notches, punch holes, and curved edges can be improved. This helps avoid test failures caused by touch points falling in inoperable areas, thus expanding the device coverage of the test system. By mapping normalized coordinates to executable operation instructions for specific platforms through "platform adaptation components," the coupling between the core test logic and the underlying input simulation mechanisms of each platform can be reduced. This helps to shield the implementation differences of different operating systems, thereby achieving a unified description of test instructions and seamless cross-platform execution, simplifying the test architecture.

[0063] Example 2 Please see Figure 2 , Figure 2 This is a schematic diagram of another cross-platform automated testing system for multi-dimensional perception and analysis disclosed in an embodiment of the present invention. Wherein, Figure 2 The described multi-dimensional perception analysis cross-platform automated testing system can be applied to system platforms such as Android, iOS, Windows, Linux, and HarmonyOS, and can also be applied to intelligent devices related to the system platform. These intelligent devices include, but are not limited to, one or more of mobile terminal devices, switch devices, cloud devices, edge computing devices, relay devices, base station devices, urban management devices, and intelligent connected devices. This invention does not limit the scope of these applications. Figure 2 As shown, this cross-platform automated testing system for multi-dimensional perception and analysis may include: Cross-platform execution layer 201 is used to obtain runtime interface data of the application under test; The multi-dimensional perception and recognition analysis layer 202 is used to perform corresponding multi-dimensional perception and analysis operations on the running interface data according to the multi-dimensional perception and analysis model, and generate a candidate element set corresponding to the running interface data. The multi-dimensional perception and analysis operations include at least two of the following: text recognition and analysis, semantic understanding and analysis, icon recognition and analysis, interface layout recognition and analysis, and action intent prediction and analysis. The comprehensive target localization layer 203 is used to determine the target element set corresponding to the running interface data based on the preset multi-dimensional feature similarity comprehensive calculation model and the candidate element set; The cross-platform execution layer 201 is also used to execute application test operations adapted to the application under test based on the target element set and the pre-parsed test instruction set.

[0064] In this embodiment of the invention, for other supplementary explanations of 201-203 above, please refer to the supplementary explanations of 101-103 in Embodiment 1. This embodiment of the invention will not repeat them here.

[0065] Assertion verification layer 204 is used to capture multi-dimensional feedback data in real time during the execution of application testing operations. The multi-dimensional feedback data includes at least one of the following: interface response status data, operation trajectory sequence, network request log data, system event log data, and performance indicator data. The assertion verification layer 204 is also used to compare and verify the multi-dimensional feedback data with the preset assertion conditions in the test instruction set to obtain multi-dimensional verification results. In this embodiment of the invention, as an optional implementation, the assertion verification layer 204 compares and verifies the multi-dimensional feedback data with the preset assertion conditions in the test instruction set to obtain the multi-dimensional verification results. The specific methods include: Cross-dimensional correlation analysis is performed on multi-dimensional feedback data to obtain inter-dimensional correlation rules. Inter-dimensional correlation rules are used to represent the statistical dependence or temporal relationship between different dimensions of feedback data. Inter-dimensional correlation rules include at least one of causal relationship rules, temporal constraint rules, and statistical correlation rules. Based on the inter-dimensional association rules and preset assertion conditions, an adaptive verification condition set is generated. The adaptive verification condition set is used to dynamically adjust the strictness or priority of verification. The adaptive verification condition set includes at least one of dynamic threshold conditions, verification priority rules, and condition combination rules. Based on an adaptive set of validation conditions, hierarchical validation is performed on multi-dimensional feedback data to generate multi-dimensional validation results.

[0066] In this embodiment of the invention, optionally, for the above-mentioned cross-dimensional correlation analysis: the system analyzes the inherent relationships between data of different dimensions and forms rules.

[0067] Causality rule: For example, "When the login button is clicked, a POST request to / api / login must appear in the network log, and the interface must redirect from the login page after the request succeeds." If there is no network request after clicking, or the interface redirects after the request fails, the rule is violated.

[0068] Timing constraint rules: For example, "The page load complete event must occur after the homepage data request is completed".

[0069] Statistical correlation rules: For example, it was found that "surge in memory usage" and "interface lag (sudden drop in FPS)" are statistically highly correlated.

[0070] Further, optionally, for the above-mentioned set of adaptive verification conditions: Dynamic threshold conditions: For example, the performance assertion "FPS should be greater than 55" is not absolute. If system correlation analysis finds that the current test equipment has low performance, the threshold can be dynamically relaxed to 50.

[0071] Validation priority rules: For example, if a network request assertion fails, the UI assertion may be marked as "blocking" level; if only the color of a UI element in a minor position is slightly off, it is marked as "minor" level.

[0072] Hierarchical verification: Based on adaptive conditions, the system performs weighted and hierarchical checks on data of different dimensions according to their importance and association rules, rather than simply making isolated true or false judgments, thereby obtaining more comprehensive and intelligent multi-dimensional verification results.

[0073] As can be seen, implementing this optional embodiment can obtain "inter-dimensional association rules" through "cross-dimensional correlation analysis" of multi-dimensional feedback data, which can improve the insight into the inherent relationships between various events during the testing process. This is conducive to discovering deeper issues such as "whether the interface jump is triggered by the success of a specific network request" or "whether the performance degradation is related to a specific operation sequence," thereby enabling verification to go beyond single-point judgment and achieve systematic verification based on causality and timing. By "generating an adaptive verification condition set" based on association rules and preset assertions, the dynamism and intelligence of the verification strategy can be improved. This is conducive to dynamically adjusting the rigor and focus of verification according to different test contexts (such as device performance and network status), making the verification process more realistic and reducing false positives and false negatives. By "performing layered verification based on the adaptive verification condition set," the systematicity and efficiency of the verification process can be improved. This is conducive to distinguishing the primary and secondary aspects of verification, making strict assertions on the critical path while flexibly handling non-core issues, thereby improving the robustness and efficiency of test execution while ensuring the depth of verification.

[0074] Assertion verification layer 204 is also used to generate traceable test reports of application test operations based on multi-dimensional verification results.

[0075] In this embodiment of the invention, optionally, for the above-mentioned real-time capture of multi-dimensional feedback data: Interface response status data: After an operation is performed, the screen can be captured again immediately, and the new interface can be analyzed through a multi-dimensional perception and recognition analysis layer to obtain UI changes (such as new page pop-ups, element disappearances, and text changes).

[0076] Operation trajectory sequence: can record the timestamps and coordinate sequences of all simulated input events (touch, key press).

[0077] Network request log data: HTTP / HTTPS requests issued by the application, along with their response status codes and data volume, can be captured through proxies (such as Charles or Fiddler) or platform debugging interfaces.

[0078] System event log data: It can monitor system logs (such as Android Logcat, iOS Console) and filter out errors, warning messages or specific lifecycle events generated by the application under test.

[0079] Performance metrics data: CPU usage, memory usage, frame rate (FPS) and other data can be collected through performance monitoring tools (such as Perfetto for Android and Instruments for iOS).

[0080] Further, optionally, for the above comparison verification and report generation: the captured data is compared with the predefined assertion conditions in the test instructions. For example: Assertion condition: "Login should redirect to the homepage." During verification, check if homepage-specific elements (such as a "Welcome Back" title) appear in the interface response data.

[0081] Assertion condition: "Operation response time should be less than 1 second". During verification, this is calculated based on the time difference between the start of the operation and the stable change of the interface.

[0082] All validation results (success / failure, actual value, expected value), raw data, and timestamps are aggregated to generate a structured (such as HTML or JSON format) and traceable test report, which facilitates problem localization.

[0083] Furthermore, the entire process also supports retrying, observation, etc., such as 1. Failure self-healing and regression (1) Multi-strategy rollback: Screenshot → Icon detection → Template matching (multi-resolution) → Accessibility tree search; (2) Context recovery: Automatically recognizes standard pop-up windows when clicking "Back / Close / Allow / Continue"; (3) Guidance and correction: When multiple candidates are in parallel, the optimal candidate is selected by combining historical paths and LLM semantic preferences; 2. Observability and Reporting (1) Evidence chain: full video recording, keyframes, identification overlay images, click hotspots, logs, device information, scripts and version fingerprints; (2) Report output: HTML / PDF / Allure and other formats, including step sequence, assertion results and failure screenshots; (3) Playback: The scene can be restored based on the timing and operation trajectory.

[0084] As can be seen, implementing the embodiments of the present invention can improve the comprehensiveness and refinement of the testing process monitoring by capturing multi-dimensional feedback data in real time, including "interface response status, operation trajectory, network requests, system events, and performance indicators." This facilitates the acquisition of rich verification information far exceeding the judgment of a single interface state, thereby providing comprehensive data support for in-depth evaluation of application quality. By "comparing and verifying" the multi-dimensional feedback data with preset assertion conditions, the objectivity and accuracy of test result judgment can be improved, which is conducive to discovering potential defects at all levels, from UI interaction to backend requests and system resource consumption, thereby enhancing the depth of testing and the ability to discover problems. By generating "traceable test reports," the transparency of the testing process and the efficiency of problem localization can be improved, which is conducive to developers quickly reproducing and locating errors found during testing, thereby accelerating the software debugging and repair process and improving the overall development and testing efficiency.

[0085] In an optional embodiment, such as Figure 2 As shown, the system may also include: The adaptive closed-loop layer 205 is used to generate a test adaptation strategy based on the multi-dimensional verification results and the test instruction set. The test adaptation strategy is used to represent the adaptive adjustment rules for the test execution process. The test adaptation strategy is obtained by analyzing the correlation between the verification abnormal events in the multi-dimensional verification results and the test instruction set. The test adaptation strategy includes at least one of the following: test step skipping conditions, retry triggering conditions, and execution order optimization rules. The adaptive closed-loop layer 205 is also used to dynamically reconstruct the test instruction set based on the test adaptation strategy to generate an adaptive test instruction sequence. Dynamic reconstruction is achieved by modifying the execution logic, parameters or order of the test instructions. The adaptive closed-loop layer 205 is also used to re-execute application test operations according to the adaptive test instruction sequence. The application test operations adapt to the dynamic behavior changes of the application under test through the closed-loop control of the adaptive test instruction sequence and real-time test feedback.

[0086] In this embodiment of the invention, optionally, for the above-mentioned generation test adaptation strategy: the system analyzes abnormal patterns in multi-dimensional verification results. For example: If the assertion fails three times consecutively after a "network request timeout" message, the system will generate a retry trigger condition: if a network timeout occurs, the operation will be automatically retried up to N times.

[0087] If a test step (such as "skip the onboarding page") is found to succeed in 95% of cases, and failure does not affect the core process, the system will generate a test step skip condition: when this step fails, it will be recorded as a warning and subsequent tests will continue to be executed, rather than terminating the entire test.

[0088] The system detected a resource contention between test sequences A and B in their execution order. The system then generates an execution order optimization rule: in the next round of testing, it attempts to swap the order of A and B.

[0089] Dynamically refactoring the test instruction set: Based on the above strategy, the system dynamically modifies the logic of the test script in memory. For example, when the retry condition is met, it does not simply repeat the original instruction, but may first insert a "wait 3 seconds" instruction, and then execute the target operation again, thereby refactoring the instruction sequence.

[0090] Closed-loop control execution: This dynamically generated adaptive test instruction sequence is immediately sent to the cross-platform execution layer for execution, forming a closed-loop feedback system of "perception -> decision -> execution -> re-perception". This enables the testing process to intelligently adapt to the uncertainties brought about by the dynamic changes of the application itself (such as network fluctuations and pop-up interference), greatly improving the robustness and automation of the test.

[0091] As can be seen, implementing this optional embodiment can improve the test system's adaptability and fault tolerance to deviations or anomalies during execution by "generating test adaptation strategies" based on multi-dimensional verification results and test instruction sets. This facilitates intelligent decision-making on how to deal with failures (such as retrying, skipping, or adjusting steps), thereby reducing interruptions to the entire test process caused by non-fatal anomalies (such as network jitter or temporary pop-ups), and improving the pass rate and continuity of testing. By "dynamically reconstructing" the test instruction set based on the test adaptation strategy, the flexibility and intelligence of the test process can be improved, which is conducive to optimizing test paths and parameters based on real-time feedback. This transforms the testing behavior from static script execution into a dynamic process with a certain degree of decision-making ability. Through "closed-loop control of adaptive test instruction sequences and real-time test feedback," the test system can achieve self-adjustment and continuous optimization, enabling the test process to proactively adapt to the dynamic behavioral changes of the application under test (such as the uncertainty of data loading speed and interface response time). This significantly improves the robustness and reliability of automated testing in real and complex environments, moving towards autonomous testing.

[0092] Example 3 Please see Figure 3 , Figure 3This is a flowchart illustrating a cross-platform automated testing method for multi-dimensional perception analysis disclosed in an embodiment of the present invention. This cross-platform automated testing method for multi-dimensional perception analysis can be applied to system platforms such as Android, iOS, Windows, Linux, and HarmonyOS, and can also be applied to intelligent devices related to the system platform. These intelligent devices include, but are not limited to, one or more of mobile terminal devices, switch devices, cloud devices, edge computing devices, relay devices, base station devices, urban management devices, and intelligent connected devices; the embodiments of the present invention do not impose limitations. Figure 3 As shown, this cross-platform automated testing method for multi-dimensional perception analysis can include: 301. Obtain the running interface data of the application under test; 302. Based on the multi-dimensional perception analysis model, perform corresponding multi-dimensional perception analysis operations on the running interface data to generate a candidate element set corresponding to the running interface data. The multi-dimensional perception analysis operations include at least two of the following: text recognition and analysis, semantic understanding and analysis, icon recognition and analysis, interface layout recognition and analysis, and action intent prediction and analysis. 303. Based on the preset multi-dimensional feature similarity comprehensive calculation model and candidate element set, determine the target element set corresponding to the running interface data; 304. Based on the target element set and the pre-parsed test instruction set, execute application test operations that are compatible with the application under test.

[0093] As can be seen, implementing the embodiments of the present invention can overcome the limitations of traditional single-reliance on control tree or image recognition technology by introducing "multi-dimensional perception analysis operations" (including at least two operations such as text, semantics, icons, layout, and action intent), thereby improving the depth and comprehensiveness of understanding the interface of the application under test. This is beneficial for accurately identifying interface elements even in scenarios where control tree information is missing or unreliable, such as games and heavily customized UIs, thus enhancing the universality of the testing system for different types of applications. Through the synergistic effect of "multi-dimensional perception analysis" and "preset multi-dimensional feature similarity comprehensive calculation model", the accuracy and robustness of target element positioning can be improved, thereby reducing the probability of element recognition failure caused by slight changes in the interface (such as theme change, resolution adaptation, and text internationalization), thus reducing the maintenance cost of test scripts and achieving a higher degree of test automation. By unifying the processing of interface acquisition and test execution through the "cross-platform execution layer," the coupling between test logic and the underlying interface of a specific platform can be reduced. This facilitates the reuse of test cases for the same application business on different operating systems (such as Android and iOS), thereby significantly improving the efficiency of cross-platform testing and achieving true cross-platform automated testing. Multimodal approaches enhance the flexibility and robustness of cross-platform automated testing.

[0094] In this embodiment of the invention, as an optional implementation, the aforementioned multi-dimensional perception analysis model includes at least a multi-dimensional feature recognition analysis model, a multi-dimensional semantic understanding analysis model, and an operation intent prediction analysis model. The multi-dimensional perception analysis model is matched with the operation. Based on the multi-dimensional perception analysis model, a corresponding multi-dimensional perception analysis operation is performed on the running interface data to generate a candidate element set corresponding to the running interface data, including: Based on the multi-dimensional feature recognition and analysis model, interface element recognition and image denoising are performed on the running interface data to obtain preliminary element recognition results. Based on the multi-dimensional semantic understanding analysis model, multi-modal semantic understanding operations are performed on the preliminary element recognition results to integrate icon detection, layout analysis and semantic partitioning features to generate a semantically enhanced element set; Based on the operation intent prediction and analysis model and the action intent instructions in the pre-parsed test instruction set, the semantic enhancement element set is subjected to intent matching and confidence calculation to generate a candidate element set corresponding to the running interface data. Each candidate element in the candidate element set includes its corresponding element attribute, element position coordinates and confidence score.

[0095] As can be seen, implementing this optional embodiment, through a progressive analysis process of "multi-dimensional feature recognition analysis model – multi-dimensional semantic understanding analysis model – operation intent prediction analysis model," can improve the hierarchical and structured nature of information extracted from the original interface data. This facilitates the gradual filtering of image noise, the fusion of multi-modal features, and the introduction of test intent, resulting in a candidate element set that not only contains geometric attributes but is also rich in semantic and intent information, achieving a more accurate semantic description of interface elements. By fusing icon, layout, and semantic partitioning features through "multi-modal semantic understanding operation," the ability to recognize the overall structure of the interface and the functions of elements can be improved. This facilitates the correct identification of visually separated but semantically related element combinations (such as label icons and the text below them), thereby reducing element misidentification and omission, and enhancing the reliability of subsequent steps. Through intent matching and confidence calculation using the "operation intent prediction analysis model," the correlation between candidate elements and the expected target of the test instruction can be improved. This facilitates the selection of the one that best matches the current operation intent from multiple similar elements, providing a quantitative basis for subsequent accurate positioning and reducing the possibility of misoperation.

[0096] In this embodiment of the invention, as another optional implementation, the above-mentioned determination of the target element set corresponding to the running interface data based on the preset multi-dimensional feature similarity comprehensive calculation model and the candidate element set includes: For each candidate element in the candidate element set, extract the similarity information between the candidate element and the target element description data in the pre-parsed test instruction set across multiple feature dimensions. The feature dimensions include at least text features, local image features, and semantic features. The similarity information on text features is used to represent the degree of matching between the candidate element and the target element description data in the text content dimension. The similarity information on local image features is used to represent the degree of matching between the candidate element and the target element description data in the element visual appearance dimension. The similarity information on semantic features is used to represent the degree of semantic association between the candidate element and the target element description data in the interface context dimension. All extracted similarity information is input into a preset multi-dimensional feature similarity comprehensive calculation model for weighted fusion calculation. The weighted fusion calculation dynamically adjusts the weight allocation of each feature dimension based on the interface context to generate a comprehensive similarity score for each element. Based on the comprehensive similarity score and the preset confidence threshold, the set of target elements corresponding to the running interface data is determined.

[0097] As can be seen, implementing this optional embodiment can improve the comprehensiveness and fault tolerance of element matching judgment by extracting and comprehensively calculating similarity information from multiple dimensions such as "text features, local image features, and semantic features." This is beneficial for ensuring the correctness of the overall result by relying on strong matching of other feature dimensions when the matching of a certain feature dimension is poor (such as blurred text or distorted icons), thereby improving the system's adaptability to complex and dynamic interfaces. By "dynamically adjusting the weight allocation of each feature dimension based on the interface context," the context awareness and intelligence level of the similarity calculation model can be improved. This is beneficial for highlighting key features in different test scenarios (such as increasing image weight when searching for icon buttons and increasing semantic weight when filling out forms), thus making the comprehensive similarity score more reflective of the true matching degree in the current context and improving positioning accuracy. Through "weighted fusion calculation" and "confidence threshold" filtering, the fuzzy multi-dimensional matching problem can be transformed into a clear quantitative decision problem, reducing the subjectivity and arbitrariness of determining the target element set. This is beneficial for achieving stable and repeatable element positioning, thus laying a solid foundation for the accurate execution of automated operations.

[0098] In another optional implementation of this invention, the above-mentioned execution of application testing operations adapted to the application under test based on the target element set and the pre-parsed test instruction set includes: The element position coordinates in the target element set are converted into device-independent normalized coordinate representations. The normalized coordinate representations are then adapted to different screen resolutions and irregular screen display characteristics through safe area correction. Based on the operation types and parameters defined in the pre-parsed test instruction set, the platform adaptation component that matches the running platform of the application under test is called to map the normalized coordinate representation into executable operation instructions for the specific platform corresponding to the application under test, so as to execute application test operations adapted to the application under test. The operation types include at least one of click, swipe, long press, double click, drag, zoom, text input, and gesture operation.

[0099] As can be seen, implementing this optional embodiment can reduce the dependence of test operations on the screen resolution of specific devices by converting the element position coordinates into a "device-independent normalized coordinate representation." This allows the same test instruction to be executed on devices of different sizes and resolutions without modification, thereby significantly improving the portability and execution efficiency of test cases. Through "safety area correction processing to adapt to different screen resolutions and irregular screen display characteristics," the success rate of test operations on devices with irregular screen shapes such as notches, punch holes, and curved edges can be improved. This helps avoid test failures caused by touch points falling in inoperable areas, thus expanding the device coverage of the test system. By mapping normalized coordinates to executable operation instructions for specific platforms through "platform adaptation components," the coupling between the core test logic and the underlying input simulation mechanisms of each platform can be reduced. This helps to shield the implementation differences of different operating systems, thereby achieving a unified description of test instructions and seamless cross-platform execution, simplifying the test architecture.

[0100] In an optional embodiment, the above method further includes: During the application testing process, multi-dimensional feedback data is captured in real time. The multi-dimensional feedback data includes at least one of the following: interface response status data, operation trajectory sequence, network request log data, system event log data, and performance indicator data. The multi-dimensional feedback data is compared and verified with the preset assertion conditions in the test instruction set to obtain the multi-dimensional verification results. Based on the multi-dimensional verification results, a traceable test report of the application test operations is generated.

[0101] As can be seen, implementing the embodiments of the present invention can improve the comprehensiveness and refinement of the testing process monitoring by capturing multi-dimensional feedback data in real time, including "interface response status, operation trajectory, network requests, system events, and performance indicators." This facilitates the acquisition of rich verification information far exceeding the judgment of a single interface state, thereby providing comprehensive data support for in-depth evaluation of application quality. By "comparing and verifying" the multi-dimensional feedback data with preset assertion conditions, the objectivity and accuracy of test result judgment can be improved, which is conducive to discovering potential defects at all levels, from UI interaction to backend requests and system resource consumption, thereby enhancing the depth of testing and the ability to discover problems. By generating "traceable test reports," the transparency of the testing process and the efficiency of problem localization can be improved, which is conducive to developers quickly reproducing and locating errors found during testing, thereby accelerating the software debugging and repair process and improving the overall development and testing efficiency.

[0102] In this optional embodiment, as an optional implementation method, the above-mentioned comparison and verification of multi-dimensional feedback data with preset assertion conditions in the test instruction set to obtain multi-dimensional verification results includes: Cross-dimensional correlation analysis is performed on multi-dimensional feedback data to obtain inter-dimensional correlation rules. Inter-dimensional correlation rules are used to represent the statistical dependence or temporal relationship between different dimensions of feedback data. Inter-dimensional correlation rules include at least one of causal relationship rules, temporal constraint rules, and statistical correlation rules. Based on the inter-dimensional association rules and preset assertion conditions, an adaptive verification condition set is generated. The adaptive verification condition set is used to dynamically adjust the strictness or priority of verification. The adaptive verification condition set includes at least one of dynamic threshold conditions, verification priority rules, and condition combination rules. Based on an adaptive set of validation conditions, hierarchical validation is performed on multi-dimensional feedback data to generate multi-dimensional validation results.

[0103] As can be seen, implementing this optional embodiment can obtain "inter-dimensional association rules" through "cross-dimensional correlation analysis" of multi-dimensional feedback data, which can improve the insight into the inherent relationships between various events during the testing process. This is conducive to discovering deeper issues such as "whether the interface jump is triggered by the success of a specific network request" or "whether the performance degradation is related to a specific operation sequence," thereby enabling verification to go beyond single-point judgment and achieve systematic verification based on causality and timing. By "generating an adaptive verification condition set" based on association rules and preset assertions, the dynamism and intelligence of the verification strategy can be improved. This is conducive to dynamically adjusting the rigor and focus of verification according to different test contexts (such as device performance and network status), making the verification process more realistic and reducing false positives and false negatives. By "performing layered verification based on the adaptive verification condition set," the systematicity and efficiency of the verification process can be improved. This is conducive to distinguishing the primary and secondary aspects of verification, making strict assertions on the critical path while flexibly handling non-core issues, thereby improving the robustness and efficiency of test execution while ensuring the depth of verification.

[0104] In another alternative embodiment, the method described above further includes: Based on the multi-dimensional verification results and the test instruction set, a test adaptation strategy is generated. The test adaptation strategy is used to represent the adaptive adjustment rules for the test execution process. The test adaptation strategy is obtained by analyzing the correlation between the verification exception events in the multi-dimensional verification results and the test instruction set. The test adaptation strategy includes at least one of the following: test step skipping conditions, retry triggering conditions, and execution order optimization rules. Based on the test adaptation strategy, the test instruction set is dynamically reconstructed to generate an adaptive test instruction sequence. Dynamic reconstruction is achieved by modifying the execution logic, parameters, or order of the test instructions. Based on the adaptive test instruction sequence, the application test operation is re-executed. The application test operation adapts to the dynamic behavior changes of the application under test through the closed-loop control of the adaptive test instruction sequence and real-time test feedback.

[0105] As can be seen, implementing this optional embodiment can improve the test system's adaptability and fault tolerance to deviations or anomalies during execution by "generating test adaptation strategies" based on multi-dimensional verification results and test instruction sets. This facilitates intelligent decision-making on how to deal with failures (such as retrying, skipping, or adjusting steps), thereby reducing interruptions to the entire test process caused by non-fatal anomalies (such as network jitter or temporary pop-ups), and improving the pass rate and continuity of testing. By "dynamically reconstructing" the test instruction set based on the test adaptation strategy, the flexibility and intelligence of the test process can be improved, which is conducive to optimizing test paths and parameters based on real-time feedback. This transforms the testing behavior from static script execution into a dynamic process with a certain degree of decision-making ability. Through "closed-loop control of adaptive test instruction sequences and real-time test feedback," the test system can achieve self-adjustment and continuous optimization, enabling the test process to proactively adapt to the dynamic behavioral changes of the application under test (such as the uncertainty of data loading speed and interface response time). This significantly improves the robustness and reliability of automated testing in real and complex environments, moving towards autonomous testing.

[0106] Example 4 Please see Figure 4 , Figure 4 This is a schematic diagram of the structure of another cross-platform automated testing system for multi-dimensional perception analysis disclosed in this invention. This cross-platform automated testing system for multi-dimensional perception analysis can be applied to system platforms such as Android, iOS, Windows, Linux, and HarmonyOS, and can also be applied to intelligent devices related to the system platform. These intelligent devices include, but are not limited to, one or more of mobile terminal devices, switch devices, cloud devices, edge computing devices, relay devices, base station devices, urban management devices, and intelligent connected devices; this invention does not limit the scope of these devices. Figure 4 As shown, this cross-platform automated testing system for multi-dimensional perception and analysis may include: Memory 401 that stores executable program code.

[0107] Processor 402 coupled to memory 401.

[0108] The processor 402 calls the executable program code stored in the memory 401 to execute the steps in the cross-platform automated testing method for multi-dimensional perception analysis described in Embodiment 3 of the present invention.

[0109] Example 5 This invention discloses a computer storage medium storing computer instructions. When these computer instructions are invoked, they are used to execute the steps in the cross-platform automated testing method for multi-dimensional perception analysis described in Embodiment 3 of this invention.

[0110] Example 6 This invention discloses a computer program product, which includes a non-transitory computer storage medium storing a computer program, and the computer program is operable to cause a computer to perform the steps in the cross-platform automated testing method for multi-dimensional perception analysis described in Embodiment 3.

[0111] The system embodiments described above are merely illustrative. The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0112] Through the detailed description of the above embodiments, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, including read-only memory (ROM), random access memory (RAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), one-time programmable read-only memory (OTPROM), electrically-Erasable Programmable Read-Only Memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, disk storage, magnetic tape storage, or any other computer-readable medium that can be used to carry or store data.

[0113] Finally, it should be noted that the cross-platform automated testing system and method for multi-dimensional perception analysis disclosed in the embodiments of the present invention are merely preferred embodiments of the present invention and are only used to illustrate the technical solutions of the present invention, not to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A cross-platform automated testing system for multi-dimensional perception and analysis, characterized in that, The system includes: A cross-platform execution layer used to obtain runtime interface data of the application under test; The multi-dimensional perception and recognition analysis layer is used to perform corresponding multi-dimensional perception and analysis operations on the running interface data according to the multi-dimensional perception and analysis model, and generate a candidate element set corresponding to the running interface data. The multi-dimensional perception and analysis operations include at least two of the following: text recognition and analysis, semantic understanding and analysis, icon recognition and analysis, interface layout recognition and analysis, and action intent prediction and analysis. The comprehensive target localization layer is used to determine the target element set corresponding to the running interface data based on the preset multi-dimensional feature similarity comprehensive calculation model and the candidate element set; The cross-platform execution layer is also used to execute application test operations adapted to the application under test based on the target element set and the pre-parsed test instruction set.

2. The cross-platform automated testing system for multi-dimensional perception analysis according to claim 1, characterized in that, The multi-dimensional perception analysis model includes at least a multi-dimensional feature recognition analysis model, a multi-dimensional semantic understanding analysis model, and an operation intent prediction analysis model. The multi-dimensional perception analysis model is matched with the operation. The multi-dimensional perception recognition analysis layer performs corresponding multi-dimensional perception analysis operations on the running interface data based on the multi-dimensional perception analysis model. The specific methods for generating the candidate element set corresponding to the running interface data include: Based on the multi-dimensional feature recognition and analysis model, the interface data is subjected to interface element recognition and image denoising to obtain preliminary element recognition results. Based on the multi-dimensional semantic understanding analysis model, multimodal semantic understanding operations are performed on the preliminary element recognition results to integrate icon detection, layout analysis, and semantic partitioning features to generate a semantically enhanced element set; Based on the operation intent prediction and analysis model and the action intent instructions in the pre-parsed test instruction set, the semantic enhancement element set is subjected to intent matching and confidence calculation to generate a candidate element set corresponding to the running interface data. Each candidate element in the candidate element set includes its corresponding element attribute, element position coordinates and confidence score.

3. The cross-platform automated testing system for multi-dimensional perception analysis according to claim 1, characterized in that, The comprehensive target localization layer determines the target element set corresponding to the running interface data in the following specific ways based on the preset multi-dimensional feature similarity comprehensive calculation model and the candidate element set: For each candidate element in the candidate element set, similarity information between the candidate element and the target element description data in the pre-parsed test instruction set is extracted across multiple feature dimensions. The feature dimensions include at least text features, local image features, and semantic features. The similarity information on the text features is used to represent the degree of matching between the candidate element and the target element description data in the text content dimension. The similarity information on the local image features is used to represent the degree of matching between the candidate element and the target element description data in the element visual appearance dimension. The similarity information on the semantic features is used to represent the degree of semantic association between the candidate element and the target element description data in the interface context dimension. All the extracted similarity information is input into a preset multi-dimensional feature similarity comprehensive calculation model for weighted fusion calculation. The weighted fusion calculation dynamically adjusts the weight allocation of each feature dimension based on the interface context to generate a comprehensive similarity score for each element. Based on the comprehensive similarity score and the preset confidence threshold, the target element set corresponding to the running interface data is determined.

4. The cross-platform automated testing system for multi-dimensional perception analysis according to claim 1, characterized in that, The specific methods by which the cross-platform execution layer executes application testing operations adapted to the application under test based on the target element set and the pre-parsed test instruction set include: The element position coordinates in the target element set are converted into device-independent normalized coordinate representations, which are then adapted to different screen resolutions and irregular screen display characteristics through safe area correction processing. Based on the operation types and parameters defined in the pre-parsed test instruction set, a platform adaptation component matching the running platform of the application under test is invoked to map the normalized coordinate representation into executable operation instructions for a specific platform corresponding to the application under test, so as to execute application test operations adapted to the application under test. The operation types include at least one of click, swipe, long press, double click, drag, zoom, text input, and gesture operation.

5. The cross-platform automated testing system for multi-dimensional perception analysis according to any one of claims 1-4, characterized in that, The system also includes: The assertion verification layer is used to capture multi-dimensional feedback data in real time during the execution of the application test operation. The multi-dimensional feedback data includes at least one of the following: interface response status data, operation trajectory sequence, network request log data, system event log data, and performance indicator data. The assertion verification layer is also used to compare and verify the multi-dimensional feedback data with the preset assertion conditions in the test instruction set to obtain multi-dimensional verification results. The assertion verification layer is also used to generate a traceable test report of the application test operation based on the multi-dimensional verification results.

6. The cross-platform automated testing system for multi-dimensional perception analysis according to claim 5, characterized in that, The assertion verification layer compares and verifies the multi-dimensional feedback data with the preset assertion conditions in the test instruction set, and obtains the multi-dimensional verification results in the following specific ways: Cross-dimensional correlation analysis is performed on the multi-dimensional feedback data to obtain inter-dimensional correlation rules. The inter-dimensional correlation rules are used to represent the statistical dependence or temporal relationship between different dimension feedback data. The inter-dimensional correlation rules include at least one of causal relationship rules, temporal constraint rules, and statistical correlation rules. Based on the inter-dimensional association rules and the preset assertion conditions, an adaptive verification condition set is generated. The adaptive verification condition set is used to dynamically adjust the strictness or priority of verification. The adaptive verification condition set includes at least one of dynamic threshold conditions, verification priority rules, and condition combination rules. Based on the adaptive verification condition set, the multi-dimensional feedback data is subjected to hierarchical verification to generate multi-dimensional verification results.

7. The cross-platform automated testing system for multi-dimensional perception analysis according to claim 6, characterized in that, The system also includes: An adaptive closed-loop layer is used to generate a test adaptation strategy based on the multi-dimensional verification results and the test instruction set. The test adaptation strategy represents an adaptive adjustment rule for the test execution process. The test adaptation strategy is obtained by analyzing the correlation between verification anomaly events in the multi-dimensional verification results and the test instruction set. The test adaptation strategy includes at least one of test step skipping conditions, retry triggering conditions, and execution order optimization rules. The adaptive closed-loop layer is also used to dynamically reconstruct the test instruction set based on the test adaptation strategy to generate an adaptive test instruction sequence. The dynamic reconstruction is achieved by modifying the execution logic, parameters, or order of the test instructions. The adaptive closed-loop layer is also used to re-execute the application test operation according to the adaptive test instruction sequence. The application test operation adapts to the dynamic behavior changes of the application under test through the closed-loop control of the adaptive test instruction sequence and real-time test feedback.

8. A cross-platform automated testing method for multi-dimensional perception analysis, characterized in that, The method includes: Obtain the running interface data of the application under test; Based on the multi-dimensional perception analysis model, the corresponding multi-dimensional perception analysis operation is performed on the running interface data to generate a candidate element set corresponding to the running interface data. The multi-dimensional perception analysis operation includes at least two of the following: text recognition and analysis, semantic understanding and analysis, icon recognition and analysis, interface layout recognition and analysis, and action intent prediction and analysis. Based on the preset multi-dimensional feature similarity comprehensive calculation model and the candidate element set, the target element set corresponding to the running interface data is determined; Based on the target element set and the pre-parsed test instruction set, execute application test operations adapted to the application under test.

9. A cross-platform automated testing system for multi-dimensional perception and analysis, characterized in that, The system includes: Memory containing executable program code; A processor coupled to the memory; The processor calls the executable program code stored in the memory to execute the cross-platform automated testing method for multi-dimensional perception analysis as described in claim 8.

10. A computer storage medium, characterized in that, The computer storage medium stores computer instructions, which, when invoked, are used to execute the cross-platform automated testing method for multi-dimensional perception analysis as described in claim 8.

Images