Cross-platform GUI test script generation method based on multimodal visual recognition

Abstract

The present invention relates to the technical field of software testing automation, and more specifically, to a cross-platform GUI test script generation method based on multimodal visual recognition. The method provided by the present invention mainly includes sending pre-processed basic data to a visual channel and a text channel for processing, the visual channel performing element detection on the basic data using a YOLOv5 model to obtain a first recognition result, and the text channel performing text recognition on the basic data using a CRNN-OCR model to obtain a second recognition result; the first recognition result and the second recognition result are feature-fused and output to a script generator. Through the above method, conventional traditional tools are no longer used for processing, but a dual-channel processing layer is adopted to analyze and extract the current basic data respectively, which simplifies the workflow and greatly reduces the probability of a large number of selector failures due to front-end framework upgrades.

Description

Technical Field

[0001] The present invention relates to the technical field of software testing automation, and in particular to a cross-platform GUI test script generation method based on multimodal visual recognition. Background Art

[0002] GUIs are graphical user interfaces. Existing techniques typically generate event streams through random interactions with screen coordinates, guide testing by building interface state machines or call graphs, and rely on predefined rules or scripts to control interface elements. LLM's natural language understanding and generation capabilities enable direct translation of test requirements into operation sequences and analysis of parameter dependencies between interfaces, enabling intelligent script generation for multi-step scenarios.

[0003] However, traditional tools (such as Selenium) rely on HTML structure parsing, and front-end framework upgrades cause many selectors to become invalid; insufficient visual adaptation: recording tools based on coordinate positioning (such as QTP) produce large positioning deviations when the resolution changes. Summary of the Invention

[0004] The purpose of the present invention is to provide a cross-platform GUI test script generation method based on multimodal visual recognition to solve the above-mentioned problems in the prior art.

[0005] The present invention is achieved through the following technical solutions:

[0006] A cross-platform GUI test script generation method based on multimodal visual recognition, comprising:

[0007] Acquire basic data to be processed, the basic data including image data or video data, and preprocess the basic data;

[0008] The pre-processed basic data is sent to the visual channel and the text channel for processing respectively. The visual channel performs element detection on the basic data using the YOLOv5 model to obtain a first recognition result, and the text channel performs text recognition on the basic data using the CRNN-OCR model to obtain a second recognition result;

[0009] The first recognition result and the second recognition result are feature fused, the layout relationship is parsed through CNN, the obtained parsing results are output to the script generator, and executable test scripts are generated respectively through the script generator.

[0010] Preferably, the preprocessing of basic data includes:

[0011] Normalizing the resolution of the basic data and outputting first basic data, eliminating noise from the first basic data and performing dynamic element filtering to output second basic data;

[0012] Image enhancement is performed on the second basic data, and the processed basic data is outputted.

[0013] Preferably, the dynamic element filtering includes:

[0014] Setting static conditions and dynamic conditions, wherein setting static conditions includes setting filtering thresholds based on numerical ranges or string matching, and setting dynamic conditions includes obtaining input or real-time data generation conditions;

[0015] Traverse the data set and set a conditional judgment model, which includes performing logical judgment on the attributes of the elements to confirm whether the attributes of the elements are within the set filtering threshold. After the judgment is completed, the data set obtained by filtering is outputted and meets the conditions.

[0016] Preferably, the feature fusion of the first recognition result and the second recognition result comprises cross-modal attention alignment, including:

[0017] Perform bilinear attention interaction:

[0018]

[0019] Perform hierarchical attention aggregation, including using a multi-head mechanism to parallelize multi-granularity associations:

[0020]

[0021] Perform residual feature fusion:

[0022]

[0023]

[0024] Set the loss function and perform contrast loss optimization;

[0025] Where, is the cross-modal similarity score between the first recognition result and the second recognition result, a device for the first recognition result, is the first words or embedding vectors, is the multi-head attention function, is the vector concatenation function, is the calculation result of the 1st to kth attention heads, is the linear transformation matrix of the multi-head attention output, is the first recognition result after residual feature fusion, is the initial first recognition result, is the layer normalization function, Each row corresponds to the association strength of a visual area with all text words, is the second recognition result, and are projection matrices, To perform the second recognition result after residual feature fusion, for The transpose of .

[0026] Preferably, the feature fusion comprises:

[0027] Performing vector normalization on the first recognition result and the second recognition result;

[0028] Concatenate the first and second recognition results into a joint vector:

[0029]

[0030] Where, is the fused joint vector, To calculate the weight.

[0031] Preferably, the performing layout relationship analysis by CNN includes:

[0032] A weighted spatial relationship matrix is ​​constructed, and hierarchical relationships are iteratively inferred through a three-layer GNN. A DOM tree with semantic annotations is output. The DOM tree includes a parent node and the first input box child node, the second input box child node, and the login button child node associated with the parent node.

[0033] Preferably, the constructing of a weighted spatial relationship matrix includes:

[0034]

[0035] Where, is the first Rank Elements of the column, is the natural exponential function, and For the The coordinates of the points, For the The coordinates of the points, is the bandwidth parameter of the Gaussian kernel function.

[0036] Preferably, the iterative inference of hierarchical relationships through a three-layer GNN includes:

[0037] Construct the first layer GNN, the second layer GNN and the third layer GNN, the first layer GNN is used to process the original node features, the second layer GNN is used to embed the nodes of the first layer GNN into the input coarsening module, the third layer GNN aggregates the clusters of the second layer to form semantic units, and maps the clusters of the target layer back to the node relationship of the original graph through the full expansion function.

[0038] Preferably, the sending to the visual channel and the text channel processing includes:

[0039] Determine whether the recognition is successful. If the first recognition result and the second recognition result are successfully obtained, perform feature fusion;

[0040] If the recognition is not successful, the detection threshold is adjusted, SIFT feature matching is used, and the recognition is judged again. If the recognition is successful, the current detection threshold is saved. If the recognition is not successful, an alarm signal is sent and an error report is output.

[0041] Preferably, the script generator generates executable test scripts respectively including:

[0042] The operation reasoning finite state machine includes determining whether the current input element type is a login button or an input box. If it is a login button, it checks the adjacent text and generates a click operation script. If it is an input box, it finds the label associated with the input box and generates a type operation script.

[0043] An assertion generation algorithm is written, wherein the assertions include textual assertions, structural assertions, and visual assertions.

[0044] The technical solution of the present invention has at least the following advantages and beneficial effects:

[0045] The method provided by the present invention mainly includes sending the pre-processed basic data to the visual channel and the text channel for processing, respectively. The visual channel performs element detection on the basic data through the YOLOv5 model to obtain a first recognition result, and the text channel performs text recognition on the basic data through the CRNN-OCR model to obtain a second recognition result; the first recognition result and the second recognition result are feature-fused and output to the script generator. With the above method, conventional traditional tools are no longer used for processing. Instead, a dual-channel processing layer is adopted to analyze and extract the current basic data separately, which simplifies the workflow and greatly reduces the probability of a large number of selector failures due to front-end framework upgrades. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. It should be understood that the following drawings only illustrate certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without paying any creative work.

[0047] Figure 1 It is the overall control flow chart of the present invention;

[0048] Figure 2 Schematic diagram of the processing flow of the visual channel and the text channel of the present invention.

[0049] Figure 3 A schematic diagram of the processing flow of layout relationship analysis of the present invention;

[0050] Figure 4 Schematic diagram of the process flow for handling element overlap conflicts of the present invention;

[0051] Figure 5 A schematic diagram of the processing flow of the operation reasoning finite state machine of the present invention;

[0052] Figure 6 Schematic diagram of the processing flow of element recognition failure of the present invention. DETAILED DESCRIPTION

[0053] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions of the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings herein can be arranged and designed in various different configurations.

[0054] The division of modules in this application is a logical division. In actual application, there may be other division methods. For example, multiple modules can be combined or integrated into another system, or some features can be ignored or not executed.

[0055] Independently described modules or submodules may or may not be physically separate; they may be implemented in software or hardware. Some modules or submodules may be implemented in software, with the processor invoking the software to implement the functionality of these modules or submodules, while other modules or submodules may be implemented in hardware, such as hardware circuits. Furthermore, some or all of the modules may be selected based on actual needs to achieve the objectives of the present application.

[0056] Please refer to Figures 1-6The present invention provides a cross-platform GUI test script generation method based on multimodal visual recognition, comprising:

[0057] S101: Acquire basic data to be processed, the basic data including image data or video data, and pre-process the basic data;

[0058] Regarding the preprocessing in more detail, the basic data is resolution-normalized to output first basic data, the first basic data is noise-eliminated and dynamic element filtered to output second basic data; the second basic data is image-enhanced to output the processed basic data.

[0059] Among them, image enhancement includes grayscale conversion after loading the image (such as converting to HSV or YCrCb space using cv2.cvtColor), and noise reduction through filtering (mean filtering, median filtering). The enhancement method is selected according to the target. For example, histogram equalization: calculate the grayscale histogram and redistribute pixel values ​​through the cumulative distribution function, which is suitable for low-contrast images; Laplacian operator enhancement: highlight edge structure, which needs to be combined with noise reduction preprocessing to prevent noise amplification; local contrast enhancement (CLAHE): adjust the contrast in blocks to solve the problem of local detail loss caused by global equalization; adjust brightness / contrast (such as gamma correction), or combine guided filtering to suppress noise.

[0060] S102: The pre-processed basic data is sent to a visual channel and a text channel for processing respectively. The visual channel performs element detection on the basic data using a YOLOv5 model to obtain a first recognition result, and the text channel performs text recognition on the basic data using a CRNN-OCR model to obtain a second recognition result.

[0061] The resulting first or second recognition results are primarily feature matrices or feature vectors, facilitating subsequent processing. YOLOv5 is employed through multiple model versions (YOLOv5s / m / l / x) to achieve a flexible balance between speed and accuracy. YOLOv5s utilizes the depth_multiple and width_multiple parameters to control network depth and width, resulting in a parameter count of only 7.2M and an inference speed of up to 140 FPS (on a Tesla V100), making it suitable for mobile deployment. Its improved CSP (CrossStage Partial) architecture reduces redundant computation, while the SPPF (Spatial Pyramid Pooling Fast) module shifts parallel max pooling to serial, improving computational efficiency by 30% while maintaining multi-scale feature fusion capabilities. Furthermore, K-means clustering and a genetic algorithm are used before training to dynamically generate optimal anchor boxes, reducing manual parameter tuning costs and improving detection box matching accuracy. Furthermore, an attention mechanism is added: the CBAM module is added at the end of the Backbone to enhance small object detection capabilities.

[0062] The CRNN-OCR model eliminates the need for character segmentation and directly uses CNN to extract image features, RNN to model sequence dependencies, and CTC to transcribe the output text, achieving one-stop recognition from image to text. The CTC loss function allows for misaligned input and output sequence lengths, eliminating the need for character-by-character annotation and significantly reducing data annotation costs. Furthermore, an improved Sauvola algorithm is used, with dynamic window size calculation and candidate sequence correction based on a GUI domain dictionary.

[0063] S103: Fusing features of the first recognition result and the second recognition result, parsing the layout relationship through CNN, outputting the obtained parsing results to the script generator, and generating executable test scripts respectively through the script generator.

[0064] In this embodiment, the generated executable test scripts are mainly Playwright scripts, Puppeteer scripts and HTML test reports, which are output by the output layer.

[0065] The method provided by the present invention mainly includes sending the pre-processed basic data to the visual channel and the text channel for processing, respectively. The visual channel performs element detection on the basic data through the YOLOv5 model to obtain a first recognition result, and the text channel performs text recognition on the basic data through the CRNN-OCR model to obtain a second recognition result; the first recognition result and the second recognition result are feature-fused and output to the script generator. With the above method, conventional traditional tools are no longer used for processing. Instead, a dual-channel processing layer is adopted to analyze and extract the current basic data separately, which simplifies the workflow and greatly reduces the probability of a large number of selector failures due to front-end framework upgrades.

[0066] In an exemplary embodiment of the present invention, dynamic element filtering includes:

[0067] Setting static conditions and dynamic conditions, wherein setting static conditions includes setting filtering thresholds based on numerical ranges or string matching, and setting dynamic conditions includes obtaining input or real-time data generation conditions; traversing the data set and setting a conditional judgment model, wherein the conditional judgment model includes performing logical judgment on the attributes of the elements to confirm whether the attributes of the elements are within the set filtering thresholds, and after the judgment is completed, outputting the filtered data set that meets the conditions.

[0068] In the above steps, the algorithms involved include:

[0069] Recursive filtering algorithm, checks child nodes layer by layer from the root node, and retains the parent node if the child node matches; semantic segmentation and dynamic mask generation, combined with online instance segmentation models (such as Track Anything) to generate semantic masks for dynamic objects, and optimize accuracy through photometric consistency verification. Application scenarios: distinguishing static background from moving objects in dynamic SLAM, improving mapping and positioning robustness; range-based filtering function, implementation logic: traverse the object array and check whether the attribute falls within the specified interval; high-order functions and Lambda expressions, Python's filter(): supports custom functions or Lambda expressions, such as filtering even numbers; dynamic kernel parameter filtering (eBPF), implementation steps: parse the filter expression to generate AST, convert the AST into BPF instructions through BTF information, and dynamically replace the filtering logic in the BPF program.

[0070] In addition, after preprocessing the basic data, an anti-interference dynamic processing mechanism is also adopted, including temporal difference method (inter-frame change detection), feature matching (ORB feature point comparison) and semantic filtering (keyword blacklist mechanism).

[0071] Specifically, the temporal difference method includes inter-frame difference calculation, performing pixel-by-pixel grayscale difference on two or three consecutive frames of images; adaptive threshold segmentation, dynamically adjusting the threshold to improve robustness, for example, based on the statistical characteristics of the difference image; morphological optimization, using dilation and erosion operations to fill holes and remove noise.

[0072] Feature matching includes FAST key point detection, detecting whether 12 consecutive points among 16 pixels around a pixel point satisfy the brightness difference threshold T (such as 20% of the central brightness), and screening corner points; direction calculation (gray centroid method), calculating the moment of the key point neighborhood to determine the main direction; BRIEF descriptor generation, randomly selecting 256 pairs of pixel points in the key point neighborhood to generate a 256-bit binary vector; feature matching (Hamming distance) calculation of descriptor differences.

[0073] Semantic filtering includes text preprocessing, using tools (such as Jieba segmentation) to segment text and filter out meaningless words (such as "de", "shi"); keyword matching, Boolean model: directly matching keywords in the blacklist; dynamic update mechanism, weighted update of the blacklist based on user feedback.

[0074] In an exemplary embodiment of the present invention, the feature fusion of the first recognition result and the second recognition result includes cross-modal attention alignment, including:

[0075] Performing bilinear attention interaction:

[0076]

[0077] Performing hierarchical attention aggregation, including using a multi-head mechanism to calculate multi-granularity associations in parallel:

[0078]

[0079] Performing residual feature fusion: <00002​​​​​​​​​​​​​​​​​​​​​​​​ is the linear transformation matrix of the multi-head attention output, is the first recognition result after residual feature fusion, is the initial first recognition result, is the layer normalization function, Each row corresponds to the association strength of a visual area with all text words, is the second recognition result, and are projection matrices, To perform the second recognition result after residual feature fusion, for The transpose of .

[0084] The text position is mapped to the visual element coordinate system through affine transformation. When the visual classification conflicts with the text semantics (for example, a button is detected but the text is "Cancel"), the rule engine arbitration is initiated.

[0085] Specifically, feature fusion includes:

[0086] Performing vector normalization on the first recognition result and the second recognition result;

[0087] Concatenate the first and second recognition results into a joint vector:

[0088]

[0089] Where, is the fused joint vector, To calculate the weight.

[0090] In an exemplary embodiment of the present invention, performing layout relationship analysis using CNN includes:

[0091] A weighted spatial relationship matrix is ​​constructed, and hierarchical relationships are iteratively inferred through a three-layer GNN. A DOM tree with semantic annotations is output. The DOM tree includes a parent node and the first input box child node, the second input box child node, and the login button child node associated with the parent node.

[0092] Among them, if there is a conflict of element overlap, the current conflict type is obtained to determine whether it is text coverage or area inclusion. If it is text coverage, the z-index priority is calculated and then the element hierarchy is adjusted. If it is area inclusion, a parent-child relationship is established and the semantic consistency is verified. If consistent, they are merged into a container node. If inconsistent, they are split into independent nodes.

[0093] The z-index priority calculation involves the interaction between the rules of the stacking context and element properties. The z-index property only takes effect when the element's position property is relative, absolute, fixed, or sticky. The z-index value of an unpositioned element (position: static) is invalid. Among elements at the same level, elements later in the document flow override those in front by default. If z-index is not set, positioned elements will override non-positioned elements.

[0094] When two elements of the same level are both positioned and their z-index is not auto: the one with a larger z-index value covers the one with a smaller one (such as z-index: 2 covers z-index: 1); if the z-index is the same, the element at the back of the document flow covers the former.

[0095] Within the same stacking context, elements are arranged from bottom to top in the following order (in increasing priority):

[0096] Background and border (the background of the parent element); positioned elements with a negative z-index; block-level non-positioned elements; floating elements; inline non-positioned elements; positioned elements with z-index: 0 or auto; positioned elements with a positive z-index.

[0097] In an exemplary embodiment of the present invention, constructing a weighted spatial relationship matrix includes:

[0098]

[0099] Where, is the first Rank Elements of the column, is the natural exponential function, and For the The coordinates of the points, For the The coordinates of the points, is the bandwidth parameter of the Gaussian kernel function.

[0100] The hierarchical relationships inferred through 3-layer GNN iteration include:

[0101] Construct the first layer GNN, the second layer GNN and the third layer GNN, the first layer GNN is used to process the original node features, the second layer GNN is used to embed the nodes of the first layer GNN into the input coarsening module, the third layer GNN aggregates the clusters of the second layer to form semantic units, and maps the clusters of the target layer back to the node relationship of the original graph through the full expansion function.

[0102] The core technology of hierarchical GNN explanation methods (such as STFExplainer) is to map the target layer clusters back to the node relationships of the original graph through a sufficient expansion function. Its core goal is to associate high-level graph structure (such as clusters) with the underlying node relationships of the original graph to preserve cross-level semantic consistency. In hierarchical graph neural networks (HGNNs), graph data is typically decomposed into multi-level cluster structures (for example, represented by cluster trees). High-level explanations (such as clusters) need to be associated with the local structure of the original graph (node ​​level), but traditional methods cannot directly handle this cross-level mapping, resulting in a lack of traceability and interpretability of high-level explanations. The sufficient expansion function mathematically links abstract high-level cluster explanations to the specific structure of the original graph, resolving the cross-level semantic gap in hierarchical GNN explanations.

[0103] In an exemplary embodiment of the present invention, the sending to the visual channel and the text channel process includes:

[0104] Determine whether the recognition is successful. If the first recognition result and the second recognition result are successfully obtained, perform feature fusion;

[0105] If the recognition is not successful, the detection threshold is adjusted, SIFT feature matching is used, and the recognition is judged again. If the recognition is successful, the current detection threshold is saved. If the recognition is not successful, an alarm signal is sent and an error report is output.

[0106] Among them, SIFT feature matching is an image matching algorithm based on local features. Its goal is to achieve stable detection and matching of key points in images under different scales, rotations, lighting and partial occlusion conditions.

[0107] Gaussian pyramid construction: Gaussian kernels with different scale parameters σ are convolved with the original image to generate a multi-scale image sequence, simulating the human eye's observation characteristics of near and far objects. A Taylor expansion is used to fit a three-dimensional quadratic function, refining the location and scale of extreme points to improve positioning accuracy. The gradient magnitude and direction of pixels in the neighborhood centered on the keypoint are calculated (usually divided into eight directions). The primary direction is the histogram peak, and the secondary directions are other directions with more than 80% of the peak value. The 16×16 neighborhood around the keypoint is divided into 4×4 subregions. The 8-directional gradient histogram is calculated for each subregion, forming a 128-dimensional vector. Normalization: Descriptors are L2-normalized and truncated (for example, limiting the maximum value to 0.2) to reduce interference from lighting changes.

[0108] In an exemplary embodiment of the present invention, the script generator generates executable test scripts respectively, including:

[0109] The operation reasoning finite state machine includes determining whether the current input element type is a login button or an input box. If it is a login button, it checks the adjacent text and generates a click operation script. If it is an input box, it finds the label associated with the input box and generates a type operation script.

[0110] An assertion generation algorithm is written, wherein the assertions include textual assertions, structural assertions, and visual assertions.

[0111] Among them, the Click action script is mainly used to simulate or respond to the user's click behavior, and the Type action script is used to process the command line parameter type or determine the system command type.

[0112] The overall plan can be divided into:

[0113] Input layer: supports multi-resolution screenshot / video stream input;

[0114] Processing layer: four-stage processing pipeline (preprocessing, recognition, parsing, generation);

[0115] Output layer: cross-framework executable scripts and test reports.

[0116] The following examples are given to illustrate the above content:

[0117] To test the generation of a cross-resolution login as background:

[0118] Input: 1920x1080 resolution screenshot of the login page (including username and password input boxes, and login button). Test requirement: Verify that the login is successful.

[0119] Processing: Preprocessing, scaling to baseline resolution, and removing interface noise; Element recognition: Locating two input boxes (Type: text) and one button (Type: button); OCR extraction: "Username" and "Password" label text; Layout analysis: Constructing a Z-shaped visual flow from input box to button.

[0120] Output script.

[0121] Taking responsive layout compatibility testing as the background:

[0122] Input: Comparison screenshots of the same page on mobile (375x812) and desktop (1440x900)

[0123] Processing: Element relationship mapping, establishing cross-resolution element correspondence table, and generating adaptive positioning strategy.

[0124] In addition, the functional units in the various embodiments of the present invention may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0125] If the integrated unit is implemented as a software functional unit and sold or used as a standalone product, it can be stored on a computer-readable storage medium. This computer software product, stored on a storage medium, includes instructions for causing a computer device (which may be a personal computer, server, or network device, etc.) to perform all or part of the steps of the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0126] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention shall be included within the scope of protection of the present invention.

Claims

1. A cross-platform GUI test script generation method based on multimodal visual recognition, characterized in that: include: Acquire basic data to be processed, the basic data including image data or video data, and preprocess the basic data; The pre-processed basic data is sent to the visual channel and the text channel for processing respectively. The visual channel performs element detection on the basic data using the YOLOv5 model to obtain a first recognition result, and the text channel performs text recognition on the basic data using the CRNN-OCR model to obtain a second recognition result; The first recognition result and the second recognition result are subjected to feature fusion, the layout relationship is parsed through CNN, the obtained parsing results are output to the script generator, and the executable test scripts are generated respectively through the script generator; The feature fusion of the first recognition result and the second recognition result includes cross-modal attention alignment, including: Perform bilinear attention interaction: Perform hierarchical attention aggregation, including using a multi-head mechanism to parallelize multi-granularity associations: Perform residual feature fusion: Set the loss function and perform contrast loss optimization; Where, is the cross-modal similarity score between the first recognition result and the second recognition result, a device for the first recognition result, is the first words or embedding vectors, is the multi-head attention function, is the vector concatenation function, is the calculation result of the 1st to kth attention heads, is the linear transformation matrix of the multi-head attention output, is the first recognition result after residual feature fusion, is the initial first recognition result, is the layer normalization function, Each row corresponds to the association strength of a visual area with all text words, is the second recognition result, and are projection matrices, To perform the second recognition result after residual feature fusion, for The transpose of is the parameter matrix.

2. The cross-platform GUI test script generation method based on multimodal visual recognition according to claim 1 is characterized in that: The preprocessing of basic data includes: Normalizing the resolution of the basic data and outputting first basic data, eliminating noise from the first basic data and performing dynamic element filtering to output second basic data; Image enhancement is performed on the second basic data, and the processed basic data is outputted.

3. The cross-platform GUI test script generation method based on multimodal visual recognition according to claim 2 is characterized in that: The dynamic element filtering includes: Setting static conditions and dynamic conditions, wherein setting static conditions includes setting filtering thresholds based on numerical ranges or string matching, and setting dynamic conditions includes obtaining input or real-time data generation conditions; Traverse the data set and set a conditional judgment model, which includes performing logical judgment on the attributes of the elements to confirm whether the attributes of the elements are within the set filtering threshold. After the judgment is completed, the data set obtained by filtering is outputted and meets the conditions.

4. The cross-platform GUI test script generation method based on multimodal visual recognition according to claim 3 is characterized in that: The feature fusion comprises: Performing vector normalization on the first recognition result and the second recognition result; Concatenate the first and second recognition results into a joint vector: Where, is the fused joint vector, To calculate the weight.

5. The cross-platform GUI test script generation method based on multimodal visual recognition according to claim 3 is characterized in that: The layout relationship analysis by CNN includes: A weighted spatial relationship matrix is ​​constructed, and hierarchical relationships are iteratively inferred through a three-layer GNN. A DOM tree with semantic annotations is output. The DOM tree includes a parent node and the first input box child node, the second input box child node, and the login button child node associated with the parent node.

6. The cross-platform GUI test script generation method based on multimodal visual recognition according to claim 5 is characterized in that: The constructing of a weighted spatial relationship matrix comprises: Where, is the first Rank Elements of the column, is the natural exponential function, and For the The coordinates of the points, For the The coordinates of the points, is the bandwidth parameter of the Gaussian kernel function.

7. The cross-platform GUI test script generation method based on multimodal visual recognition according to claim 6 is characterized in that: The hierarchical relationship inference through the three-layer GNN iteration includes: Construct the first layer GNN, the second layer GNN and the third layer GNN, the first layer GNN is used to process the original node features, the second layer GNN is used to embed the nodes of the first layer GNN into the input coarsening module, the third layer GNN aggregates the clusters of the second layer to form semantic units, and maps the clusters of the target layer back to the node relationship of the original graph through the full expansion function.

8. The cross-platform GUI test script generation method based on multimodal visual recognition according to claim 7 is characterized in that: The sending to the visual channel and the text channel processing includes: Determine whether the recognition is successful. If the first recognition result and the second recognition result are successfully obtained, perform feature fusion; If the recognition is not successful, the detection threshold is adjusted, SIFT feature matching is used, and the recognition is judged again. If the recognition is successful, the current detection threshold is saved. If the recognition is not successful, an alarm signal is sent and an error report is output.

9. The cross-platform GUI test script generation method based on multimodal visual recognition according to claim 8, characterized in that: The script generator generates executable test scripts respectively including: The operation reasoning finite state machine includes determining whether the current input element type is a login button or an input box. If it is a login button, it checks the adjacent text and generates a click operation script. If it is an input box, it finds the label associated with the input box and generates a type operation script. An assertion generation algorithm is written, wherein the assertions include textual assertions, structural assertions, and visual assertions.

Images