Contour matching method and device for UI screenshot

Abstract

The invention discloses a UI screenshot contour matching method and device, and the method comprises the steps: obtaining a basic framework model, positioning at least one business scene specified by an automatic test request, and continuously collecting multiple frames of UI screenshots from each business scene; after the UI screenshots are preprocessed, candidate contour images are extracted to construct a training image set; performing structure fine tuning on the basic framework model and migrating an initial weight to generate a fine tuning model; performing iterative training on the fine tuning model by adopting the training image set to obtain a target matching model; after a to-be-matched UI screenshot is received, contour matching is conducted on the to-be-matched UI screenshot through the target matching model, and an initial contour diagram is generated; and performing post-processing optimization on the initial contour map to obtain a target contour map. Therefore, the image set is constructed in combination with preprocessing of dynamic and static UI screenshots, contour matching is performed through iterative training of the fine tuning model, core elements and background interference are effectively distinguished, and the positioning accuracy of the target contour is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of contour matching, in particular to a UI screenshot contour matching method and device. BACKGROUND

[0002] With the rapid development of digital economy, the e-commerce industry has entered the stage of fine operation, and the iteration speed of online business scenarios continues to accelerate. As the core carrier of user interaction and business bearing, the stability and consistency of the UI (User Interface) of APP and PC end become the key link to ensure user experience.

[0003] In the e-commerce platform, the page composition covers multi-dimensional business scenarios such as product detail page, home page, shopping cart, order management, etc. The UI interface of APP and PC end contains a large number of text blocks, function icons, interactive components and other core elements, and a large number of UI screenshots (cumulative tens of thousands) will be generated in the process of business iteration. In order to meet the business needs of internal UI automation testing and cross-version page structure comparison, it is necessary to efficiently extract the contour information of key elements from these screenshots for subsequent contour matching process.

[0004] Although traditional image recognition technology has been applied in contour extraction field, due to the characteristics of high element density, style diversification and obvious scene differentiation in the complex UI scene of e-commerce platform, the traditional method can only extract the contour by fixed threshold edge detection, which cannot distinguish the core elements from the background interference, and the extraction accuracy is limited, which is difficult to meet the demand of accurate positioning of core elements in the automation process, and restricts the accuracy and reliability of subsequent contour matching. SUMMARY

[0005] The present application provides a UI screenshot contour matching method and device, which solves the technical problem that the traditional image recognition technology extracts the contour by fixed threshold edge detection, which cannot distinguish the core elements from the background interference, and the extraction accuracy is limited, which is difficult to meet the demand of accurate positioning of core elements in the automation process, and restricts the accuracy and reliability of subsequent contour matching.

[0006] The present application provides a UI screenshot contour matching method, comprising:

[0007] In response to an automation test request, a basic framework model is obtained, and at least one business scenario specified by the automation test request is located, and a plurality of frames of UI screenshots are continuously collected from each of the business scenarios;

[0008] After preprocessing each of the UI screenshots, a candidate contour image is extracted to construct a training image set;

[0009] Structurally fine-tune the base framework model, and migrate the feature extraction layer parameters of the pre-trained model as initial weights to generate a fine-tuned model;

[0010] Iteratively train the fine-tuned model using the training image set to obtain a target matching model;

[0011] When receiving a to-be-matched UI screenshot, perform contour matching on the to-be-matched UI screenshot using the target matching model to generate an initial contour map;

[0012] Post-process and optimize the initial contour map to obtain a target contour map.

[0013] Optionally, the UI screenshots include static UI screenshots and dynamic UI screenshots; after pre-processing each of the UI screenshots, the step of extracting candidate contour images to construct a training image set includes:

[0014] After standardizing each of the static UI screenshots, extract at least one static initial contour map;

[0015] According to a preset UI design specification, perform contour correction on the static initial contour map to obtain a static corrected contour map;

[0016] Associate the static corrected contour map with a structured label to obtain a static candidate contour image;

[0017] According to a preset screening rule, perform key frame screening and standardization on the dynamic UI screenshots to obtain a plurality of initial dynamic screenshots;

[0018] From each of the initial dynamic screenshots, extract at least one dynamic initial contour map;

[0019] According to the time sequence relationship of each of the initial dynamic screenshots, assign a time sequence association identifier to the same dynamic initial contour map within different initial dynamic screenshots to obtain a dynamic candidate contour image;

[0020] Use all of the static candidate contour images and the dynamic candidate contour images to construct a training image set.

[0021] Optionally, the base framework model is a ResNet network; the step of structurally fine-tuning the base framework model and migrating the feature extraction layer parameters of the pre-trained model as initial weights to generate a fine-tuned model includes:

[0022] Fuse a convolution layer in a residual layer of the ResNet network to form a 5-layer convolution structure;

[0023] A first convolutional kernel is deployed at the output of each residual layer, and a second convolutional kernel is deployed for each residual layer. A third convolutional kernel is deployed in the last two layers of the convolutional structure, and then connected to a pooling layer to obtain a first intermediate model.

[0024] A second intermediate model is obtained by associating a 3D convolutional layer after the first intermediate model; the 3D convolutional layer is used to extract the temporal motion features of the dynamic candidate contour image.

[0025] Freeze the first three convolutional parameters of the second intermediate model, transfer the feature extraction layer parameters of the pre-trained model as initial weights, and generate a fine-tuned model.

[0026] Optionally, the step of iteratively training the fine-tuned model using the training image set to obtain the target matching model includes:

[0027] The training image set is input into the fine-tuning model in batches, and the model outputs a predicted contour map.

[0028] The loss function value is calculated based on the predicted contour map and the real contour map corresponding to the training image set;

[0029] If the loss function value is less than the preset loss threshold, the fine-tuned model at the current moment is determined as the target matching model.

[0030] If the loss function value is not less than the preset loss threshold, then the model parameters of the fine-tuned model are adjusted according to the preset adjustment gradient, and the process jumps to the step of inputting the training image set into the fine-tuned model in batches and outputting the predicted contour map.

[0031] Optionally, the step of post-processing and optimizing the initial contour map to obtain the target contour map includes:

[0032] Select an initial contour map whose contour area is greater than or equal to a preset contour area threshold;

[0033] Calculate the intersection-union ratio between adjacent initial contour maps;

[0034] If the cross-union ratio is greater than or equal to a preset cross-union ratio threshold, then adjacent initial contour maps are merged to obtain a combined contour map.

[0035] The target contour map is obtained by prioritizing the UI element types corresponding to each of the combined contour maps.

[0036] The present invention also provides a contour matching device for UI screenshots, comprising:

[0037] The screenshot acquisition module is used to respond to automated test requests, obtain the basic framework model, locate at least one business scenario specified by the automated test request, and continuously collect multiple frames of UI screenshots from each of the business scenarios.

[0038] The training image set construction module is used to preprocess each of the UI screenshots and extract candidate contour images to construct a training image set;

[0039] The model fine-tuning module is used to fine-tune the structure of the basic framework model and transfer the feature extraction layer parameters of the pre-trained model as initial weights to generate a fine-tuned model.

[0040] The model training module is used to iteratively train the fine-tuned model using the training image set to obtain a target matching model.

[0041] The contour matching module is used to perform contour matching on the UI screenshot to be matched using the target matching model after receiving the UI screenshot to be matched, and generate an initial contour image.

[0042] The contour image post-processing module is specifically used to perform post-processing optimization on the initial contour image to obtain the target contour image.

[0043] Optionally, the UI screenshots include static UI screenshots and dynamic UI screenshots; the training image set construction module is specifically used for:

[0044] After standardizing each of the static UI screenshots, at least one static initial outline image is extracted;

[0045] The static initial outline is corrected according to the preset UI design specifications to obtain the static corrected outline.

[0046] By associating structured labels with the static corrected contour map, a static candidate contour image is obtained;

[0047] The dynamic UI screenshots are filtered and standardized according to preset filtering rules to obtain multiple initial dynamic screenshots.

[0048] Extract at least one dynamic initial contour map from each of the initial dynamic screenshots;

[0049] According to the temporal relationship of each initial dynamic screenshot, the same dynamic initial contour image in different initial dynamic screenshots is assigned a temporal association identifier to obtain dynamic candidate contour images;

[0050] A training image set is constructed using all the static candidate contour images and the dynamic candidate contour images.

[0051] Optionally, the basic framework model is a ResNet network; the model fine-tuning module is specifically used for:

[0052] The residual layers of the ResNet network are fused with convolutional layers to form a 5-layer convolutional structure;

[0053] A first convolutional kernel is deployed at the output of each residual layer, and a second convolutional kernel is deployed for each residual layer. A third convolutional kernel is deployed in the last two layers of the convolutional structure, and then connected to a pooling layer to obtain a first intermediate model.

[0054] A second intermediate model is obtained by associating a 3D convolutional layer after the first intermediate model; the 3D convolutional layer is used to extract the temporal motion features of the dynamic candidate contour image.

[0055] Freeze the first three convolutional parameters of the second intermediate model, transfer the feature extraction layer parameters of the pre-trained model as initial weights, and generate a fine-tuned model.

[0056] Optionally, the model training module is specifically used for:

[0057] The training image set is input into the fine-tuning model in batches, and the model outputs a predicted contour map.

[0058] The loss function value is calculated based on the predicted contour map and the real contour map corresponding to the training image set;

[0059] If the loss function value is less than the preset loss threshold, the fine-tuned model at the current moment is determined as the target matching model.

[0060] If the loss function value is not less than the preset loss threshold, then the model parameters of the fine-tuned model are adjusted according to the preset adjustment gradient, and the process jumps to the step of inputting the training image set into the fine-tuned model in batches and outputting the predicted contour map.

[0061] Optionally, the contour map post-processing module is specifically used for:

[0062] Select an initial contour map whose contour area is greater than or equal to a preset contour area threshold;

[0063] Calculate the intersection-union ratio between adjacent initial contour maps;

[0064] If the cross-union ratio is greater than or equal to a preset cross-union ratio threshold, then adjacent initial contour maps are merged to obtain a combined contour map.

[0065] The target contour map is obtained by prioritizing the UI element types corresponding to each of the combined contour maps.

[0066] As can be seen from the above technical solutions, the present invention has the following advantages:

[0067] This invention responds to automated testing requests, acquires a basic framework model, and locates at least one business scenario specified in the automated testing request. Multiple UI screenshots are continuously collected from each business scenario. After preprocessing each UI screenshot, candidate contour images are extracted to construct a training image set. The basic framework model is structurally fine-tuned, and the feature extraction layer parameters of the pre-trained model are transferred as initial weights to generate a fine-tuned model. The fine-tuned model is iteratively trained using the training image set to obtain a target matching model. When a UI screenshot to be matched is received, the target matching model is used to perform contour matching on the UI screenshot to be matched, generating an initial contour image. The initial contour image is post-processed and optimized to obtain the target contour image. Thus, by combining the preprocessing of dynamic and static UI screenshots to construct an image set, and simultaneously performing contour matching through iterative training of the fine-tuned model, the accuracy of target contour localization is effectively distinguished from core elements and background interference. Attached Figure Description

[0068] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0069] Figure 1 A flowchart illustrating the steps of a UI screenshot contour matching method provided in this embodiment of the invention;

[0070] Figure 2 This is a structural block diagram of a UI screenshot contour matching device provided in an embodiment of the present invention. Detailed Implementation

[0071] This invention provides a method and apparatus for contour matching of UI screenshots, which solves the technical problem that traditional image recognition technology extracts contours through edge detection with a fixed threshold, which cannot distinguish between core elements and background interference. At the same time, the extraction accuracy is limited, making it difficult to meet the requirements of accurate positioning of core elements in automated processes, thus restricting the accuracy and reliability of subsequent contour matching.

[0072] To make the objectives, features, and advantages of this invention more apparent and understandable, the technical solutions of the embodiments of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the embodiments described below are only some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0073] Please see Figure 1 , Figure 1 The flowchart illustrates the steps of a UI screenshot contour matching method provided in this embodiment of the invention.

[0074] This invention provides a contour matching method for UI screenshots, comprising:

[0075] Step 101: Respond to the automated testing request, obtain the basic framework model, and locate at least one business scenario specified in the automated testing request. Collect multiple frames of UI screenshots continuously from each business scenario.

[0076] An automated test request refers to an automated execution instruction initiated by the testing system to verify the integrity, consistency, or interactive functionality of UI elements. It includes key information such as test objectives, business scope, and business scenarios.

[0077] The basic framework model refers to a fully convolutional neural network model, such as the ResNet network, that is built based on an open-source semantic segmentation library and has basic contour extraction capabilities.

[0078] Business scenarios refer to the business interaction scenarios of the e-commerce platform that the automated testing request needs to test, including but not limited to the APP and PC. Different UI elements may exist in the same business scenario on different terminals.

[0079] UI screenshots refer to continuous interface images captured in a time sequence within the target business scenario, including but not limited to static stable frames and dynamic transition frames, which fully reproduce the state changes of UI elements.

[0080] In the UI screenshot contour matching method of this embodiment, data acquisition and processing can be authorized beforehand. For parts involving user privacy, such as shopping cart operations and order submissions, desensitization processing can be performed on an offline deployed edge device before subsequent processing. This method can be implemented through edge-cloud collaboration. For example, UI screenshot acquisition and preprocessing can be performed on a local edge device, while operations requiring large computational resources, such as model fine-tuning and training, can be executed on a cloud server. After training the target matching model on the cloud server, the UI screenshot to be matched is obtained from the business scenario to be matched. After preprocessing on the edge device, the target matching model is called via the cloud to perform contour matching on the UI screenshot to be matched to generate an initial contour map. Post-processing optimization is then performed to obtain the target contour map, which is stored and awaits subsequent retrieval by automated testers or software.

[0081] In this embodiment, after responding to the automated testing request initiated by the testing system, a fully convolutional neural network is obtained from the open-source semantic-segmentation library as the basic framework model. Based on the test scope specified in the request, at least one core business scenario is located (such as browsing product details page, shopping cart operation, order submission, etc.). Through embedded acquisition tools or system interfaces, multiple frames of UI screenshots are continuously captured during the normal operation of each business scenario, covering different states such as static elements and dynamic interactions (such as pop-up / collapse, page scrolling). All screenshots are classified and stored according to the "business scenario-frame sequence" rule to ensure the temporal correlation between frames.

[0082] Step 102: After preprocessing each UI screenshot, extract candidate contour images to construct a training image set;

[0083] Candidate contour images refer to image regions that are initially consistent with valid contour features and extracted from preprocessed screenshots using image recognition technology. They include the contours of core UI elements and a small number of interfering contours.

[0084] The training image set refers to a dataset used for model training, consisting of candidate contour images and corresponding annotation information (such as valid contour labels and business attribute labels).

[0085] In this embodiment, the collected multi-frame UI screenshots are preprocessed, including filtering out low-quality screenshots that are blurry, distorted, or severely occluded; unifying the image format and resolution; performing noise reduction, brightness and contrast calibration; using an optimized traditional image recognition algorithm (combining edge detection and dynamic threshold adjustment) to extract candidate contour images; filtering out obvious invalid interference contours (such as background texture edges and pixel-level noise); labeling the candidate contour images; supplementing business attribute tags and temporal correlation information (for dynamic frames); dividing the training subset and validation subset proportionally; and constructing a training image set with complete structure and standardized labeling.

[0086] Furthermore, adaptive preprocessing strategies can be introduced to automatically adjust processing parameters based on the brightness, resolution, and scene type of the screenshot (e.g., enhancing motion blur correction for dynamic scene screenshots). Human-computer collaborative annotation tools are employed to automatically verify the compliance of candidate contours through a rule base, requiring only manual intervention to supplement missing contours and correct erroneous annotations, thus improving annotation efficiency.

[0087] In one example of this application, the UI screenshot includes static UI screenshots and dynamic UI screenshots; step 102 may include the following sub-steps:

[0088] After standardizing each static UI screenshot, extract at least one static initial outline image;

[0089] The static initial outline is corrected according to the preset UI design specifications to obtain the static corrected outline.

[0090] By associating structured labels with the static modified contour map, a static candidate contour image is obtained;

[0091] The dynamic UI screenshots are filtered and standardized according to the preset filtering rules to obtain multiple initial dynamic screenshots.

[0092] Extract at least one dynamic initial contour map from each initial dynamic screenshot;

[0093] Based on the temporal relationship of each initial dynamic screenshot, the same initial dynamic contour image in different initial dynamic screenshots is assigned a temporal association identifier to obtain dynamic candidate contour images;

[0094] A training image set is constructed using all static and dynamic candidate contour images.

[0095] A static UI screenshot refers to a single-frame image of a UI interface in a stable state where elements do not move.

[0096] Dynamic UI screenshots refer to continuous frame images of a UI interface during element movement or state transitions, including dynamic transitions and interactive processes.

[0097] In this embodiment, the collected static UI screenshots are standardized to unify image format and quality parameters. A static initial outline is extracted using traditional image recognition technology. Then, based on preset UI design specifications, irregular and positionally offset outlines in the static initial outline are corrected to obtain a static corrected outline that conforms to design standards. Specifically, jagged rounded button outlines are corrected to standard rounded corners (e.g., 8px / 12px), and slanted text block outlines are corrected to horizontal / vertical orientations. Outlines that do not conform to design specifications are marked with warnings (e.g., "Text block font should be Source Han Sans, current detection shows deviation," "Button area does not meet 100×40 pixel standard"). Subsequently, structured tags containing information such as business type and interaction attributes are associated with the static corrected outline to form static candidate outline images.

[0098] Simultaneously, the collected dynamic UI screenshots are filtered for keyframes according to preset filtering rules, such as frames where pop-ups are completely closed, frames where scrolling is stable, frames where loading is complete, transition frames where pop-ups reach 50% completion, and frames where product blocks are centered during scrolling. Frame images that can represent the core state and key transition nodes are retained and standardized to obtain multiple initial dynamic screenshots. After extracting dynamic initial contour maps from each initial dynamic screenshot, a unified temporal association identifier is assigned to the dynamic initial contour maps corresponding to the same UI elements in different frames according to the temporal relationship of each initial dynamic screenshot, clarifying the continuity of elements between frames and forming dynamic candidate contour images.

[0099] Finally, all static candidate contour images are integrated with dynamic candidate contour images to construct a training image set that covers both static and dynamic business scenarios and contains business semantics and temporal correlation information.

[0100] Step 103: Fine-tune the structure of the basic framework model and transfer the feature extraction layer parameters of the pre-trained model as initial weights to generate the fine-tuned model.

[0101] Pre-trained models refer to neural network models that have been trained on large-scale general image or UI-specific datasets and have mature feature extraction capabilities.

[0102] Feature extraction layer parameters refer to the weight parameters of network layers (such as convolutional layers) in a pre-trained model that are responsible for extracting features such as image edges, shapes, and semantics.

[0103] In this embodiment, based on the characteristics of UI elements (such as text blocks, function icons, and the shape and distribution characteristics of interactive components), the basic framework model is structurally fine-tuned, including adjusting the convolution kernel combination method and the number of convolution layers, optimizing the activation function and pooling logic, and embedding a spatial attention module and a semantic association inference layer. A pre-trained model optimized for UI element segmentation is selected, and its feature extraction layer parameters are transferred to the fine-tuned basic framework model as initial weights. The parameters of the underlying core feature extraction layer are frozen, and only the trainability of the top network layer and the newly added modules is retained, generating a fine-tuned model adapted to the business scenario.

[0104] Furthermore, the basic framework model is a ResNet network; step 103 may include the following sub-steps:

[0105] The residual layers of the ResNet network are fused with convolutional layers to form a 5-layer convolutional structure;

[0106] The first convolutional kernel is deployed at the output of each residual layer, and the second convolutional kernel is deployed for the residual layer. The third convolutional kernel is deployed in the last two layers of the convolutional structure, and then connected to the pooling layer to obtain the first intermediate model.

[0107] A 3D convolutional layer is associated after the first intermediate model to obtain the second intermediate model; the 3D convolutional layer is used to extract the temporal motion features of the dynamic candidate contour image.

[0108] Freeze the first three convolutional parameters of the second intermediate model, transfer the feature extraction layer parameters of the pre-trained model as initial weights, and generate a fine-tuned model.

[0109] In this embodiment, the ResNet network is used as the basic framework model, and a dedicated convolutional layer is fused into its residual layer to construct a 5-layer convolutional structure, which not only ensures the ability to extract deep features, but also avoids the performance loss caused by too many layers.

[0110] Specifically, a 1×1 first convolutional kernel is deployed at the output of each residual layer to achieve efficient compression of feature dimensions and enhancement of core features. At the same time, a 3×3 second convolutional kernel is deployed inside the residual layer to accurately capture low- to mid-dimensional local features such as edges and textures of UI elements. A 5×5 third convolutional kernel is deployed in the last two layers of the 5-layer convolutional structure to extract high-dimensional global features and business semantic features of UI elements. Then, a pooling layer is connected to complete feature dimensionality reduction and key information retention to obtain the first intermediate model.

[0111] To adapt to the contour extraction requirements of dynamic UI scenarios, a 3D convolutional layer is associated after the first intermediate model, enabling it to capture the temporal motion features of dynamic candidate contour images, thus forming a second intermediate model.

[0112] To further reduce training computation, accelerate model convergence, and retain general feature extraction capabilities, the first three convolutional parameters of the second intermediate model are frozen. A pre-trained model optimized for UI element segmentation is selected, and its feature extraction layer parameters are transferred to the second intermediate model as initial weights. Finally, a fine-tuned model that balances static feature extraction accuracy, dynamic temporal adaptability, and training efficiency is generated.

[0113] The fine-tuning model also includes corresponding activation and loss functions.

[0114] Step 104: Iteratively train the fine-tuning model using the training image set to obtain the target matching model;

[0115] In this embodiment, the constructed training image set is input into the fine-tuning model in batches. A multi-loss function collaborative optimization strategy (combining classification loss, small target extraction loss, temporal consistency loss, etc.) is used to calculate the model prediction error, and the trainable layer parameters are updated through the backpropagation algorithm. After each iteration, the model's contour extraction accuracy, the proportion of interfering contours, and other indicators are evaluated using a validation subset. The learning rate and loss function weights are dynamically adjusted. If the indicators do not improve significantly after several consecutive iterations, an early stopping mechanism is triggered, the current optimal parameters are saved, and a target matching model with accurate contour matching capability is obtained.

[0116] In one example of this application, step 104 may include the following sub-steps:

[0117] The training image set is input into the model in batches for fine-tuning, and the predicted contour map is output.

[0118] The loss function value is calculated based on the predicted contour map and the real contour map corresponding to the training image set.

[0119] If the loss function value is less than the preset loss threshold, the fine-tuned model at the current time is determined as the target matching model.

[0120] If the loss function value is not less than the preset loss threshold, then adjust the model parameters of the fine-tuning model according to the preset gradient adjustment, and jump to the step of inputting the training image set into the fine-tuning model in batches and outputting the predicted contour map.

[0121] In this embodiment, the constructed training image set is divided into reasonable batches and input into the fine-tuning model in batches. The model, through its learned UI feature extraction logic and dynamic temporal feature capture capability, outputs the predicted contour map for each batch. Based on a preset loss function (adapting to the classification and regression requirements of contour extraction), the difference between the predicted contour map of each batch and the true contour map of the training image set is calculated to obtain the loss function value. It should be noted that this loss function value can be obtained by weighting classification loss, small object extraction loss (adapting to small UI icons), and temporal consistency loss (adapting to dynamic scenes) with preset weights.

[0122] The calculated loss function value is compared with the preset loss threshold. If the loss function value is less than the preset loss threshold, it means that the current contour prediction accuracy of the model meets the business requirements, and the fine-tuned model at the current moment is determined as the target matching model.

[0123] If the loss function value does not reach the preset loss threshold, it indicates that the model still has room for optimization. Following the preset gradient adjustment parameter update process, the trainable parameters of the fine-tuned model (such as convolutional layer weights and attention module coefficients) are adjusted using the backpropagation algorithm. After the update is complete, the process jumps to the step of inputting the training image set into batches to fine-tune the model, repeating iterative training until the loss function value meets the preset conditions. During training, an adaptive learning rate strategy can be used, dynamically adjusting the learning rate and iteration step size according to the rate of change of the loss function value. When the loss decreases rapidly, the step size is reduced for accurate convergence; when the loss stagnates, the step size is appropriately increased to break through the bottleneck.

[0124] Step 105: After receiving the UI screenshot to be matched, use the target matching model to perform contour matching on the UI screenshot to generate an initial contour image.

[0125] The UI screenshot to be matched refers to the target UI interface image that needs to be extracted and matched during the automated testing process.

[0126] The initial contour map refers to the image directly output by the target matching model, which contains valid contours and a small number of residual interference contours.

[0127] In this embodiment, after receiving the UI screenshot to be matched transmitted by the automated testing system, the screenshot is formatted and optimized according to the preprocessing standard and then input into the target matching model. The model identifies the outlines of valid UI elements (such as text blocks, core icons, and interactive components) in the screenshot through feature extraction, semantic association reasoning, and other processes, marks the position, shape, and business attribute information of the outlines, and outputs an initial outline map containing all identified outlines, along with the priority and association group information of the outlines.

[0128] Furthermore, after inputting a UI screenshot to be matched, the model first determines its scene type (static / dynamic, specific business scenario) and automatically loads the corresponding scene's optimized parameters to improve the scene adaptability of contour matching. A multi-model fusion matching mechanism is further introduced to cross-validate the outputs of the target matching model and the lightweight emergency model, correcting misidentified contours from a single model.

[0129] Step 106: Perform post-processing optimization on the initial contour map to obtain the target contour map.

[0130] In this embodiment, after obtaining the initial contour map, residual interfering contours, such as ultra-small noise contours and irregularly shaped false contours, are filtered and removed according to business rules. The split contours of the same UI component are merged, and broken contours in dynamic or occluded scenarios are completed through contextual feature inference. Combining the UI design specifications of the e-commerce platform, the shape and positional deviations of the contours are corrected. The combined contour maps are prioritized according to UI element types to generate a precise, complete, and structured target contour map.

[0131] In addition, accuracy scores can be given for each valid contour in the target contour map.

[0132] In one example of this application, step 106 may include the following sub-steps:

[0133] Select an initial contour map whose contour area is greater than or equal to a preset contour area threshold;

[0134] Calculate the intersection-union ratio between adjacent initial contour maps;

[0135] If the cross-union ratio is greater than or equal to the preset cross-union ratio threshold, then the adjacent initial contour maps are merged to obtain a combined contour map.

[0136] The target outline is obtained by prioritizing the UI element types corresponding to each combined outline.

[0137] In this embodiment, images with contour areas greater than or equal to a preset contour area threshold are selected from the initial contour map output by the target matching model, and ultra-small noise interference contours are filtered out, retaining valid initial contour maps with practical business significance.

[0138] By using image geometric calculation methods, the cross-union ratio (CUNR) between adjacent valid initial contour images is calculated to quantify the degree of overlap between adjacent contours. The calculated CUNR is compared with a preset CUNR threshold. If the CUNR is greater than or equal to the threshold, the corresponding adjacent initial contour images are determined to belong to the same UI component. They are then merged into a combined contour image with a complete shape and accurate boundaries to avoid contour redundancy caused by splitting the same component.

[0139] Then, based on the characteristics of the business scenario, all combined outlines are prioritized according to the business importance of UI element types (such as core interaction buttons > text blocks > function icons > auxiliary decorative elements), clarifying the processing order of subsequent outline matching, and finally forming a target outline that is accurate and adaptable to business needs.

[0140] In addition, when prioritizing, UI element type and business semantic weight can be combined simultaneously. For example, according to the business process of browsing-adding to cart-placing an order-paying, different business weights can be assigned to different types of UI elements, making the sorting more suitable for business needs.

[0141] It should be understood that although the steps in the flowcharts of the above embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0142] The outline matching device for UI screenshots provided in the embodiments of this application will be described below. The outline matching device for UI screenshots described below can be referred to in correspondence with the outline matching method for UI screenshots described above.

[0143] Please see Figure 2 The present invention also provides a contour matching device for UI screenshots, comprising:

[0144] The screenshot acquisition module 201 is used to respond to automated test requests, obtain the basic framework model, locate at least one business scenario specified in the automated test request, and continuously collect multiple frames of UI screenshots from each business scenario.

[0145] The training image set construction module 202 is used to preprocess each UI screenshot and extract candidate contour images to construct a training image set;

[0146] The model fine-tuning module 203 is used to fine-tune the structure of the basic framework model and transfer the feature extraction layer parameters of the pre-trained model as initial weights to generate a fine-tuned model.

[0147] The model training module 204 is used to iteratively train the fine-tuned model using a training image set to obtain a target matching model.

[0148] The contour matching module 205 is used to perform contour matching on the UI screenshot to be matched using the target matching model after receiving the UI screenshot to be matched, and generate an initial contour image.

[0149] The contour image post-processing module 206 is specifically used to perform post-processing optimization on the initial contour image to obtain the target contour image.

[0150] Optionally, UI screenshots include static UI screenshots and dynamic UI screenshots; the training image set construction module 202 is specifically used for:

[0151] After standardizing each static UI screenshot, extract at least one static initial outline image;

[0152] The static initial outline is corrected according to the preset UI design specifications to obtain the static corrected outline.

[0153] By associating structured labels with the static modified contour map, a static candidate contour image is obtained;

[0154] The dynamic UI screenshots are filtered and standardized according to the preset filtering rules to obtain multiple initial dynamic screenshots.

[0155] Extract at least one dynamic initial contour map from each initial dynamic screenshot;

[0156] Based on the temporal relationship of each initial dynamic screenshot, the same initial dynamic contour image in different initial dynamic screenshots is assigned a temporal association identifier to obtain dynamic candidate contour images;

[0157] A training image set is constructed using all static and dynamic candidate contour images.

[0158] Optionally, the base framework model is a ResNet network; the model fine-tuning module 203 is specifically used for:

[0159] The residual layers of the ResNet network are fused with convolutional layers to form a 5-layer convolutional structure;

[0160] The first convolutional kernel is deployed at the output of each residual layer, and the second convolutional kernel is deployed for the residual layer. The third convolutional kernel is deployed in the last two layers of the convolutional structure, and then connected to the pooling layer to obtain the first intermediate model.

[0161] A 3D convolutional layer is associated after the first intermediate model to obtain the second intermediate model; the 3D convolutional layer is used to extract the temporal motion features of the dynamic candidate contour image.

[0162] Freeze the first three convolutional parameters of the second intermediate model, transfer the feature extraction layer parameters of the pre-trained model as initial weights, and generate a fine-tuned model.

[0163] Optionally, the model training module 204 is specifically used for:

[0164] The training image set is input into the model in batches for fine-tuning, and the predicted contour map is output.

[0165] The loss function value is calculated based on the predicted contour map and the real contour map corresponding to the training image set.

[0166] If the loss function value is less than the preset loss threshold, the fine-tuned model at the current time is determined as the target matching model.

[0167] If the loss function value is not less than the preset loss threshold, then adjust the model parameters of the fine-tuning model according to the preset gradient adjustment, and jump to the step of inputting the training image set into the fine-tuning model in batches and outputting the predicted contour map.

[0168] Optionally, the contour plot post-processing module 205 is specifically used for:

[0169] Select an initial contour map whose contour area is greater than or equal to a preset contour area threshold;

[0170] Calculate the intersection-union ratio between adjacent initial contour maps;

[0171] If the cross-union ratio is greater than or equal to the preset cross-union ratio threshold, then the adjacent initial contour maps are merged to obtain a combined contour map.

[0172] The target outline is obtained by prioritizing the UI element types corresponding to each combined outline.

[0173] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or modules may be electrical, mechanical, or other forms.

[0174] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0175] Furthermore, the functional modules in the various embodiments of the present invention can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The integrated modules described above can be implemented in hardware or as software functional modules.

[0176] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for contour matching of UI screenshots, characterized in that, include: In response to an automated testing request, obtain the basic framework model and locate at least one business scenario specified in the automated testing request, and continuously collect multiple frames of UI screenshots from each of the business scenarios. After preprocessing the UI screenshots, candidate contour images are extracted to construct a training image set; The basic framework model is structurally fine-tuned, and the feature extraction layer parameters of the pre-trained model are transferred as initial weights to generate the fine-tuned model. The fine-tuned model is iteratively trained using the training image set to obtain a target matching model; Upon receiving a UI screenshot to be matched, the target matching model is used to perform contour matching on the UI screenshot to generate an initial contour image. The initial contour map is post-processed and optimized to obtain the target contour map.

2. The UI screenshot contour matching method according to claim 1, characterized in that, The UI screenshots include static UI screenshots and dynamic UI screenshots; The step of preprocessing each of the UI screenshots and extracting candidate contour images to construct a training image set includes: After standardizing each of the static UI screenshots, at least one static initial outline image is extracted; The static initial outline is corrected according to the preset UI design specifications to obtain the static corrected outline. By associating structured labels with the static corrected contour map, a static candidate contour image is obtained; The dynamic UI screenshots are filtered and standardized according to preset filtering rules to obtain multiple initial dynamic screenshots. Extract at least one dynamic initial contour map from each of the initial dynamic screenshots; According to the temporal relationship of each initial dynamic screenshot, the same dynamic initial contour image in different initial dynamic screenshots is assigned a temporal association identifier to obtain dynamic candidate contour images; A training image set is constructed using all the static candidate contour images and the dynamic candidate contour images.

3. The UI screenshot contour matching method according to claim 2, characterized in that, The basic framework model is a ResNet network; the steps of fine-tuning the basic framework model and transferring the feature extraction layer parameters of the pre-trained model as initial weights to generate the fine-tuned model include: The residual layers of the ResNet network are fused with convolutional layers to form a 5-layer convolutional structure; A first convolutional kernel is deployed at the output of each residual layer, and a second convolutional kernel is deployed for each residual layer. A third convolutional kernel is deployed in the last two layers of the convolutional structure, and then connected to a pooling layer to obtain a first intermediate model. A second intermediate model is obtained by associating a 3D convolutional layer after the first intermediate model; the 3D convolutional layer is used to extract the temporal motion features of the dynamic candidate contour image. Freeze the first three convolutional parameters of the second intermediate model, transfer the feature extraction layer parameters of the pre-trained model as initial weights, and generate a fine-tuned model.

4. The UI screenshot contour matching method according to claim 1, characterized in that, The step of iteratively training the fine-tuned model using the training image set to obtain the target matching model includes: The training image set is input into the fine-tuning model in batches, and the model outputs a predicted contour map. The loss function value is calculated based on the predicted contour map and the real contour map corresponding to the training image set; If the loss function value is less than the preset loss threshold, the fine-tuned model at the current moment is determined as the target matching model. If the loss function value is not less than the preset loss threshold, then the model parameters of the fine-tuned model are adjusted according to the preset adjustment gradient, and the process jumps to the step of inputting the training image set into the fine-tuned model in batches and outputting the predicted contour map.

5. The UI screenshot contour matching method according to claim 1, characterized in that, The step of post-processing and optimizing the initial contour map to obtain the target contour map includes: Select an initial contour map whose contour area is greater than or equal to a preset contour area threshold; Calculate the intersection-union ratio between adjacent initial contour maps; If the cross-union ratio is greater than or equal to a preset cross-union ratio threshold, then adjacent initial contour maps are merged to obtain a combined contour map. The target contour map is obtained by prioritizing the UI element types corresponding to each of the combined contour maps.

6. A contour matching device for UI screenshots, characterized in that, include: The screenshot acquisition module is used to respond to automated test requests, obtain the basic framework model, locate at least one business scenario specified by the automated test request, and continuously collect multiple frames of UI screenshots from each of the business scenarios. The training image set construction module is used to preprocess each of the UI screenshots and extract candidate contour images to construct a training image set; The model fine-tuning module is used to fine-tune the structure of the basic framework model and transfer the feature extraction layer parameters of the pre-trained model as initial weights to generate a fine-tuned model. The model training module is used to iteratively train the fine-tuned model using the training image set to obtain a target matching model. The contour matching module is used to perform contour matching on the UI screenshot to be matched using the target matching model after receiving the UI screenshot to be matched, and generate an initial contour image. The contour image post-processing module is specifically used to perform post-processing optimization on the initial contour image to obtain the target contour image.

7. The UI screenshot contour matching device according to claim 6, characterized in that, The UI screenshots include static UI screenshots and dynamic UI screenshots; the training image set construction module is specifically used for: After standardizing each of the static UI screenshots, at least one static initial outline image is extracted; The static initial outline is corrected according to the preset UI design specifications to obtain the static corrected outline. By associating structured labels with the static corrected contour map, a static candidate contour image is obtained; The dynamic UI screenshots are filtered and standardized according to preset filtering rules to obtain multiple initial dynamic screenshots. Extract at least one dynamic initial contour map from each of the initial dynamic screenshots; According to the temporal relationship of each initial dynamic screenshot, the same dynamic initial contour image in different initial dynamic screenshots is assigned a temporal association identifier to obtain dynamic candidate contour images; A training image set is constructed using all the static candidate contour images and the dynamic candidate contour images.

8. The UI screenshot contour matching device according to claim 7, characterized in that, The basic framework model is a ResNet network; the model fine-tuning module is specifically used for: The residual layers of the ResNet network are fused with convolutional layers to form a 5-layer convolutional structure; A first convolutional kernel is deployed at the output of each residual layer, and a second convolutional kernel is deployed for each residual layer. A third convolutional kernel is deployed in the last two layers of the convolutional structure, and then connected to a pooling layer to obtain a first intermediate model. A second intermediate model is obtained by associating a 3D convolutional layer after the first intermediate model; the 3D convolutional layer is used to extract the temporal motion features of the dynamic candidate contour image. Freeze the first three convolutional parameters of the second intermediate model, transfer the feature extraction layer parameters of the pre-trained model as initial weights, and generate a fine-tuned model.

9. The UI screenshot contour matching device according to claim 6, characterized in that, The model training module is specifically used for: The training image set is input into the fine-tuning model in batches, and the model outputs a predicted contour map. The loss function value is calculated based on the predicted contour map and the real contour map corresponding to the training image set; If the loss function value is less than the preset loss threshold, the fine-tuned model at the current moment is determined as the target matching model. If the loss function value is not less than the preset loss threshold, then the model parameters of the fine-tuned model are adjusted according to the preset adjustment gradient, and the process jumps to the step of inputting the training image set into the fine-tuned model in batches and outputting the predicted contour map.

10. The UI screenshot contour matching device according to claim 6, characterized in that, The contour map post-processing module is specifically used for: Select an initial contour map whose contour area is greater than or equal to a preset contour area threshold; Calculate the intersection-union ratio between adjacent initial contour maps; If the cross-union ratio is greater than or equal to a preset cross-union ratio threshold, then adjacent initial contour maps are merged to obtain a combined contour map. The target contour map is obtained by prioritizing the UI element types corresponding to each of the combined contour maps.

Images