Large model-based program interface and ui style anomaly detection method

Abstract

The application belongs to the technical field of intelligence, and discloses a program interface and UI style abnormality detection method based on a large model, which comprises the following steps: deploying a plurality of large models special for image comparison; obtaining a UI style atlas and a program interface screenshot set, screening sample pairs with a similarity lower than a first preset threshold based on structural similarity comparison to form a first sample pair set; using a plurality of large models to respectively perform image comparison on the sample pairs in the first sample pair set, and forming a second sample pair set by selecting sample pairs with a number of inconsistent comparison results exceeding half of the number of large models; and locating error pixels of the sample pairs in the second sample pair set through pixel-level comparison, drawing error boxes and reporting. The application adopts a plurality of large models special for image comparison to perform comprehensive judgment. The large model has strong image understanding and feature extraction capabilities, can more accurately identify subtle differences between program interfaces and UI style images, and significantly reduces false positive rate and false negative rate.

Description

Technical Field

[0001] This invention relates to the field of intelligent technology, and in particular to a method for detecting anomalies in program interfaces and UI styles based on large models. Background Technology

[0002] In today's rapidly evolving software development landscape, the quality of the user interface (UI) has become a critical factor in determining a product's success or failure. A visually appealing, consistent UI that conforms to design guidelines not only enhances user experience but also strengthens brand image and market competitiveness. To ensure that the actual UI presented in the application matches the design guidelines provided by the UI designer (usually in the form of UI style diagrams), the development team needs to conduct rigorous UI style anomaly detection.

[0003] Traditional UI style anomaly detection relies primarily on manual methods. Testers or developers need to manually compare program interface screenshots with UI style diagrams, checking for differences in color, layout, font, spacing, icons, etc. This approach has obvious drawbacks:

[0004] 1. Inefficient: Manual inspection is time-consuming and labor-intensive, especially in large projects where there are numerous UI interfaces and complex elements, making manual inspection extremely inefficient.

[0005] 2. High cost: It requires a lot of human resources to check, which increases development costs.

[0006] 3. Prone to errors: Manual inspection is easily affected by subjective factors, which may lead to omissions or misjudgments, resulting in inaccurate test results.

[0007] 4. Difficult to handle complex scenarios: For complex UI interfaces, such as those containing a lot of text, dynamic content, and interactive elements, manual inspection is more difficult.

[0008] To improve the efficiency and accuracy of UI style anomaly detection, the industry has proposed several automated detection methods based on image processing technology. These methods typically utilize traditional image comparison algorithms (such as pixel difference comparison, edge detection, and template matching) to compare program interface screenshots and UI design drafts. However, these methods also have limitations:

[0009] 1. Limited accuracy: Traditional image comparison algorithms have high requirements for image quality and are easily affected by factors such as image noise, lighting changes, and alignment deviations, leading to false alarms or missed alarms.

[0010] 2. Poor adaptability: It is difficult to handle complex UI layouts and diverse UI elements, such as different fonts, icons, rounded corners, shadows, etc.

[0011] 3. Low level of intelligence: It usually requires manual setting of various parameters and thresholds, making it difficult to adapt to different UI design specifications and project requirements.

[0012] 4. Difficulty in handling text: Simple image comparison cannot effectively distinguish differences in text content, which is a very important part of UI style anomalies. Summary of the Invention

[0013] To address the aforementioned problems with existing UI style anomaly detection methods, this invention provides a more accurate, efficient, and intelligent solution, specifically including:

[0014] A method for detecting program interface and UI style anomalies based on large models, including the following steps:

[0015] S1. Deploy several large models specifically for image comparison;

[0016] S2. Obtain the UI style image set and the program interface screenshot set. Based on structural similarity comparison, filter out sample pairs with similarity lower than the first preset threshold to form the first sample pair set.

[0017] S3. Using multiple large models, image comparison is performed on the sample pairs in the first sample pair set respectively. The sample pairs whose number of inconsistencies in the comparison results exceeds half the number of large models are formed into a second sample pair set.

[0018] S4. Locate the erroneous pixels of the sample pairs in the second sample pair set by pixel-level comparison, draw the error box and report it.

[0019] Preferably, step S2 further includes a preprocessing step: ensuring that the images in the UI style image set and the program interface screenshot set have the same pixel size and pixel resolution.

[0020] Preferably, the method for structural similarity comparison includes:

[0021] S21. Calculate the brightness, contrast, and structural similarity between each UI sample in the UI style image set and each program interface sample in the program interface screenshot set;

[0022] S22. Calculate a comprehensive structural similarity score based on the brightness, contrast and structural similarity.

[0023] Preferably, step S4 includes:

[0024] S42. Convert the UI style image and program interface screenshot in the second sample set into RGB three-channel images;

[0025] S43. Perform histogram comparison on the R, G, and B channels of the UI style image and the program interface screenshot respectively, calculate the difference in the number of pixel values ​​in each channel, and record the coordinates of the pixels with different numbers of pixel values.

[0026] S44. Draw error boxes in the screenshot of the program interface based on the pixel coordinates that differ, and report them.

[0027] Preferably, step S4 further includes:

[0028] S41. Perform text region recognition on the UI style diagram and program interface screenshot in the second sample set, and record the coordinates of the recognized text region. Exclude the text region in steps S42 and S43.

[0029] Preferably, if the text area in the program interface screenshot is missing text, an error box is drawn in that text area.

[0030] Preferably, the method further includes: floating the recorded text region coordinates upwards and downwards by several pixels in the vertical direction to obtain extended text region coordinates.

[0031] Beneficial effects

[0032] 1. High accuracy: This invention employs a large-scale model for comprehensive judgment using multiple image comparisons. This large-scale model possesses powerful image understanding and feature extraction capabilities, enabling it to more accurately identify subtle differences between the program interface and the UI style diagram, significantly reducing false positive and false negative rates.

[0033] 2. High Efficiency: This invention employs an automated detection process, eliminating the need for manual comparison, significantly improving the efficiency of UI style anomaly detection and shortening the development cycle. It reduces reliance on manual labor, saving costs and enhancing quality and user experience.

[0034] 3. Precise Positioning: This invention combines pixel-level comparison technology to precisely locate specific pixels where there are differences between the UI style image and the program interface screenshot, providing developers with more accurate and detailed error information for quick and easy repair.

[0035] 4. Intelligent Text Processing: This invention can identify and eliminate interference from text regions, avoiding the impact of differences in text content on image comparison results. Simultaneously, it can also detect issues such as missing text, further improving the comprehensiveness of the detection.

[0036] 5. Good adaptability: This invention does not rely on specific image comparison algorithms or fixed threshold settings, and can adapt to different UI design specifications and project requirements. Attached Figure Description

[0037] Figure 1This is a schematic diagram of a method for detecting program interface and UI style anomalies based on a large model, provided in a preferred embodiment of the present invention. Detailed Implementation

[0038] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described below with reference to the accompanying drawings. In the description of this invention, it should be understood that the terms "upper," "lower," "front," "rear," "left," "right," "top," "bottom," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings, and are only for the convenience of describing the invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the invention.

[0039] like Figure 1 As shown, this invention provides a method for detecting anomalies in program interfaces and UI styles based on a large model, including the following steps:

[0040] S1. Deploy several large models specifically for image comparison.

[0041] The image comparison-specific large-scale model is a large language model specifically applied to or optimized for tasks such as image similarity judgment and difference recognition. Examples include the Vision Transformer (ViT) series based on the Transformer architecture, and ResNet, EfficientNet, and Swin Transformer based on Convolutional Neural Networks (CNNs). These models possess powerful image feature extraction capabilities, and the extracted feature vectors can be used to calculate the similarity or difference between images through metric learning and other methods. Deployment can be on local servers or data centers, or using machine learning platforms provided by public or private clouds. The specific model and deployment method can be selected by those skilled in the art based on actual needs and circumstances from existing technologies; this invention does not impose further limitations.

[0042] S2. Obtain the UI style image set and the program interface screenshot set. Based on structural similarity comparison, filter out sample pairs with similarity lower than the first preset threshold to form the first sample pair set.

[0043] The UI style atlas contains interface images provided by UI designers as design guidelines. These images are typically exported files from design tools (such as Sketch, Figma, Adobe XD, etc.) and precisely define the appearance, layout, color, size, etc., of each interface element. The program interface screenshot atlas contains screenshots of the application in specific states or on specific pages during actual operation. These screenshots can be captured manually, or more preferably, automatically captured in predetermined scenarios using automated test scripts to ensure coverage of critical user interfaces.

[0044] Structural similarity is a metric for measuring the similarity between two images. Compared to simple pixel differences (such as mean square error MSE), it better simulates the human visual system's perception of changes in image structural information. In some preferred embodiments, specific steps are provided, including:

[0045] S21. Calculate the brightness, contrast, and structural similarity between each UI sample in the UI style image set and each program interface sample in the program interface screenshot set.

[0046] S22. Calculate a comprehensive structural similarity score based on the brightness, contrast, and structural similarity. Preferably, the three components can be weighted and summed to calculate the comprehensive structural similarity score.

[0047] The first preset threshold is an empirical value that can be set by those skilled in the art according to actual needs. For example, it can be set to 0.99, 0.98, or adjusted according to the actual application scenario. The basis for setting it is that sample pairs above the threshold are considered to be highly similar in structure, and are likely to have no UI style anomalies, or the differences are very subtle and negligible; sample pairs below the threshold are considered to have significant visual differences.

[0048] This step is essentially a coarse screening, effectively eliminating a large number of visually highly consistent image pairs and concentrating computational resources on samples that are more likely to be problematic. Thus, only the first set of sample pairs initially judged as "suspicious" is passed to subsequent steps, where a more powerful but also more time-consuming image comparison model performs further, deeper analysis and judgment. This significantly improves the efficiency of the entire anomaly detection process.

[0049] S3. Using multiple large models, image comparisons are performed on the sample pairs in the first sample pair set. Sample pairs whose number of inconsistencies exceeds half the number of large models are formed into a second sample pair set.

[0050] Specifically, each large model receives a pair of images (a UI style image and a corresponding screenshot of the program interface) as input. Based on its complex internal neural network structure and knowledge learned from large-scale data, each model independently performs deep feature extraction and comparative analysis on the input image pair. Each model ultimately outputs a judgment result on whether the pair of images is consistent. This result is typically a binary label (e.g., "consistent" / "inconsistent"), or a confidence score or difference measure that can be converted into a binary label (e.g., by comparison with a model-specific internal threshold). After all the large models have completed the comparison of the same sample pair, the system collects the judgment results from each model. Then, the following decision steps are taken:

[0051] S31. Count the number of large models that determine the sample pair as "inconsistent" (or "discrepancies exist", "mismatch", or similar conclusions).

[0052] S32. Compare this number to half the total number of large models deployed. Specifically, determine whether the number of "inconsistent" conclusions exceeds half the total number of large models.

[0053] S33. Select the sample pairs that meet the majority voting criteria (i.e., are judged as "inconsistent" by more than half of the large models) and form a second sample pair set.

[0054] The technical considerations behind this invention, which involve setting up multiple large models for majority voting, are as follows:

[0055] 1. A single model may perform poorly or produce false positives on certain types of images or differences. By combining the judgment results of multiple models, the advantages of different models can be effectively utilized to complement each other, significantly reducing false positives (misclassifying consistent data as inconsistent data) and false negatives (misclassifying inconsistent data as consistent data) caused by the bias of a single model.

[0056] 2. The majority voting strategy makes the detection results less sensitive to model selection, minor differences in training data, or certain image noise, resulting in more stable and reliable overall performance.

[0057] 3. Following the rapid screening in step S2 and the in-depth confirmation in step S3, the second sample set contains image pairs that are determined with high confidence to have UI style differences. This allows the subsequent step S4 to focus resources on accurately locating these truly problematic areas, avoiding unnecessary pixel-level analysis of a large number of samples that are problem-free or have only minor differences.

[0058] S4. Locate the erroneous pixels of the sample pairs in the second sample pair set by pixel-level comparison, draw the error box and report it.

[0059] It should be understood that pixel-level comparison refers to the process of directly comparing the attribute values ​​(mainly color values) of each corresponding pixel between two precisely aligned (i.e., spatially corresponding) images, which are usually of the same size. Most existing technologies employ threshold-based differential image analysis, which calculates the absolute difference between corresponding pixels in two images. Pixels with a difference greater than a threshold T are marked as "difference points" (e.g., white), while pixels with a difference less than or equal to T are marked as "no difference points" (e.g., black), thus generating a binary difference mask image.

[0060] The limitations of this method are: it is extremely sensitive to noise and minute changes, easily generating a large number of false positives. For example, it is highly sensitive to minor jitter in image rendering, differences in anti-aliasing, compression artifacts, and even slight changes in lighting. This can lead to the method marking numerous differences and generating many distracting false positives even when the UI has no substantial functional or visual errors. Furthermore, the method lacks intelligent processing of text content, failing to distinguish between style errors in the UI elements themselves and normal changes in dynamic text content (such as usernames, times, counters, etc.), resulting in a large number of text differences unrelated to style being falsely reported.

[0061] Therefore, in some preferred embodiments, the present invention provides a preferred method for pixel-level comparison and localization of erroneous pixels, specifically including:

[0062] S42. Convert the UI style image and program interface screenshot in the second sample set into RGB three-channel images. This step ensures that the UI style image and program interface screenshot are compared in the same color space.

[0063] S43. Perform histogram comparison on the R, G, and B channels of the UI style image and the program interface screenshot respectively, calculate the difference in the number of pixel values ​​for each channel, and record the coordinates of pixels with different pixel values. It should be noted that the histogram comparison refers to dividing the image into a grid, calculating the color histogram for each channel within each grid, and comparing the similarity of the histograms between corresponding grids (e.g., using chi-square distance, Bach distance, etc.). Grids with significant differences indicate that there is an abnormal color distribution in that area. For each pixel in a grid with abnormal distribution, calculate the difference in pixel values ​​for each channel. When calculating the difference, a small tolerance threshold can be set, for example, the absolute value of the difference is greater than 5, to ignore extremely small color fluctuations that are imperceptible to the human eye. Preferably, if a difference in pixel value is detected in at least one color channel (exceeding the tolerance), the coordinates (x, y) of that pixel are recorded as an erroneous pixel.

[0064] S44. Draw error boxes in the screenshot of the program interface based on the pixel coordinates that differ, and report them.

[0065] Those skilled in the art will understand that after obtaining the coordinates of the differing pixels, it is necessary to organize these discrete points into meaningful error regions and visualize them. Specifically, algorithms such as connected component analysis in image processing can be used to group spatially adjacent error pixels into one or more consecutive error regions. For each identified error region (or directly based on the smallest bounding rectangle of all error pixels), a bounding box that can surround the region is calculated and drawn on the screenshot of the program interface, usually using a striking color (such as red). This error box visually indicates the specific location and extent of the UI style inconsistency.

[0066] Finally, a screenshot of the program interface including the error boxes, along with any additional information (such as error type, error area coordinates, corresponding UI style image, and detected difference metrics), is reported. The reporting can take the form of a graphically illustrated detection report, or the results can be integrated into an automated testing platform or continuous integration / continuous deployment process; this invention does not impose further limitations on this.

[0067] Before performing pixel-level comparison, to avoid misreporting normal text content differences (such as the difference between the example icon name and the final version name) as UI style errors, it is necessary to first identify and process the text areas in the image. Specifically, step S4 also includes:

[0068] S41. Perform text region recognition on the UI style images and program interface screenshots in the second sample set, and record the coordinates of the recognized text regions. Exclude the text regions in steps S42 and S43. The text region recognition can be achieved using Optical Character Recognition (OCR) technology or a deep learning-based text detection model. Based on the text region coordinates, generate a binary mask for each image. In this mask, pixels corresponding to text regions are marked as "ignore" or "exclude," while pixels in other regions are marked as "participate in comparison."

[0069] In some preferred embodiments, to further reduce interference from text edge anti-aliasing or minor rendering differences, the identified text region coordinates can be expanded. Specifically, the recorded text region coordinates are floated up and down by several pixels (e.g., expanded by 1-3 pixels) in the vertical direction to obtain expanded text region coordinates. Preferably, a similar expansion can also be performed in the horizontal direction. This creates an "exclusion region" slightly larger than the actual text content, ensuring that subtle changes in the text itself and its edges are more thoroughly ignored during pixel comparison.

[0070] When recognizing text regions, if a text region clearly exists in the UI stylesheet but is missing text in the corresponding program interface screenshot (e.g., OCR does not recognize any text in that region, or the recognition result is blank), this usually represents a significant UI defect. In this case, even if the region is excluded in pixel comparison, it should still be considered a specific error type. Specifically, an error box should be drawn around the region in the program interface screenshot that should have text but is actually missing, possibly with a specific error label (such as "Text Missing").

[0071] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of this invention is defined by the appended claims and their equivalents.

Claims

1. A method for detecting anomalies in program interfaces and UI styles based on large models, characterized in that, Including the following steps: S1. Deploy several large models specifically for image comparison; S2. Obtain the UI style image set and the program interface screenshot set. Based on structural similarity comparison, filter out sample pairs with similarity lower than the first preset threshold to form the first sample pair set. S3. Using multiple large models, image comparison is performed on the sample pairs in the first sample pair set respectively. The sample pairs whose number of inconsistencies in the comparison results exceeds half the number of large models are formed into a second sample pair set. S4. Locate the erroneous pixels of the sample pairs in the second sample pair set by pixel-level comparison, draw the error box and report it; Step S4 includes: S41. Perform text region recognition on the UI style image and program interface screenshot in the second sample set, and record the coordinates of the recognized text region. Exclude the text region in steps S42 and S43. S42. Convert the UI style image and program interface screenshot in the second sample set into RGB three-channel images; S43. Perform histogram comparison on the R, G, and B channels of the UI style image and the program interface screenshot respectively, calculate the difference in the number of pixel values ​​in each channel, and record the coordinates of the pixels with different numbers of pixel values. S44. Draw error boxes in the screenshot of the program interface based on the pixel coordinates that differ, and report them.

2. The method for detecting program interface and UI style anomalies based on a large model as described in claim 1, characterized in that, Step S2 also includes a preprocessing step: ensuring that the images in the UI style image set and the program interface screenshot set have the same pixel size and pixel resolution.

3. The method for detecting program interface and UI style anomalies based on a large model as described in claim 1, characterized in that, The method for structural similarity comparison includes: S21. Calculate the brightness, contrast, and structural similarity between each UI sample in the UI style image set and each program interface sample in the program interface screenshot set; S22. Calculate a comprehensive structural similarity score based on the brightness, contrast and structural similarity.

4. The method for detecting program interface and UI style anomalies based on a large model as described in claim 1, characterized in that: If the text area in the program interface screenshot is missing text, an error box will be drawn in that text area.

5. The method for detecting program interface and UI style anomalies based on a large model as described in claim 1, characterized in that, Also includes: The recorded text region coordinates are floated up and down by several pixels in the vertical direction to obtain the expanded text region coordinates.

Images