Interface anomaly detection method based on multimodal large model comparison and double verification

Abstract

The present invention relates to the field of large model technology, and discloses an interface anomaly detection method based on multimodal large model comparison and double verification, including: deploying several large models dedicated to image comparison; obtaining a UI style atlas and a program interface screenshot set, screening out sample pairs whose similarity is lower than a first preset threshold to form a first sample pair set; using multiple large models to perform image comparison on the sample pairs in the first sample pair set, and forming a second sample pair set with the number of sample pairs whose comparison results are inconsistent exceeding half of the number of large models; locating the error pixel points of the sample pairs in the second sample pair set by pixel-level comparison, and drawing a first error box; locating the error pixel points of the sample pairs in the second sample pair set by comparing the style information of each level of each UI style, and drawing a second error box; fusing the first error box and the second error box to obtain the error-labeled picture and report it. The present invention effectively solves the pain points of the existing technology in UI consistency automation testing.

Description

Technical Field

[0001] The present invention relates to the field of large model technology, and in particular to an interface anomaly detection method based on multimodal large model comparison and double verification. Background Art

[0002] The user interface (UI) is the gateway through which users interact with software applications. Its design quality and consistency directly impact the user experience, brand image, and even the product's market competitiveness. During software development and iteration, ensuring that the actual program interface strictly adheres to pre-defined UI style specifications (usually provided by UI designers in the form of UI style sheets or design drafts) is a crucial task.

[0003] Traditional UI style consistency testing methods rely primarily on manual visual comparison. Testers or designers need to compare the actual running program interface with the design draft, screen by screen, element by element, to identify discrepancies in color, font, size, spacing, alignment, and other aspects. This method is not only time-consuming and labor-intensive, but also prone to missed detections or misjudgments due to human negligence, visual fatigue, or subjective judgment. Especially in an agile development environment with a large number of screens and frequent iterations, its efficiency and reliability are insufficient to meet requirements. For example, a slight deviation in color code or a displacement of a few pixels can easily be overlooked during lengthy manual inspections.

[0004] Therefore, this field urgently needs a technical solution that can overcome the shortcomings of existing technologies and automatically detect UI style anomalies more intelligently and accurately. Summary of the Invention

[0005] In order to solve the problems existing in the above-mentioned prior art, the present invention provides the following technical solutions, which specifically include:

[0006] The interface anomaly detection method based on multimodal large model comparison and double verification includes the following steps:

[0007] S1. Deploy several large models dedicated to image comparison;

[0008] S2. Obtain a UI style atlas and a program interface screenshot set, and based on structural similarity comparison, select sample pairs whose similarity is lower than a first preset threshold to form a first sample pair set;

[0009] S3. Use the large model to perform image comparison on the sample pairs in the first sample set, and the sample pairs whose comparison results show that the number of inconsistencies exceeds half of the number of the large model are used to form the second sample set;

[0010] S41. Locate the error pixel points of the sample pairs in the second sample pair set by pixel-level comparison, and draw a first error box;

[0011] S42. Locate the error pixel points of the sample pairs in the second sample pair set by comparing the style information of each level of each UI style, and draw a second error box;

[0012] S5. Merge the first error box and the second error box to obtain the error-labeled image and report it.

[0013] Preferably, step S2 further includes a preprocessing step: ensuring that the images in the UI style atlas and the program interface screenshot set have the same pixel size and pixel resolution.

[0014] Preferably, the method for comparing structural similarity comprises:

[0015] S21. Calculate the brightness, contrast and structural similarity between each UI sample in the UI style atlas and each program interface sample in the program interface screenshot set;

[0016] S22. Calculate a comprehensive structural similarity score based on the brightness, contrast, and structural similarity.

[0017] Preferably, step S41 includes:

[0018] S412. Converting the UI style diagram and program interface screenshot in the second sample set into an RGB three-channel image;

[0019] S413. Perform a histogram comparison on the R, G, and B channels of the UI style image and the program interface screenshot, calculate the difference in the number of pixel values ​​of each channel, and record the coordinates of the first pixel point where the number of pixel values ​​differs;

[0020] S414. Draw a first error box in the program interface screenshot according to the coordinates of the first pixel point.

[0021] Preferably, step S41 further includes:

[0022] S411. Perform text area recognition on the UI style diagrams and program interface screenshots in the second sample pair set, record the recognized text area coordinates, and exclude the text area in steps S412 and S413.

[0023] Preferably, if the text area in the program interface screenshot is missing text, an error box is drawn in the text area.

[0024] Preferably, the method further comprises: floating the recorded text area coordinates upward and downward by a number of pixels in the vertical direction to obtain extended text area coordinates.

[0025] Preferably, step S42 includes:

[0026] S421. Obtain coordinate information and color information of each level of the UI sample in the second sample pair set;

[0027] S422. Locate the target pixels corresponding to each level in the program interface screenshots in the second sample set according to the coordinate information;

[0028] S423. Compare the RGB value of the target pixel in the program interface screenshot with the corresponding color information, and record the coordinates of the second pixel point where there is a difference;

[0029] S424. Draw a second error box in the program interface screenshot according to the second pixel coordinates.

[0030] Preferably, the fusion method in step S5 includes:

[0031] Obtain a union area of ​​the first error box and the second error box in the program interface screenshot and mark it as an error area.

[0032] Beneficial effects

[0033] 1. Significantly improve detection accuracy and recall: By integrating multiple large models dedicated to image comparison and leveraging their powerful image understanding and subtle difference recognition capabilities, this approach can more accurately identify complex or hidden UI style anomalies that are difficult to detect with traditional methods. Combined with structural similarity pre-screening, it can quickly eliminate clearly identical samples, allowing the large model to focus on potentially problematic samples, further improving overall judgment accuracy.

[0034] 2. Enhanced robustness against dynamic content and reduced false positives: By identifying and excluding text areas before pixel-level comparison, and optionally making special judgments on missing text areas, false positives caused by changes in dynamic text content are effectively avoided, allowing detection to focus more on real UI style defects.

[0035] 3. More accurate error location: A dual-path error location mechanism using pixel-level comparison and UI-level style information comparison, ultimately fusing the two error boxes, can cross-validate errors from different dimensions, thereby more accurately marking the specific location and scope of the anomaly, making it easier for developers to quickly locate and fix the problem.

[0036] 4. Improved automation and efficiency of testing: The entire testing process is highly automated, from sample acquisition, preprocessing, multi-model comparison to error location and reporting, which greatly reduces manual intervention and significantly improves the efficiency of UI testing. It is especially suitable for large-scale, high-frequency regression testing scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1This is a flow chart of an interface anomaly detection method based on multimodal large model comparison and double verification provided in a preferred embodiment of the present invention. DETAILED DESCRIPTION

[0038] Embodiments of the present invention provide an interface anomaly detection method based on multimodal large-scale model comparison and dual validation. This method aims to address existing issues such as low automation, limited detection dimensions, and insufficient accuracy in identifying complex or subtle style differences in user interface (UI) style consistency detection. In current software development and iteration processes, ensuring that the actual application interface strictly adheres to design specifications (usually embodied in UI style atlases) is crucial for improving user experience and maintaining brand image. However, manual UI style comparison is time-consuming, labor-intensive, and prone to errors. Some automated tools may rely solely on pixel-level comparison, making it difficult to distinguish dynamic content changes from actual style errors or accurately understanding the hierarchical structure and design intent of UI elements. For example, a button color that changes from the "brand blue" in the design draft to the "light blue" in the actual interface, or an icon's position that shifts slightly, can be overlooked or misidentified by traditional methods. This invention significantly improves the accuracy and robustness of UI style anomaly detection by incorporating the image understanding capabilities of a multimodal large-scale model, combined with structural similarity pre-screening and a dual-path error pixel localization mechanism.

[0039] This invention belongs to the technical fields of computer vision, software testing, and artificial intelligence, and is particularly suitable for scenarios requiring large-scale, high-precision UI automated testing, such as UI consistency verification during mobile application development, web design, and desktop software updates. Typical target application scenarios include, but are not limited to: integration into continuous integration / continuous deployment (CI / CD) processes to automatically perform UI style regression testing for each code commit or version release; use as a plug-in for design tools to assist designers in verifying that implemented interfaces conform to design drafts; and serving as the core detection engine for independent UI quality monitoring platforms.

[0040] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, a UI style atlas refers to a set of images or structured data that contains user interface design specifications, representing the expected and correct UI expression. A program interface screenshot set refers to a collection of images obtained by taking screenshots of the actual running application interface, representing the actual expression of the UI. A large model specifically refers to a multimodal artificial intelligence model with powerful image understanding and comparison capabilities, such as a visual language model based on the Transformer architecture.

[0041] like Figure 1As shown, an interface anomaly detection method based on multimodal large model comparison and double verification provided by an embodiment of the present invention will be described in detail below.

[0042] S1. Deploy several dedicated large models for image comparison in the system environment.

[0043] At the initial stage of the present invention, one or more large multimodal models (MMLMs) specifically designed for image comparison or visual difference detection need to be pre-deployed. These large multimodal models refer to artificial intelligence models capable of processing and understanding information from multiple sources (e.g., images and text). In this invention, they specifically refer to models that have been trained and possess advanced image content understanding and comparison capabilities, such as visual models based on the Transformer architecture (e.g., VisionTransformer, ViT), multimodal fusion models, or customized models fine-tuned for specific UI visual styles and elements. Several large models, such as three or five, are deployed, and these models can differ in architecture, training data, or fine-tuning strategies. The purpose is to introduce diversity in the subsequent comparison steps and improve the robustness and accuracy of the judgment results through mechanisms such as ensemble learning or majority voting, thereby reducing the bias or misjudgment that may exist in a single model. These models can be deployed on cloud servers or local high-performance computing devices and provide API interfaces for subsequent calls.

[0044] S2. Obtain a UI style atlas and a program interface screenshot set, and based on structural similarity comparison, screen out sample pairs whose similarity is lower than a first preset threshold to form a first sample pair set.

[0045] To improve the efficiency and accuracy of subsequent comparisons, a preferred preprocessing step is to ensure that the images in the UI style atlas and the application interface screenshot collection have the same pixel size and pixel resolution. This can be achieved through standardization operations such as image scaling, cropping, or padding to eliminate interference introduced by inconsistent size or resolution.

[0046] This step first requires the preparation of two types of image data:

[0047] A UI style atlas is a collection of standard, correct user interface design images. These images typically come from UI designer drafts (e.g., images exported from tools like Sketch and Figma), screenshots of verified "golden versions" of apps, or a standard library of key UI elements. They serve as a benchmark for subsequent style comparisons.

[0048] Program interface screenshots: A collection of user interface images captured from the actual running application under test. These screenshots reflect the actual UI performance of the current application.

[0049] Structural similarity is an indicator that measures the similarity between two images. It generally reflects the human eye's perception of image structural distortion better than simple pixel-level mean square error (MSE). In some embodiments of the present invention, the method for calculating structural similarity can be more detailed:

[0050] S21. Calculate the brightness similarity, contrast similarity, and structural information similarity between each UI sample in the UI style atlas and each program interface sample in the program interface screenshot collection. S22. Based on these three components, calculate a comprehensive structural similarity score. This score typically ranges from 0 to 1 (or -1 to 1, depending on the specific algorithm implementation), with higher scores indicating greater structural similarity between the two images. This score can assess macroscopic visual differences between images from multiple dimensions.

[0051] The first preset threshold is a configurable parameter used to balance the stringency of the screening; a lower threshold means that only sample pairs with very significant differences will be initially judged as potentially abnormal. The purpose of this step is to quickly eliminate interfaces that are highly consistent with the design in terms of overall structure, reducing the computational burden of subsequent large-scale model analysis.

[0052] After calculating the comprehensive structural similarity score for each sample pair, it is compared with a pre-set first preset threshold (for example, 0.95, 0.9, or other values ​​adjusted based on experience). If the structural similarity score of a sample pair is lower than the first preset threshold, it is considered that there may be significant macro-structural or visual differences between this pair of UI style diagrams and the program interface screenshots. Such sample pairs are screened out to form the first sample pair set. The purpose of this step is to serve as a quick preliminary filter to exclude those interfaces that are too different from the design draft and are almost certainly abnormal, or to exclude those interfaces that are highly similar and are likely to have no style issues (depending on the threshold setting logic, described here as screening below the threshold, i.e., with large differences), thereby reducing the number of samples that need to be analyzed using the more computationally expensive MMLM.

[0053] This step is essentially a coarse screening process, effectively eliminating a large number of highly visually consistent image pairs and focusing computing resources on samples more likely to contain problems. This way, only the first set of sample pairs initially identified as possible are passed to subsequent steps, where they are subjected to further, more in-depth analysis and judgment by the more powerful, but also more time-consuming, large-scale image comparison model. This significantly improves the efficiency of the entire anomaly detection process.

[0054] S3. Use the large model to perform image comparison on the sample pairs in the first sample pair set, and form a second sample pair set with the sample pairs whose comparison results show that the number of inconsistencies exceeds half of the number of the large model.

[0055] For the first set of sample pairs obtained after screening in step S2, this step utilizes the multiple large image comparison models deployed in S1 for a deeper and more intelligent image comparison. For each sample pair in the first set (consisting of a UI style image and a program interface screenshot), each deployed large model is used for comparative analysis.

[0056] Each large model will output a comparison result, which can be a binary classification (for example, "consistent" / "inconsistent", "similar" / "dissimilar"), or a continuous similarity score or difference score. The present invention preferably adopts a majority voting mechanism: counting the number of large models whose comparison results indicate "inconsistent" (or the similarity is lower than a certain internal threshold) for the same sample pair. If the number of large models judged to be "inconsistent" exceeds half of the total number of deployed large models (for example, if 3 models are deployed, at least 2 models are required to be judged as inconsistent; if 5 models are deployed, at least 3 models are required to be judged as inconsistent), then this sample pair is finally confirmed to have a UI style anomaly and is included in the second sample pair set.

[0057] The benefits of using multiple models and majority voting are:

[0058] 1. Improve robustness: Specific biases that a single model may have or problems with poor performance on certain types of images can be corrected by the correct judgment of other models.

[0059] 2. Improved accuracy: Combining the opinions of multiple models can usually produce more reliable conclusions than a single model.

[0060] 3. Enhanced generalization ability: Different models may be good at capturing different types of visual differences, and their combination can cover a wider range of abnormal situations.

[0061] Through this step, the sample set can be further refined to screen out those sample pairs that have been jointly authenticated by multiple intelligent models and do have visual inconsistencies, laying the foundation for subsequent precise error positioning.

[0062] After determining the second set of sample pairs (i.e., sample pairs with confirmed pattern anomalies), it is necessary to accurately locate the specific location where the anomaly occurred. The present invention adopts a dual-path error pixel location strategy and ultimately fuses the results of the two paths.

[0063] The first path ( S41 ) locates the error pixel based on pixel-level comparison and draws the first error box.

[0064] It should be understood that pixel-level comparison refers to the process of directly comparing the attribute values ​​(primarily color) of each corresponding pixel between two precisely aligned (i.e., spatially aligned) images, typically of the same size. Existing techniques often employ threshold-based differential image analysis, which calculates the absolute difference between corresponding pixels in the two images. All pixels with a difference greater than a threshold T are marked as "difference points" (e.g., set to white), while pixels with a difference less than or equal to T are marked as "no difference points" (e.g., set to black), thereby generating a binary difference mask image.

[0065] The shortcomings of this method are that it is extremely sensitive to noise and small changes, and is prone to a large number of false positives. For example, it is very sensitive to small jitters in image rendering, anti-aliasing differences, compression artifacts, and even slight changes in lighting. This means that even if there are no substantial functional or visual errors in the UI, the method may mark a large number of differences, resulting in many interfering false positives. In addition, the method lacks intelligent processing of text content and cannot distinguish whether it is a style error of the UI element itself or simply because of normal changes in dynamic text content (such as user names, time, counters, etc.), resulting in a large number of text differences unrelated to style being falsely reported.

[0066] Therefore, in some preferred embodiments, the present invention provides a preferred method for locating erroneous pixels by pixel-level comparison, which specifically includes:

[0067] S412. Convert the UI style images and program interface screenshots in the second sample pair set into RGB three-channel images. If the original images are not in RGB format (e.g., RGBA or grayscale), convert them to RGB format to facilitate subsequent color channel analysis.

[0068] S413. Perform a histogram comparison on the R, G, and B channels of the UI style image and the program interface screenshot. Specifically, the difference in the number of pixel values ​​in each channel can be calculated. If a pixel has a significant difference in the number of pixel values ​​in the corresponding channel between the UI style image and the program interface screenshot (or the pixel values ​​themselves are different), the coordinates of that pixel are recorded as the first pixel coordinates. It should be noted that the histogram comparison involves dividing the image into grids, calculating the color histogram of each channel within each grid, and comparing the similarity of the histograms between corresponding grids (e.g., using chi-squared distance, Bhattacharyya distance, etc.). Grids with significant differences indicate that the color distribution in that area is abnormal. For each pixel in the grid with the abnormal distribution, the difference in pixel values ​​in each channel is calculated. When calculating the difference, a small tolerance threshold can be set, such as a difference with an absolute value greater than 5, to ignore extremely small color fluctuations that are imperceptible to the human eye. Preferably, if a difference in pixel value (outside the tolerance) is detected in at least one color channel, the coordinates (x, y) of that pixel are recorded as an erroneous pixel.

[0069] S414. In the program interface screenshot, draw a first error box based on the coordinates of all recorded first pixels. This typically involves finding the area formed by all the different pixels and marking them with one or more minimum bounding boxes that enclose these different pixels. If the different pixels are widely distributed, multiple separate first error boxes may be formed.

[0070] Those skilled in the art will know that after obtaining the coordinates of the pixel points with differences, these discrete points need to be organized into meaningful error areas and visualized. Specifically, algorithms such as connected component analysis in image processing can be used to combine spatially adjacent error pixels into one or more continuous error areas. For each identified error area (or directly based on the minimum enclosing rectangle of all error pixels), a bounding box that can enclose the area is calculated and drawn on the program interface screenshot, usually using a striking color (such as red). This error box intuitively marks the specific location and range of the UI style inconsistency.

[0071] Before performing pixel-level comparison, in order to avoid misreporting normal text content differences (such as the difference between the sample icon name and the final version name) as UI style errors, it is necessary to first identify and process the text areas in the image. Specifically, step S4 also includes:

[0072] S411. In order to avoid misjudging changes in dynamic text content as style errors, text area identification can be performed on the UI style diagram and program interface screenshots in the second sample set, and the coordinates of the identified text areas can be recorded. In subsequent pixel comparisons, these identified text areas will be excluded. Optionally, in order to more fully cover the pixel changes in the text and its surroundings that may be caused by rendering differences, the recorded text area coordinates can be floated vertically (or even horizontally) up and down (left and right) by several pixels, for example 2-5 pixels, to obtain an extended text area coordinate, and this extended area is excluded in subsequent comparisons. In addition, a special case is that if an area in the program interface screenshot where text is expected to be present (judged based on the UI style diagram or text area identification results) lacks text content, this itself may also be a UI defect. In this case, an error box can be directly drawn in the text area to mark it as a specific error type.

[0073] The second path locates the erroneous pixel points based on the comparison of style information at each level of each UI style, and then draws a second error box. This step usually requires the UI style diagram to provide structured hierarchical information, such as layer data exported from design software, which contains the coordinate information (such as position and size) and color information (such as background color, border color, text color, etc.) of each UI element (level). Specific implementation methods may include:

[0074] S421. Obtain the coordinate information and color information for each layer of the UI sample (i.e., the UI style image) in the second sample pair set. This means that the UI style image must be associated with its structured design specification data. For example, for a UI style image named "login_screen_style.png," the system must be able to find its corresponding "login_screen_style.json" (or similar format), which contains the expected style attributes for all elements on the screen.

[0075] S422. Based on the coordinate information of a layer in the UI style diagram, locate the target pixel area that the layer should occupy in the corresponding program interface screenshot. For example, if the design specification indicates that a button is expected to be at the position (100, 200, 80, 30), then check this rectangular area in the program interface screenshot.

[0076] S423. Compare the RGB values ​​of the actual pixels in the target pixel area with the expected color information for that level in the UI style diagram. If the colors of some pixels in the target area are different from the expected colors (for example, the background color of a button is different from the design draft), record the coordinates of these different pixels as the second pixel coordinates.

[0077] S424. Draw a second error box based on all recorded second pixel coordinates in the program interface screenshot. Again, this is typically the minimum bounding rectangle that encloses the logically different areas. For example, if the background color of a button doesn't match the design, an error box is drawn in the button area; if an icon is offset, an error may be marked at both its intended location and its actual location, or only at its actual location.

[0078] The advantage of this path is that it can perform comparison based on design intent, and can detect logical-level style anomalies such as incorrect element color, position offset, size mismatch, font error, missing elements (if the corresponding element is not found in the expected location), etc. These anomalies may be difficult to detect through pixel-level comparison alone, or the pixel-level comparison results are not accurate enough.

[0079] S5. Merge the first error box and the second error box to obtain an error-labeled image and report it. After obtaining the first error box (from pixel-level comparison) and the second error box (from hierarchical style information comparison), the final step is to fuse the two error boxes to obtain a final error-labeled image and report it. There are various methods for fusion. In a preferred embodiment of the present invention, the fusion method includes: obtaining the union area of ​​the first error box and the second error box in the program interface screenshot. This union area is considered the error area with the highest confidence because it is simultaneously confirmed by detection methods from two different dimensions. This union area is highlighted on the program interface screenshot (for example, by framing it with a bright color) to form an error-labeled image. Subsequently, this error-labeled image, along with relevant error information (such as the sample pair to which it belongs, the error box coordinates, possible error type, etc.), can be reported to the test management system, defect tracking system, or directly notified to the relevant developers.

[0080] Those skilled in the art will appreciate that the technical solutions described in the embodiments of the present invention can be modified and combined according to actual needs. For example, the first preset threshold, the number and type of large models, the number of pixels used to expand the text area, and the error box fusion strategy (e.g., in addition to intersection, union or weighted fusion can also be considered) can all be adjusted according to the application scenario and performance requirements. These modifications and variations, as long as they do not depart from the spirit and scope of the present invention, should be considered included within the scope of protection of the present invention.

[0081] The basic principles, main features, and advantages of the present invention are shown and described above. Those skilled in the art should understand that the present invention is not limited to the foregoing embodiments. The foregoing embodiments and descriptions are merely illustrative of the principles of the present invention. Various changes and modifications may be made to the present invention without departing from the spirit and scope of the present invention. Such changes and modifications are intended to fall within the scope of the present invention. The scope of protection claimed in the present invention is defined by the appended claims and their equivalents.

Claims

1. The interface anomaly detection method based on multimodal large model comparison and double verification is characterized by: Including steps: S1. Deploy several large models dedicated to image comparison; S2. Obtain a UI style atlas and a program interface screenshot set, and based on structural similarity comparison, select sample pairs whose similarity is lower than a first preset threshold to form a first sample pair set; S3. Use the large model to perform image comparison on the sample pairs in the first sample set, and select the sample pairs whose number of inconsistencies in the comparison results exceeds half of the number of the large model to form the second sample set; S41. Locate the error pixel points of the sample pairs in the second sample pair set by pixel-level comparison, and draw a first error box; S42. Locate the error pixel points of the sample pairs in the second sample pair set by comparing the style information of each level of each UI style, and draw a second error box; S5. Merge the first error box and the second error box to obtain the error-labeled image and report it.

2. The interface anomaly detection method based on multimodal large model comparison and double verification according to claim 1 is characterized in that: Step S2 also includes a pre-processing step: ensuring that the images in the UI style atlas and the program interface screenshot set have the same pixel size and pixel resolution.

3. The interface anomaly detection method based on multimodal large model comparison and double verification according to claim 1 is characterized in that: The method for comparing structural similarity includes: S21. Calculate the brightness, contrast and structural similarity between each UI sample in the UI style atlas and each program interface sample in the program interface screenshot set; S22. Calculate a comprehensive structural similarity score based on the brightness, contrast, and structural similarity.

4. The interface anomaly detection method based on multimodal large model comparison and double verification according to claim 1 is characterized in that: Step S41 includes: S412. Converting the UI style diagram and program interface screenshot in the second sample set into an RGB three-channel image; S413. Perform a histogram comparison on the R, G, and B channels of the UI style image and the program interface screenshot, calculate the difference in the number of pixel values ​​of each channel, and record the coordinates of the first pixel point where the number of pixel values ​​differs; S414. Draw a first error box in the program interface screenshot according to the coordinates of the first pixel point.

5. The interface anomaly detection method based on multimodal large model comparison and double verification according to claim 4 is characterized in that: Step S41 further includes: S411. Perform text area recognition on the UI style diagrams and program interface screenshots in the second sample pair set, record the recognized text area coordinates, and exclude the text area in steps S412 and S413.

6. The interface anomaly detection method based on multimodal large model comparison and double verification according to claim 5 is characterized in that: If text is missing in the text area in the program interface screenshot, an error box is drawn in the text area.

7. The interface anomaly detection method based on multimodal large model comparison and double verification according to claim 5 is characterized in that: Also includes: The recorded text area coordinates are floated vertically upward and downward by a number of pixels to obtain extended text area coordinates.

8. The interface anomaly detection method based on multimodal large model comparison and double verification according to claim 1 is characterized in that: Step S42 includes: S421. Obtain coordinate information and color information of each level of the UI sample in the second sample pair set; S422. Locate the target pixels corresponding to each level in the program interface screenshots in the second sample set according to the coordinate information; S423. Compare the RGB value of the target pixel in the program interface screenshot with the corresponding color information, and record the coordinates of the second pixel point where there is a difference; S424. Draw a second error box in the program interface screenshot according to the second pixel coordinates.

9. The interface anomaly detection method based on multimodal large model comparison and double verification according to claim 1 is characterized in that: The fusion method in step S5 includes: Obtain a union area of ​​the first error box and the second error box in the program interface screenshot and mark it as an error area.

Images