UI consistency comparison method and system based on hybrid model and terminal

Abstract

The invention discloses a UI consistency comparison method and system based on a hybrid model and a terminal, and the method comprises the steps: obtaining a historical data set, training an initial model through the historical data set, and obtaining the hybrid model; inputting the UI design effect picture and the web actual development page picture into a hybrid model in batches, and carrying out image pairing by the hybrid model through calculating a structural similarity index; segmenting the UI design effect picture and the web actual development page picture paired with the UI design effect picture according to different levels to obtain a plurality of modules, a plurality of containers and a plurality of components; comparing the UI design effect picture with the paired web actual development page picture layer by layer according to the module level, the container level and the component level, and outputting a comparison result. According to the method, the consistency of the UI design effect picture and the web actual development page picture can be accurately, efficiently and automatically checked and compared, the delivery quality of products is improved, and the development cost is reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of user interface testing, and in particular to a UI consistency comparison method, system, terminal and computer-readable storage medium based on a hybrid model. Background Art

[0002] UI (User Interface) is the medium for interaction and information exchange between the system and the user. It realizes the conversion between the internal form of information and the form acceptable to humans. In the front-end development of web (World Wide Web) applications, the page layout style depends on the designer's renderings. In the actual development process, there will be a large style difference between the web page and the rendering UI, which seriously affects the quality of product delivery.

[0003] In the process of outputting the user interface, it is necessary to ensure the consistency between the UI actually displayed on the web and the UI of the design renderings, so the consistency between the two needs to be tested. The existing verification of whether the actual web development UI is consistent with the design renderings UI basically relies on designers and testers to perform manual comparison and correction, including raising questions, modifying questions, and regressing questions. This is a complete closed-loop process, which undoubtedly increases labor costs and time costs, and may even result in missed comparisons, resulting in product delivery defects.

[0004] Therefore, the prior art still needs to be improved and developed. Summary of the invention

[0005] The main purpose of the present invention is to provide a UI consistency comparison method, system, terminal and computer-readable storage medium based on a hybrid model, aiming to solve the problem that the existing verification of the consistency between UI design renderings and actual web development page images requires manual comparison, resulting in high development costs and low verification accuracy and efficiency.

[0006] To achieve the above-mentioned object of the invention, the present invention provides a UI consistency comparison method based on a hybrid model, and the UI consistency comparison method based on a hybrid model includes:

[0007] Acquire a historical data set, and use the historical data set to train an initial model to obtain a hybrid model;

[0008] Batch input of UI design renderings and web actual development page images into the hybrid model, and the hybrid model performs image pairing by calculating a structural similarity index;

[0009] Segment the UI design rendering and the web actual development page image paired with the UI design rendering according to different levels to obtain multiple modules, multiple containers and multiple components;

[0010] The UI design renderings and the paired web actual development page images are compared layer by layer according to the module level, container level and component level, and the comparison results are output.

[0011] Optionally, the hybrid model includes a segmentation model and a feature extraction model;

[0012] The acquiring of the historical data set and the use of the historical data set to train the initial model to obtain the hybrid model specifically include:

[0013] Obtain a historical data set, and divide the historical data set into a training set and a test set;

[0014] Preprocessing the historical data set, wherein the preprocessing includes mask generation and label file generation;

[0015] Using the training set to train the segmentation model and the feature extraction model to obtain a trained segmentation model and a trained feature extraction model;

[0016] Using the test set to test the performance of the trained segmentation model and the trained feature extraction model;

[0017] The trained segmentation model whose performance meets the test requirements is combined with the trained feature extraction model to obtain a hybrid model.

[0018] Optionally, the step of batch-inputting UI design renderings and web actual development page images into the hybrid model, wherein the hybrid model performs image pairing by calculating a structural similarity index, specifically includes:

[0019] Batch input of UI design renderings and web actual development page images into the hybrid model;

[0020] The hybrid model traverses and reads the UI design renderings and the web actual development page images, measures the image similarity from three aspects: brightness, contrast, and structure, and calculates the structural similarity index SSIM between the UI design renderings and the web actual development page images:

[0021]

[0022] Among them, SSIM(x,y) represents the structural similarity index, x represents the UI design effect diagram, y represents the actual web development page image, μ x and μ y Represents the mean of UI design renderings and web actual development page images, σ x 2 and σ y2 Respectively represent the variance of UI design renderings and web actual development page images, σ xy Represents the covariance between UI design renderings and actual web development page images, C1 and C2 are constants;

[0023] Among the multiple structural similarity indexes corresponding to the calculated UI design renderings, the web actual development page image corresponding to the highest structural similarity index is used as the web actual development page image paired with the UI design renderings.

[0024] Optionally, the step of segmenting the UI design rendering and the web actual development page image paired with the UI design rendering according to different levels to obtain multiple modules, multiple containers and multiple components specifically includes:

[0025] Segment the UI design rendering and the web actual development page image paired with the UI design rendering according to different modules to obtain multiple modules in the UI design rendering and multiple modules in the web actual development page image paired with the UI design rendering;

[0026] Divide the interior of each module into left and right containers and upper, middle and lower containers to obtain multiple containers in the UI design rendering and multiple containers in the paired web actual development page image;

[0027] Each container is divided into different components to obtain multiple components in the UI design rendering and multiple components in the paired web actual development page image.

[0028] Optionally, the UI design renderings and the paired web actual development page images are compared layer by layer according to the module level, container level and component level, and the hybrid model outputs the comparison results, specifically including:

[0029] Compare the modules in the UI design rendering with the corresponding modules in the paired web actual development page image to see if they are consistent, wherein the modules include a header module, a content module, and a bottom module;

[0030] Compare the containers in the UI design rendering with the corresponding containers in the paired web actual development page image to see if they are consistent;

[0031] Comparing the components in the UI design rendering with the corresponding components in the paired web actual development page image to see if they are consistent, wherein the components include buttons, colors, and texts;

[0032] The hybrid model outputs a comparison result, and annotates inconsistent modules, containers or components on the web actual development page image;

[0033] The comparison results include a difference report and a visualization diagram.

[0034] Optionally, comparing whether the components in the UI design rendering are consistent with the corresponding components in the paired web actual development page image specifically includes:

[0035] A template matching algorithm is adopted, the button in the UI design rendering is used as a template, the paired web actual development page image is used as a target image, the template is slid in the target image, the structural similarity index of each region in the template and the target image is calculated, the region corresponding to the highest structural similarity index among multiple structural similarity indexes is marked as the target region, the features of the button and the target region are extracted, and the feature differences between the button and the target region are compared, and it is determined whether the button in the UI design rendering and the button in the paired web actual development page image are consistent according to the feature differences;

[0036] Using a color histogram comparison algorithm, calculate the color histograms of the UI design rendering and the paired web actual development page image, use a similarity index to calculate the color histogram similarity of the UI design rendering and the paired web actual development page image, compare the color difference of the UI design rendering and the paired web actual development page image according to the color histogram similarity, and determine whether the color in the UI design rendering and the color in the paired web actual development page image are consistent according to the color difference;

[0037] An OCR text recognition and comparison algorithm is adopted to extract the text of the UI design rendering and the paired web actual development page image, and the extracted text of the UI design rendering and the paired web actual development page image are preprocessed, feature extracted and serialized in sequence to generate a text sequence of the UI design rendering and the paired web actual development page image, calculate the text sequence similarity of the UI design rendering and the paired web actual development page image, compare the text difference between the UI design rendering and the paired web actual development page image according to the text sequence similarity, and judge whether the text in the UI design rendering and the text in the paired web actual development page image are consistent according to the text difference.

[0038] Optionally, the similarity index includes Bhattacharyya distance;

[0039] The using of a similarity index to calculate the color histogram similarity of the UI design rendering and the paired web actual development page image specifically includes:

[0040] According to the color histogram of the UI design rendering and the color histogram of the paired web actual development page image, the color histogram similarity of the UI design rendering and the paired web actual development page image is calculated using the Bhattacharyya distance DB:

[0041] DB(p,q)=-ln(BC(p,q));

[0042]

[0043] Among them, DB(p,q) represents the Bhattacharyya distance, BC(p,q) represents the Bhattacharyya coefficient, p(a) and q(a) represent the probability values ​​of the color histogram of the UI design rendering and the color histogram of the paired web actual development page image at point a, a represents the independent variable, p and q represent the probabilities of the color histogram of the UI design rendering and the color histogram of the paired web actual development page image, respectively.

[0044] To achieve the above-mentioned object of the invention, the present invention further provides a UI consistency comparison system based on a hybrid model, and the UI consistency comparison system based on a hybrid model includes:

[0045] A hybrid model building module: used to obtain a historical data set, train an initial model using the historical data set, and obtain a hybrid model;

[0046] Image pairing module: used to batch input UI design renderings and web actual development page images into the hybrid model, and the hybrid model performs image pairing by calculating the structural similarity index;

[0047] A segmentation module is used to segment the UI design rendering and the web actual development page image paired with the UI design rendering according to different levels to obtain multiple modules, multiple containers and multiple components;

[0048] Comparison module: used to compare the UI design renderings and the paired web actual development page images layer by layer according to the module level, container level and component level, and output the comparison results.

[0049] In order to achieve the above-mentioned purpose of the invention, the present invention also provides a terminal, which includes: a memory, a processor, and a hybrid model-based UI consistency comparison program stored in the memory and executable on the processor, wherein the hybrid model-based UI consistency comparison program, when executed by the processor, implements the steps of the hybrid model-based UI consistency comparison method as described above.

[0050] To achieve the above-mentioned purpose of the invention, the present invention also provides a computer-readable storage medium, which stores a UI consistency comparison program based on a hybrid model. When the UI consistency comparison program based on the hybrid model is executed by a processor, the steps of the UI consistency comparison method based on the hybrid model as described above are implemented.

[0051] In the present invention, a historical data set is obtained, and an initial model is trained using the historical data set to obtain a hybrid model; UI design renderings and web actual development page images are batch-inputted into the hybrid model, and the hybrid model performs image pairing by calculating a structural similarity index; the UI design renderings and the web actual development page images paired with the UI design renderings are segmented at different levels to obtain multiple modules, multiple containers, and multiple components; the UI design renderings and the paired web actual development page images are compared layer by layer at the module level, the container level, and the component level, and the comparison results are output. The present invention segments the UI design renderings and web actual development page images (i.e., web actual development UI) from different levels, and segments the UI into finer granularity for comparison, thereby achieving accurate and efficient automatic verification and comparison of the consistency of the UI design renderings and web actual development page images, quickly finding different UI content, and helping developers to make timely adjustments and corrections, thereby improving the delivery quality of products and reducing development costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 It is a flow chart of a preferred embodiment of the UI consistency comparison method based on the hybrid model of the present invention;

[0053] Figure 2 is another flow chart of a preferred embodiment of the UI consistency comparison method based on the hybrid model of the present invention;

[0054] Figure 3 It is a schematic diagram of UI segmentation of the present invention;

[0055] Figure 4 It is a structural diagram of a preferred embodiment of a UI consistency comparison system based on a hybrid model of the present invention;

[0056] Figure 5 It is a structural diagram of a preferred embodiment of the terminal of the present invention. DETAILED DESCRIPTION

[0057] In order to make the purpose, technical solution and advantages of the present invention clearer and more specific, the present invention is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0058] UI is the medium for interaction and information exchange between the system and the user. It realizes the conversion between the internal form of information and the form acceptable to humans. In the front-end development of web applications, the page layout style depends on the designer's renderings. In the actual development process, there will be a large difference in style (size, color, font size, etc.) between the web page and the rendering UI, which seriously affects the quality of product delivery. At this time, UI correction requires the participation of designers, and feedback to developers for correction if any differences are found with the original design.

[0059] In the process of outputting the user interface, it is necessary to ensure the consistency between the UI actually displayed on the web and the UI of the design renderings, so the consistency between the two needs to be tested. The existing verification of whether the actual web development UI is consistent with the design renderings UI basically relies on designers and testers to perform manual comparison and correction, including raising questions, modifying questions, and regressing questions. This is a complete closed-loop process, which undoubtedly increases labor costs and time costs, and may even result in missing comparisons (functional comparisons, etc.), resulting in product delivery defects.

[0060] In order to solve the above technical problems, the present invention provides a UI consistency comparison method based on a hybrid model, which obtains a historical data set, trains an initial model using the historical data set, and obtains a hybrid model; batches of UI design renderings and web actual development page images are input into the hybrid model, and the hybrid model performs image pairing by calculating a structural similarity index; the UI design renderings and the web actual development page images paired with the UI design renderings are segmented at different levels to obtain multiple modules, multiple containers, and multiple components; the UI design renderings and the paired web actual development page images are compared layer by layer according to the module level, container level, and component level, and the comparison results are output. The present invention segments the UI design renderings and web actual development page images (i.e., web actual development UI) from different levels, divides the UI into finer granularity for comparison, realizes accurate and efficient automatic verification and comparison of the consistency of the UI design renderings and web actual development page images, quickly finds different UI content, helps developers to make timely adjustments and corrections, improves product delivery quality, and reduces development costs.

[0061] The application content is further explained below through the description of embodiments in conjunction with the accompanying drawings.

[0062] A preferred embodiment of the UI consistency comparison method based on the hybrid model of the present invention is as follows: Figure 1 As shown, specifically including:

[0063] S1. Obtain a historical data set, and use the historical data set to train an initial model to obtain a hybrid model.

[0064] In one implementation of this embodiment, the hybrid model includes a segmentation model and a feature extraction model;

[0065] The acquiring of the historical data set and the use of the historical data set to train the initial model to obtain the hybrid model specifically include:

[0066] Obtain a historical data set, and divide the historical data set into a training set and a test set;

[0067] Preprocessing the historical data set, wherein the preprocessing includes mask generation and label file generation;

[0068] Using the training set to train the segmentation model and the feature extraction model to obtain a trained segmentation model and a trained feature extraction model;

[0069] Using the test set to test the performance of the trained segmentation model and the trained feature extraction model;

[0070] The trained segmentation model whose performance meets the test requirements is combined with the trained feature extraction model to obtain a hybrid model.

[0071] Specifically, the hybrid model of the present invention includes a segmentation model and a feature extraction model, and the hybrid model construction process includes: data preparation: preparing a historical data set, dividing the historical data set into a training set and a test set, and then preprocessing the historical data set, including generating mask and label files; segmentation model training, using PyTorch tools or TensorFlow / Keras tools to implement Mask R-CNN (Mask Region-Convolutional Neural Network) or U-Net (U-type neural network), using cross entropy loss and IoU (Intersection over Union) loss loss function to calculate the loss, and then using Adam (Adaptive Moment Estimation) or SGD (Stochastic Gradient Descent, stochastic gradient descent) optimizer to obtain a trained segmentation model; feature extraction model training: directly use the ImageNet dataset (image dataset) to pre-train weights, perform fine-tuning, and obtain a trained feature extraction model; model testing: use segmentation accuracy and comparison accuracy as performance indicators to test the trained segmentation model and feature extraction model, combine the trained segmentation model and feature extraction model that meet the performance indicators to obtain a hybrid model, deploy the hybrid model to the cloud or local service, and provide an API interface (Application Program Interface). The hybrid model usage process includes: inputting UI design renderings (i.e. renderings UI or UI renderings, etc.) and web actual development page images (i.e. developed web page images, web actual development UI, developed web UI or web page screenshots, etc.), using the segmentation model to segment the image, generating the boundaries (i.e. borders) of modules, containers and components, using the feature extraction model to extract features and compare them module by module, container and component, and the hybrid model outputs the comparison results, including difference reports and visualizations.

[0072] In addition, the hybrid model algorithm of the present invention includes traditional image processing algorithms and machine learning algorithms, such as Figure 2 As shown, the traditional image processing algorithms include the Structural Similarity Index (SSIM) algorithm, the template matching algorithm and the color histogram comparison algorithm, and the machine learning algorithm includes the OCR (Optical Character Recognition) text recognition comparison algorithm, which are as follows:

[0073] Structural Similarity Index (SSIM) algorithm: SSIM is an indicator to measure the similarity between two images. It takes into account three dimensions: brightness, contrast, and structure. The specific process includes: calculating the brightness comparison function of the two images to reflect the brightness difference of the images; calculating the contrast comparison function of the two images to reflect the contrast difference of the images; calculating the structure comparison function of the two images to reflect the difference in image texture structure; combining these three comparison functions to obtain the final SSIM value. The closer the SSIM value is to 1, the more similar the two images are.

[0074] Template matching algorithm: Template matching is a method for finding and identifying the position of a small image (template) in a larger image. The specific process includes: selecting a template image; sliding the template in the target image and calculating the similarity between the template and the target image at each position; commonly used similarity measurement methods include sum of squared differences, normalized cross-correlation, etc.; finding the area with the highest similarity and determining the position of the template in the target image.

[0075] Color histogram comparison algorithm: The color histogram comparison algorithm evaluates the similarity of two images by comparing their color distributions. The specific process includes: calculating the color histograms of the two images, usually using the RGB color space; normalizing the color histograms so that the comparison is not affected by the image size; using statistical methods (such as the chi-square test) to compare the similarity of the two color histograms; the higher the similarity value, the more similar the color distributions of the two images are.

[0076] OCR text recognition and comparison algorithm: OCR text recognition and comparison algorithm is used to recognize text in images and compare text content in different images. The specific process includes: using OCR technology to extract text from images; preprocessing the extracted text, such as removing noise, standardizing the format, etc.; comparing the similarity of two text strings, and judging whether the text content is the same or similar based on the similarity value.

[0077] S2. Inputting UI design renderings and web actual development page images in batches into the hybrid model, and the hybrid model performs image pairing by calculating a structural similarity index.

[0078] In an implementation of this embodiment, the UI design renderings and web actual development page images are batch-inputted into the hybrid model, and the hybrid model performs image pairing by calculating a structural similarity index, specifically including:

[0079] Batch input of UI design renderings and web actual development page images into the hybrid model;

[0080] The hybrid model traverses and reads the UI design renderings and the web actual development page images, measures the image similarity from three aspects: brightness, contrast, and structure, and calculates the structural similarity index SSIM between the UI design renderings and the web actual development page images:

[0081]

[0082] Among them, SSIM(x,y) represents the structural similarity index, x represents the UI design effect diagram, y represents the actual web development page image, μ x and μ y Represents the mean of UI design renderings and web actual development page images, σ x 2 and σ y 2 Respectively represent the variance of UI design renderings and web actual development page images, σ xy Represents the covariance between the UI design renderings and the actual web development page images. C1 and C2 are constants used to avoid the situation where the denominator is 0. The value of the constant varies according to the color range. In general, C1=(K1×L) 2 , C2=(K2×L) 2 , where L is the range of pixel values, L-1 is the maximum pixel value, K1 and K2 are constants less than 1;

[0083] Among the multiple structural similarity indexes corresponding to the calculated UI design renderings, the web actual development page image corresponding to the highest structural similarity index is used as the web actual development page image paired with the UI design renderings.

[0084] Specifically, the present invention imports a large number of UI design renderings and web actual development page images into a hybrid model. The hybrid model retrieves the web actual development page images that match the UI design renderings. The purpose of using the retrieval is to facilitate batch comparison scenarios, avoid artificial one-to-one import, and improve the efficiency of comparison. The hybrid model traverses and reads the UI design renderings and web actual development page images, and measures the similarity of the two images based on brightness, contrast, and structure. The closer the SSIM value is to 1, the more similar the two images are. After this step is completed, the design rendering UI and the front-end page UI (i.e., the web actual development page image) can be associated one-to-one.

[0085] In addition, the structural similarity index (SSIM) is an indicator that measures the similarity between two images and is widely used to evaluate image quality or compare the similarity between two images. It calculates a value between 0 and 1 by comprehensively considering the similarity of brightness, contrast and structure. The closer the value is to 1, the more similar the images are.

[0086] S3. Segment the UI design rendering and the web actual development page image paired with the UI design rendering according to different levels to obtain multiple modules, multiple containers and multiple components.

[0087] In an implementation of this embodiment, the UI design rendering and the web actual development page image paired with the UI design rendering are segmented at different levels to obtain multiple modules, multiple containers and multiple components, specifically including:

[0088] Segment the UI design rendering and the web actual development page image paired with the UI design rendering according to different modules to obtain multiple modules in the UI design rendering and multiple modules in the web actual development page image paired with the UI design rendering;

[0089] Divide the interior of each module into left and right containers and upper, middle and lower containers to obtain multiple containers in the UI design rendering and multiple containers in the paired web actual development page image;

[0090] Each container is divided into different components to obtain multiple components in the UI design rendering and multiple components in the paired web actual development page image.

[0091] Specifically, the comparison is performed from global to local, layer by layer, so it is necessary to split layer by layer first, such as Figure 3 As shown, the effect diagram UI and the web page UI are divided into three modules according to the layout: the header, the body content, and the footer bottom. Each module is divided into left and right, upper, middle and lower containers, and each container is divided again according to components such as buttons, texts, and input boxes, thereby obtaining the smallest unit of each module, and then comparing them from the outer layer to the inner layer in sequence, and finally outputting the difference between each module and the sample (effect diagram UI) in the web page UI. The present invention divides the effect diagram UI and the web page UI into modules, and divides the UI into the smallest units according to different granularities (modules, containers, buttons, icons, texts, etc.), so as to compare the buttons, texts, and colors of the subsequent modules. In summary, the present invention divides the UI into finer granularities for comparison through module division, and realizes automatic UI comparison more accurately and efficiently.

[0092] S4. Compare the UI design renderings and the paired web actual development page images layer by layer according to the module level, container level and component level, and the hybrid model outputs the comparison results.

[0093] In an implementation of this embodiment, the UI design renderings and the paired web actual development page images are compared layer by layer according to the module level, container level and component level, and the hybrid model outputs the comparison results, which specifically include:

[0094] Compare the modules in the UI design rendering with the corresponding modules in the paired web actual development page image to see if they are consistent, wherein the modules include a header module, a content module, and a bottom module;

[0095] Compare the containers in the UI design rendering with the corresponding containers in the paired web actual development page image to see if they are consistent;

[0096] Comparing the components in the UI design rendering with the corresponding components in the paired web actual development page image to see if they are consistent, wherein the components include buttons, colors, and texts;

[0097] The hybrid model outputs a comparison result, and annotates inconsistent modules, containers or components on the web actual development page image;

[0098] The comparison results include a difference report and a visualization diagram.

[0099] In an implementation of this embodiment, comparing whether the components in the UI design rendering are consistent with the corresponding components in the paired web actual development page image specifically includes:

[0100] A template matching algorithm is adopted, the button in the UI design rendering is used as a template, the paired web actual development page image is used as a target image, the template is slid in the target image, the structural similarity index of each region in the template and the target image is calculated, the region corresponding to the highest structural similarity index among multiple structural similarity indexes is marked as the target region, the features of the button and the target region are extracted, and the feature differences between the button and the target region are compared, and it is determined whether the button in the UI design rendering and the button in the paired web actual development page image are consistent according to the feature differences;

[0101] Using a color histogram comparison algorithm, calculate the color histograms of the UI design rendering and the paired web actual development page image, use a similarity index to calculate the color histogram similarity of the UI design rendering and the paired web actual development page image, compare the color difference of the UI design rendering and the paired web actual development page image according to the color histogram similarity, and determine whether the color in the UI design rendering and the color in the paired web actual development page image are consistent according to the color difference;

[0102] An OCR text recognition and comparison algorithm is adopted to extract the text of the UI design rendering and the paired web actual development page image, and the extracted text of the UI design rendering and the paired web actual development page image are preprocessed, feature extracted and serialized in sequence to generate a text sequence of the UI design rendering and the paired web actual development page image, calculate the text sequence similarity of the UI design rendering and the paired web actual development page image, compare the text difference between the UI design rendering and the paired web actual development page image according to the text sequence similarity, and judge whether the text in the UI design rendering and the text in the paired web actual development page image are consistent according to the text difference.

[0103] In one implementation of this embodiment, the similarity index includes Bhattacharyya distance;

[0104] The using of a similarity index to calculate the color histogram similarity of the UI design rendering and the paired web actual development page image specifically includes:

[0105] According to the color histogram of the UI design rendering and the color histogram of the paired web actual development page image, the color histogram similarity of the UI design rendering and the paired web actual development page image is calculated using the Bhattacharyya distance DB:

[0106] DB(p,q)=-ln(BC(p,q));

[0107]

[0108] Among them, DB(p,q) represents Bhattacharyya distance, BC(p,q) represents Bhattacharyya coefficient, p(a) and q(a) represent the probability values ​​of the color histogram of UI design renderings and the color histogram of the paired web actual development page images at a, a represents the independent variable, p and q represent the probabilities of the color histogram of UI design renderings and the color histogram of the paired web actual development page images. Bhattacharyya distance is used to measure the similarity between two probability distributions. Color contrast is calculated using discrete probability distribution. The value of Bhattacharyya distance DB(p,q) is between 0 and positive infinity. The smaller the value, the more similar the two probability distributions are.

[0109] Specifically, in button consistency detection, a template matching algorithm is used to obtain the pixel position of the button in the UI design rendering as a template, and then search for it in the corresponding web actual development page image. By extracting the features of the template, the best matching area is determined based on the similarity metric, and a rectangular box is drawn around the found area to mark it, and the difference in features (button size, color, font, inner and outer margins, etc.) between the actual effect and the template is compared. In color consistency detection, the color histogram similarity is calculated, and the UI design rendering and the web actual development page image are read separately, converted into a specific color space (such as RGB), and then the frequency of occurrence of different colors is counted to generate a color histogram. The similarity of the two color histograms is compared using a similarity metric (such as Bhattacharyya distance, chi-square distance, or correlation coefficient) to determine whether the colors are consistent. In text consistency detection, the text comparison step is part of the module comparison, such as comparing the "xx system name" in the header of the UI design rendering with the "xx system name" in the header of the web actual development page image. The deep learning method Transformer-based OCR is used to recognize text. Transformer-based OCR is a text recognition method based on the Transformer model. It has excellent long sequence modeling capabilities and strong context understanding capabilities. It is especially suitable for processing complex text scenarios (such as irregularly arranged characters, handwritten text, etc.). The main process is as follows:

[0110] 1) Text detection: Usually, the text area is located through the text detection model and then cropped for subsequent recognition;

[0111] 2) Feature extraction: Use convolutional neural network (CNN) to extract visual features from text areas;

[0112] 3) Transformer module: The extracted visual features are serialized and sent to the Transformer module, which mainly completes the following tasks:

[0113] 31) Modeling the global context of character sequences;

[0114] 32) Learn complex character interaction information (such as semantic associations between characters);

[0115] 4) Decoding and output: Use CTC (Connectionist Temporal Classification) or Seq2Seq decoder (combined with attention mechanism) to generate recognized text sequence.

[0116] Finally, after the hybrid model comparison is completed, the comparison results are output and the different UI contents are marked on the web page to facilitate subsequent developers to repair the page.

[0117] The present invention is mainly used in the development, testing or acceptance stages to compare the consistency of the web UI actually developed with the design rendering UI, quickly find the UI modules with differences, and help developers to repair and adjust.

[0118] In addition, based on the above-mentioned UI consistency comparison method based on the hybrid model, the present invention also provides a UI consistency comparison system based on the hybrid model, wherein a preferred embodiment of the UI consistency comparison system based on the hybrid model is as follows: Figure 4 As shown, specifically including:

[0119] Hybrid model construction module 01: used to obtain a historical data set, train an initial model using the historical data set, and obtain a hybrid model;

[0120] Image pairing module 02: used to batch input UI design renderings and web actual development page images into the hybrid model, and the hybrid model performs image pairing by calculating the structural similarity index;

[0121] Segmentation module 03: used to segment the UI design rendering and the web actual development page image paired with the UI design rendering according to different levels to obtain multiple modules, multiple containers and multiple components;

[0122] Comparison module 04: used to compare the UI design renderings and the paired web actual development page images layer by layer according to the module level, container level and component level, and output the comparison results.

[0123] In addition, based on the above-mentioned hybrid model-based UI consistency comparison method and system, the present invention also provides a terminal accordingly, wherein a preferred embodiment of the terminal is as follows: Figure 5 As shown, it specifically includes a processor 10, a memory 20 and a display 30. Figure 5 Only some components of the terminal are shown, but it should be understood that it is not required to implement all of the components shown, and more or fewer components may be implemented instead.

[0124] In some embodiments, the memory 20 may be an internal storage unit of the terminal, such as a hard disk or memory of the terminal. In other embodiments, the memory 20 may also be an external storage device of the terminal, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, and a flash card (Flash Card) equipped on the terminal. Furthermore, the memory 20 may also include both an internal storage unit of the terminal and an external storage device. The memory 20 is used to store application software and various types of data installed on the terminal, such as program codes of the storage terminal. The memory 20 can also be used to temporarily store data that has been output or is to be output. In one embodiment, a UI consistency comparison program 40 based on a hybrid model is stored on the memory 20, and the UI consistency comparison program 40 based on the hybrid model can be executed by the processor 10, thereby implementing the steps of the UI consistency comparison method based on the hybrid model in the present application.

[0125] In some embodiments, the processor 10 may be a central processing unit (CPU), a microprocessor or other data processing chip, used to run program codes or process data stored in the memory 20, such as executing a hybrid model-based UI consistency comparison program 40.

[0126] In some embodiments, the display 30 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, an OLED (Organic Light-Emitting Diode) touch device, etc. The display 30 is used to display information on the terminal and to display a visual user interface.

[0127] In one embodiment, when the processor 10 executes the hybrid model-based UI consistency comparison program 40 in the memory 20 , the steps of the hybrid model-based UI consistency comparison method described above are implemented.

[0128] The present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a UI consistency comparison program based on a hybrid model, and when the UI consistency comparison program based on a hybrid model is executed by a processor, the steps of the UI consistency comparison method based on a hybrid model are implemented.

[0129] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal. In the absence of further restrictions, an element defined by the sentence "includes a ..." does not exclude the existence of other identical elements in the process, method, article or terminal including the element.

[0130] Of course, those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing related hardware (such as a processor, a controller, etc.) through a computer program, and the program can be stored in a computer-readable storage medium that can be read by a computer, and the program can include the processes of the above-mentioned method embodiments when executed. The computer-readable storage medium can be a memory, a disk, an optical disk, etc.

[0131] It should be understood that the application of the present invention is not limited to the above examples. For ordinary technicians in this field, improvements or changes can be made based on the above description. All these improvements and changes should fall within the scope of protection of the claims attached to the present invention.

Claims

1. A UI consistency comparison method based on a hybrid model, characterized in that: The UI consistency comparison method based on the hybrid model includes: Acquire a historical data set, and use the historical data set to train an initial model to obtain a hybrid model; Batch input of UI design renderings and web actual development page images into the hybrid model, and the hybrid model performs image pairing by calculating a structural similarity index; Segment the UI design rendering and the web actual development page image paired with the UI design rendering according to different levels to obtain multiple modules, multiple containers and multiple components; The UI design renderings and the paired web actual development page images are compared layer by layer according to the module level, container level and component level, and the comparison results are output.

2. The UI consistency comparison method based on the hybrid model according to claim 1 is characterized in that: The hybrid model includes a segmentation model and a feature extraction model; The acquiring of the historical data set and the use of the historical data set to train the initial model to obtain the hybrid model specifically include: Obtain a historical data set, and divide the historical data set into a training set and a test set; Preprocessing the historical data set, wherein the preprocessing includes mask generation and label file generation; Using the training set to train the segmentation model and the feature extraction model to obtain a trained segmentation model and a trained feature extraction model; Using the test set to test the performance of the trained segmentation model and the trained feature extraction model; The trained segmentation model whose performance meets the test requirements is combined with the trained feature extraction model to obtain a hybrid model.

3. The UI consistency comparison method based on the hybrid model according to claim 1, characterized in that: The step of batch-inputting UI design renderings and web actual development page images into the hybrid model, and performing image pairing by calculating a structural similarity index, specifically includes: Batch input of UI design renderings and web actual development page images into the hybrid model; The hybrid model traverses and reads the UI design renderings and the web actual development page images, measures the image similarity from three aspects: brightness, contrast, and structure, and calculates the structural similarity index SSIM between the UI design renderings and the web actual development page images: Among them, SSIM(x,y) represents the structural similarity index, x represents the UI design effect diagram, y represents the actual web development page image, μ x and μ y Represents the mean of UI design renderings and web actual development page images, σ x 2 and σ y 2 Respectively represent the variance of UI design renderings and web actual development page images, σ xy Represents the covariance between UI design renderings and actual web development page images, C1 and C2 are constants; Among the multiple structural similarity indexes corresponding to the calculated UI design renderings, the web actual development page image corresponding to the highest structural similarity index is used as the web actual development page image paired with the UI design renderings.

4. The UI consistency comparison method based on the hybrid model according to claim 3 is characterized in that: The step of dividing the UI design rendering and the web actual development page image paired with the UI design rendering into different levels to obtain multiple modules, multiple containers and multiple components specifically includes: Segment the UI design rendering and the web actual development page image paired with the UI design rendering according to different modules to obtain multiple modules in the UI design rendering and multiple modules in the web actual development page image paired with the UI design rendering; Divide the interior of each module into left and right containers and upper, middle and lower containers to obtain multiple containers in the UI design rendering and multiple containers in the paired web actual development page image; Each container is divided into different components to obtain multiple components in the UI design rendering and multiple components in the paired web actual development page image.

5. The UI consistency comparison method based on the hybrid model according to claim 4 is characterized in that: The UI design renderings and the paired web actual development page images are compared layer by layer according to the module level, container level and component level, and the hybrid model outputs the comparison results, specifically including: Compare the modules in the UI design rendering with the corresponding modules in the paired web actual development page image to see if they are consistent, wherein the modules include a header module, a content module, and a bottom module; Compare the containers in the UI design rendering with the corresponding containers in the paired web actual development page image to see if they are consistent; Comparing the components in the UI design rendering with the corresponding components in the paired web actual development page image to see if they are consistent, wherein the components include buttons, colors, and texts; The hybrid model outputs a comparison result, and annotates inconsistent modules, containers or components on the web actual development page image; The comparison results include a difference report and a visualization diagram.

6. The UI consistency comparison method based on the hybrid model according to claim 5 is characterized in that: The comparing whether the components in the UI design rendering are consistent with the corresponding components in the paired web actual development page image specifically includes: A template matching algorithm is adopted, the button in the UI design rendering is used as a template, the paired web actual development page image is used as a target image, the template is slid in the target image, the structural similarity index of each region in the template and the target image is calculated, the region corresponding to the highest structural similarity index among multiple structural similarity indexes is marked as the target region, the features of the button and the target region are extracted, and the feature differences between the button and the target region are compared, and it is determined whether the button in the UI design rendering and the button in the paired web actual development page image are consistent according to the feature differences; Using a color histogram comparison algorithm, calculate the color histograms of the UI design rendering and the paired web actual development page image, use a similarity index to calculate the color histogram similarity of the UI design rendering and the paired web actual development page image, compare the color difference of the UI design rendering and the paired web actual development page image according to the color histogram similarity, and determine whether the color in the UI design rendering and the color in the paired web actual development page image are consistent according to the color difference; An OCR text recognition and comparison algorithm is adopted to extract the text of the UI design rendering and the paired web actual development page image, and the extracted text of the UI design rendering and the paired web actual development page image are preprocessed, feature extracted and serialized in sequence to generate a text sequence of the UI design rendering and the paired web actual development page image, calculate the text sequence similarity of the UI design rendering and the paired web actual development page image, compare the text difference between the UI design rendering and the paired web actual development page image according to the text sequence similarity, and judge whether the text in the UI design rendering and the text in the paired web actual development page image are consistent according to the text difference.

7. The UI consistency comparison method based on the hybrid model according to claim 6 is characterized in that: The similarity index includes Bhattacharyya distance; The using of a similarity index to calculate the color histogram similarity of the UI design rendering and the paired web actual development page image specifically includes: According to the color histogram of the UI design rendering and the color histogram of the paired web actual development page image, the color histogram similarity of the UI design rendering and the paired web actual development page image is calculated using the Bhattacharyya distance DB: DB(p,q)=-ln(BC(p,q)); Among them, DB(p,q) represents the Bhattacharyya distance, BC(p,q) represents the Bhattacharyya coefficient, p(a) and q(a) represent the probability values ​​of the color histogram of the UI design rendering and the color histogram of the paired web actual development page image at point a, a represents the independent variable, p and q represent the probabilities of the color histogram of the UI design rendering and the color histogram of the paired web actual development page image, respectively.

8. A UI consistency comparison system based on a hybrid model, characterized in that: The hybrid model-based UI consistency comparison system includes: A hybrid model building module: used to obtain a historical data set, train an initial model using the historical data set, and obtain a hybrid model; Image pairing module: used to batch input UI design renderings and web actual development page images into the hybrid model, and the hybrid model performs image pairing by calculating the structural similarity index; A segmentation module is used to segment the UI design rendering and the web actual development page image paired with the UI design rendering according to different levels to obtain multiple modules, multiple containers and multiple components; Comparison module: used to compare the UI design renderings and the paired web actual development page images layer by layer according to the module level, container level and component level, and output the comparison results.

9. A terminal, characterized in that: The terminal includes: a memory, a processor, and a hybrid model-based UI consistency comparison program stored in the memory and executable on the processor. When the hybrid model-based UI consistency comparison program is executed by the processor, the steps of the hybrid model-based UI consistency comparison method as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a hybrid model-based UI consistency comparison program, and when the hybrid model-based UI consistency comparison program is executed by a processor, the steps of the hybrid model-based UI consistency comparison method as described in any one of claims 1-7 are implemented.

Images