Method, system and equipment for automatically testing user interface by utilizing machine learning and medium

Abstract

The invention provides a method, a system, computer equipment and a computer readable medium for performing automatic evaluation and visual inspection on a graphical user interface (GUI) design based on machine learning. Wherein the graphical user interface design is scored by combining a series of image processing and conversion technologies, and a machine learning architecture with certain logic and condition rules is used for realizing automatic evaluation and visual inspection of the graphical user interface design. According to the technical scheme provided by the invention, the similarity score and the underlying structure of the graphic user interface design are obtained by using a plurality of mutually linked image conversion technologies and algorithms, and then the data structure is learned by using a machine; estimates for correlation errors are generated based on underlying structures involved by the graphical user interface in conjunction with particular logic and conditional rules.

Description

Technical Field

[0001] The present invention relates to the field of deep learning and image processing technology, and in particular to a system and method for automatically testing and evaluating a user interface (UI) by using machine learning. Background Art

[0002] Visual testing is a software testing technique used during software development to evaluate the visual appearance and interactive actions of the user interface (UI) of a software application. The purpose of visual testing is to confirm that the visual elements of the user interface (e.g., color, images, fonts, and layout) are displayed correctly and are presented consistently across different devices, operating systems, and browsers. Possible errors in user interface images may include color differences, mismatched user interface elements, incorrect graphics resolution, incorrect font size, scaling errors, inaccurate operating system rendering, inaccurate text rendering, inaccurate presentation of special characters, etc. The traditional method of visual testing is for the tester or designer to manually compare the target user interface design with the test user interface. However, there is currently no reliable method that can accurately and reliably automatically capture relevant visual features and descriptors while achieving size invariance for different devices, operating systems, and browsers.

[0003] The technical challenge of manual visual inspection of user interfaces is that the comparison process is inefficient and subject to human error. The traditional process usually requires manual inspectors to compare two user interfaces at the pixel level, relying only on human vision to confirm whether the two user interface images are consistent. This process is usually a purely manual process without tools or testing rules to assist. This results in a large number of manual inspectors required for this work. For large organizations, this work requires a large number of manual inspectors and corresponding resources to match this demand. In addition, generally speaking, user interfaces often have certain specific elements that require trained inspectors to test their functionality. This also results in human resources that could have been used for design being anchored in testing positions, and resource utilization is not efficient.

[0004] When performing manual comparison, testers can generally only select one or two test devices at a time (for example, iPhone 14) to confirm whether the tested user interface image is correct in perimeter size. For example, for buttons at the edge of the user interface, since the screen of iPhone 16pro max is slightly larger than that of iPhone 14, buttons that can be displayed normally on iPhone 16pro max may not be displayed on iPhone 14. At this point, testers can try to scale the image as a whole to match different types of devices. However, this operation may cause the pixel ratio of the final user interface to be inconsistent with the original interface. In addition, existing third-party user interface testing service providers generally do not provide automated software support and other services.

[0005] In view of the above reasons, a reliable method and system are needed to automatically compare two user interface images (e.g., a test user interface image and a target user interface design image) and evaluate their similarity, so as to minimize the human resources spent on testing. In addition, the method and system need to be robust enough to capture the inherent characteristics of the user interface image and maintain scaling invariance. Summary of the invention

[0006] In order to overcome the problems existing in the related art, the present invention provides a method, system, computer device and computer-readable medium for automatically testing user interfaces using machine learning. The technical solution provided by the present invention uses a series of image processing techniques for scoring user interface designs based on machine learning to obtain similarity scores and underlying distributions of user interface designs, and then uses a machine learning data architecture combined with a series of logic and conditional rules to generate error estimate predictions based on the underlying distribution of user interface designs. The method provided by the present invention uses a structured machine learning architecture to automatically evaluate and visually test user interfaces, and can automatically compare and evaluate design elements of static designs with target graphical user interface design data objects to generate a prediction report of error estimates, and in some embodiments, infer it as an improved user interface rendering code, so as to be deployed for dynamic use on various user interfaces.

[0007] The graphical user interface design used in the present invention may be provided in the form of, for example, a wireframe data object, a picture rendering data object, or in a data object format used by other interface design applications. During the initial design and preparation process, the graphical user interface design data object is configured for original design, and the design data object may include vector graphics representations, including static placeholders for dynamic interactive user interface control elements.

[0008] The technical solution provided by the present invention is based on artificial intelligence (AI) technology, which describes a method of using a scoring model to compare and score a user interface test image and a reference user interface design image using a similarity scoring algorithm. The score is then passed to a test module, which receives the score and generates a user interface test error report by chain evaluation of the corresponding image. For example, the scoring model can first use optical character recognition (OCR) technology to segment the user interface image into a set of pixel clusters using a segmentation algorithm, so as to use other similarity scoring algorithms for comparison and scoring. In this process, the input of each algorithm can be converted into an intermediate version of the input, which is then passed as input to the next scoring algorithm. The test module can be configured to check the degree of match between a candidate test user interface image data object and a reference user interface design image data object, and the degree of match can be represented by different degrees of match level thresholds, which can range from perfect match (high threshold) to broad match (relatively loose threshold).

[0009] The technical solution provided by the present invention uses artificial intelligence technology to efficiently and accurately evaluate the errors and differences between two user interface images, reducing development time and saving human resources consumption in visual inspection. In addition, the method provided by the present invention also enhances the integrity of error detection and enhances the scalability and robustness of mobile application development. Many large organizations invest huge economic and human costs in the testing process during the research and development process. Improving the efficiency of the testing process can bring tangible economic benefits to these large organizations.

[0010] One use scenario of the technical solution provided by the present invention is as follows: During the development of a mobile application, the designer hopes to achieve a certain creative design, but the graphical user interface design presented by the designed rendering code is quite different from the expected design (for example, icons are missing, visual elements deviate, etc., which can only be noticed by the designer). Generally speaking, traditional image comparison tools only use regression testing to perform corresponding tests. However, the technical solution provided by the present invention can effectively detect and mark all differences between the test image and the target image, and provide a report to the user (for example, the designer) for the designer to consider and improve the code presentation during the development process.

[0011] The technical solution provided by the present invention can be applied to multiple operating systems, test platforms, devices and solutions across platforms. All data can be stored in the internal server without external links.

[0012] The method provided according to some embodiments of the present invention includes first converting a graphical user interface design data object into a plurality of pixel clusters. Subsequently, a series of chain image processing is performed on these pixel clusters to obtain similarity scores representing the similarity levels of various aspects between the graphical user interface design data objects, including pure text similarity, text area segment similarity, graphic area segment similarity, image semantic similarity, graphic structure similarity, graphic features and key point similarity, etc. The generated similarity scores are then input into a test module, which may further include a rule module and a machine learning computer architecture to scan the differences between the graphical user interface design data objects according to the similarity scores. The test module then generates and outputs a graphical user interface test error report. The report may include an estimated error prediction list based on the underlying distribution of the graphical user interface design data object, which records the different OCR areas between the test graphical user interface image and the reference graphical user interface design image, including the location of each different OCR area.

[0013] In some embodiments, the technical solution provided by the present invention further includes generating an improved deployable code package based on the generated user interface test report to construct a graphical user interface data object that is closer to the target graphical user interface design data object. The improved deployable code package can automatically adapt to the test or production environment and be rendered on a user interface rendering module running on a computing device (e.g., a mobile device), or can be provided to a hosting facility (e.g., a network server), which can use a network development script engine (e.g., a PHP interpreter) to dynamically generate an output interface and provide it to the requesting computing device for rendering. In this example, the generated deployable code package can be a PHP code processed on a network server, and when the requesting computing device interacts, the network server sends the generated hypertext markup language (HTML) or other binary image data as a hypertext transfer protocol response to the requesting computing device for subsequent operations.

[0014] In some embodiments, the technical solution provided by the present invention further generates a similar user interface design to output to the user. The output is automatically adapted based on the actual device and rendering engine limitations and features. Based on the generated user interface test report, the system can generate a deployable user interface rendering code package for the improved user interface design, which can be deployed to the test or production environment in real time. The deployable user interface rendering code package can be upgraded through the package manager or pushed as an application update. For example, when the mobile application of a bank platform online system is updated, the user interface rendering code can be received and automatically upgraded on the client side device.

[0015] In one aspect, the present invention provides a method for automatic visual testing of a graphical user interface design, comprising: receiving a candidate data object and a reference data object, each of which represents a graphical user interface design; using optical character recognition to convert the candidate data object and the reference data object into a candidate machine-coded text object and a reference machine-coded text object, respectively, and performing text and layout feature comparison to generate a first similarity score; using an object mask to segment the candidate machine-coded text object and the reference machine-coded text object into a candidate pixel cluster set and a reference pixel cluster set, respectively; and generating a candidate image color histogram based on the pixel distribution of the candidate pixel cluster set and the reference pixel cluster set. and a reference image color histogram, comparing the color chalk between the candidate image color histogram and the reference image color histogram to generate a second similarity score; generating a candidate embedding vector and a reference embedding vector based on the candidate pixel cluster set and the reference pixel cluster set by image embedding technology to calculate a third similarity score, wherein the third similarity score is a cosine similarity score between the candidate embedding vector and the reference embedding vector; locating a sub-cluster of the candidate pixel cluster set and the reference pixel cluster set, performing template matching between the candidate data object and the reference data object, and generating a fourth similarity score; calculating and extracting a candidate description based on the local intensity extreme values ​​of the candidate pixel cluster set and the reference pixel cluster set. The method comprises the steps of: selecting a candidate pixel cluster set and a reference pixel cluster set for comparison and generating a fifth similarity score, wherein each descriptor object represents a feature point in the data object, and comparing each descriptor object of the candidate descriptor object set with each descriptor object of the reference descriptor object set; converting the candidate pixel cluster set and the reference pixel cluster set into a grayscale format to compare the brightness and contrast between each pixel of the candidate pixel cluster set and each pixel of the reference pixel cluster set, and generating a sixth similarity score; providing the first, second, third, fourth, fifth and sixth similarity scores to a trained machine learning model data architecture for scanning and comparing with a set of similarity scores that pass a threshold set; and generating an output structure data object based on the result of the comparison between the first, second, third, fourth, fifth and sixth similarity scores and the passed threshold set, the output structure data object comprising a list of ordered data pairs, wherein each ordered data pair comprises a candidate pixel cluster in the candidate pixel cluster set and a reference pixel cluster in the corresponding reference pixel cluster set, wherein the candidate pixel cluster and its corresponding reference pixel cluster are detected by the trained machine learning model data structure as differences between the candidate data object and the reference data object, wherein the output structure data object can be used to correct the differences between the candidate data object and the reference data object.

[0016] According to a further embodiment of the present invention, the method further includes: inputting the reference data object and the output structure data object into a system for automatically generating user interface rendering code, generating a recommended text instruction set to generate an improved reference data object; the recommended text instruction set is transmitted to a development and testing environment for packaging and executing to generate user-accessible rendered user interface visual elements.

[0017] According to a further embodiment of the present invention, the method further includes: obtaining a similarity score set of the improved reference data object; setting a first threshold for the similarity score set; determining whether the similarity scores in the similarity score set reach or exceed the first threshold; and if the similarity scores in the similarity score set reach or exceed the first threshold, automatically transmitting the recommended text instruction set to a development test environment to encapsulate and execute to generate user-accessible rendered user interface visual elements.

[0018] According to a further embodiment of the present invention, the method further includes: setting a second threshold for the similarity score set; determining whether the similarity scores in the similarity score set reach or exceed the second threshold; and if the similarity scores in the similarity score set reach or exceed the second threshold, automatically deploying the recommended text instruction set to a production server accessible to multiple users.

[0019] According to a further embodiment of the present invention, the method further comprises: packaging the recommendation text instruction set into a machine language instruction set; and outputting the machine language instruction set to the user to re-implement the user interface visual elements.

[0020] According to a further embodiment of the present invention, the method further comprises: linking the machine language instruction set to an executable binary file, wherein the executable binary file is an aggregation of the machine language instruction set; and outputting the executable binary to a user to re-implement the user interface visual elements.

[0021] According to a further embodiment of the present invention, the method further comprises: running the executable binary file to render a graphical user interface; and outputting the rendered graphical user interface to a user to re-implement the user interface visual elements.

[0022] According to a further embodiment of the present invention, the method further includes: setting a third threshold for the similarity score set; determining whether the similarity scores in the similarity score set reach the third threshold; and automatically discarding the recommended text instruction set if the similarity scores in the similarity score set do not reach the third threshold.

[0023] According to a further embodiment of the present invention, the above method of using an object mask to segment the candidate machine-coded text object and the reference machine-coded text object into a candidate pixel cluster set and a reference pixel cluster set, respectively, includes: applying an object mask to the candidate machine-coded text object and the reference machine-coded text object, and replacing the text objects in the candidate data object and the reference data object with one or more white pixel clusters.

[0024] According to a further embodiment of the present invention, the above method of using an object mask to segment the candidate machine-coded text object and the reference machine-coded text object into a candidate pixel cluster set and a reference pixel cluster set, respectively, further includes: applying an object mask to the candidate machine-coded text object and the reference machine-coded text object, replacing the graphic symbol objects in the candidate data object and the reference data object with one or more white pixel clusters.

[0025] Another aspect of the present invention provides a system for automatic visual testing of a graphical user interface design, the system comprising: a test module, configured to receive a candidate data object and a reference data object, each of which represents a graphical user interface design; an optical character recognition module, configured to receive the candidate data object and the reference data object, use optical character recognition to convert the candidate data object and the reference data object into a candidate machine-coded text object and a reference machine-coded text object, respectively, compare text and layout features, and generate a first similarity score; use an object mask to segment the candidate machine-coded text object and the reference machine-coded text object into a candidate pixel cluster set and a reference pixel cluster set, respectively; the test module is further configured to generate a candidate image color histogram and a reference image color histogram based on the pixel distribution of the candidate pixel cluster set and the reference pixel cluster set, compare the color chalk between the candidate image color histogram and the reference image color histogram, and generate a second similarity score; generate a candidate embedding vector and a reference embedding vector based on the candidate pixel cluster set and the reference pixel cluster set by image embedding technology. the candidate pixel cluster set and the reference pixel cluster set to calculate a third similarity score, wherein the third similarity score is a cosine similarity score between the candidate embedding vector and the reference embedding vector; locate subclusters of the candidate pixel cluster set and the reference pixel cluster set, perform template matching between the candidate data object and the reference data object, and generate a fourth similarity score; calculate and extract a candidate descriptor object set and a reference descriptor object set based on local intensity extreme values ​​of the candidate pixel cluster set and the reference pixel cluster set for comparison and generating a fifth similarity score, each of the descriptor objects representing a feature point in the data object, the comparison comprising comparing each descriptor object of the candidate descriptor object set with each descriptor object of the reference descriptor object set; convert the candidate pixel cluster set and the reference pixel cluster set into a grayscale format to compare the brightness and contrast between each pixel of the candidate pixel cluster set and each pixel of the reference pixel cluster set, and generate a sixth similarity score; provide the first, second, third, fourth, fifth and sixth similarity scores to a rule module for scanning and comparison with a set of passing threshold sets;The system further comprises a rule module, which is configured to: generate an output structure data object based on the results of the comparison between the first, second, third, fourth, fifth and sixth similarity scores and the passing threshold set, wherein the output structure data object comprises a list of ordered data pairs, wherein each of the ordered data pairs comprises a candidate pixel cluster in the candidate pixel cluster set and a reference pixel cluster in the corresponding reference pixel cluster set, wherein the candidate pixel cluster and the corresponding reference pixel cluster are detected by the trained machine learning model data structure as differences between the candidate data object and the reference data object, wherein the output structure data object can be used to correct the differences between the candidate data object and the reference data object.;

[0026] According to a further embodiment of the present invention, the rule module in the above system is further configured to: input the reference data object and the output structure data object into a system for automatically generating user interface rendering code, generate a recommended text instruction set to generate an improved reference data object; the recommended text instruction set is transmitted to the development and testing environment for packaging and executing to generate user-accessible rendered user interface visual elements.

[0027] According to a further embodiment of the present invention, the rule module in the above-mentioned system is further configured to: obtain a set of similarity scores of the improved reference data objects; set a first threshold for the similarity score set; determine whether the similarity scores in the similarity score set reach or exceed the first threshold; and if the similarity scores in the similarity score set reach or exceed the first threshold, automatically transmit the recommended text instruction set to the development test environment to encapsulate and execute to generate user-accessible rendered user interface visual elements.

[0028] According to a further embodiment of the present invention, the rule module in the above system is further configured to: set a second threshold for the similarity score set; determine whether the similarity score in the similarity score set reaches or exceeds the second threshold; and if the similarity score in the similarity score set reaches or exceeds the second threshold, automatically deploy the recommended text instruction set to a production server accessible to multiple users.

[0029] According to a further embodiment of the present invention, the rule module in the above system is further configured to: encapsulate the recommended text instruction set into a machine language instruction set; and output the machine language instruction set to the user to re-implement the user interface visual elements.

[0030] According to a further embodiment of the present invention, the rule module in the above system is further configured to: link the machine language instruction set to an executable binary file, wherein the executable binary file is an aggregation of the machine language instruction set; and output the executable binary to the user to re-implement the user interface visual elements.

[0031] According to a further embodiment of the present invention, the rule module in the above system is further configured to: run the executable binary file to render a graphical user interface; and output the rendered graphical user interface to the user to re-implement the user interface visual elements.

[0032] According to a further embodiment of the present invention, the rule module in the above system is further configured to: set a third threshold for the similarity score set; determine whether the similarity score in the similarity score set reaches the third threshold; and if the similarity score in the similarity score set does not reach the third threshold, automatically discard the recommended text instruction set.

[0033] According to a further embodiment of the present invention, the operation of the optical character recognition module in the above-mentioned system using an object mask to segment the candidate machine-coded text object and the reference machine-coded text object into a candidate pixel cluster set and a reference pixel cluster set respectively may further include: applying an object mask to the candidate machine-coded text object and the reference machine-coded text object, replacing the text objects in the candidate data object and the reference data object with one or more white pixel clusters.

[0034] According to a further embodiment of the present invention, the operation of the optical character recognition module in the above-mentioned system using an object mask to segment the candidate machine-coded text object and the reference machine-coded text object into a candidate pixel cluster set and a reference pixel cluster set respectively may further include: applying an object mask to the candidate machine-coded text object and the reference machine-coded text object, and replacing the graphic symbol objects in the candidate data object and the reference data object with one or more white pixel clusters.

[0035] On the other hand, the present invention provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the method for automatic visual testing of graphical user interface design described above is implemented, which will not be described in detail here.

[0036] On the other hand, the present invention provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the above-mentioned method for automatic visual testing of graphical user interface design, which will not be described in detail here.

[0037] The technical solution provided by the present invention improves the evaluation and visual testing process of graphical user interface designs, and solves the problem that manual comparison of user interface designs is time-consuming, resource-consuming, and prone to human errors, thereby hindering the rapid development of applications. Existing tools have limitations in accurately capturing and comparing graphical user interface specifications and components, which often leads to inaccurate test reports of graphical user interface designs. In addition, since the comparison of graphical user interface designs is all done manually, existing tools cannot handle the problem of significant expansion. This gap between design and functional implementation leads to increased development time and inconsistencies in user interfaces on different platforms.

[0038] Many further features and combinations of the embodiments described herein will be apparent to those skilled in the art after reading this disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 is a flow chart of an example system for automatically evaluating and visually testing a test graphical user interface image and a graphical reference user interface design image according to some embodiments of the present invention;

[0040] Figure 2 is a block diagram of an example OCR module of a system for automatically evaluating and visually testing a test graphical user interface image and a graphical reference user interface design image according to some embodiments of the present invention;

[0041] Figure 3 is a block diagram of an example testing module of a system for automatically evaluating and visually testing an overall test graphical user interface image and a graphical reference user interface design image according to some embodiments of the present invention;

[0042] Figure 4 is a block diagram of an example testing module of a system for automatically evaluating and visually testing segmented OCR text regions and graphical reference user interface design images of a test graphical user interface according to some embodiments of the present invention;

[0043] Figure 5 Example comparison results of automatic evaluation and visual testing of a test GUI image and a GUI reference design image using technical solutions currently available in the market according to some embodiments of the present invention are shown.

[0044] Figure 6 An example user interface for a user to input a graphical user interface design image and a test graphical user interface image to a system for automatic evaluation and visual testing of input images according to some embodiments of the present invention is shown.

[0045] Figure 7An example test graphical user interface image and an example graphical reference user interface design image are shown for comparison according to some embodiments of the present invention.

[0046] Figure 8 An example graphical user interface test report generated by comparing an example test graphical user interface image and an example graphical reference user interface design image by the technical solution of the present invention according to some embodiments of the present invention is shown.

[0047] Fig. 9 It is a schematic diagram of the structure of a computer device and a computer-readable medium carried thereon provided according to one aspect of the present invention. DETAILED DESCRIPTION

[0048] The preferred embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although the preferred embodiments of the present invention are shown in the accompanying drawings, it should be understood that the embodiments shown in the accompanying drawings are only examples, and the present invention can be implemented in various forms in practical applications and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided to make the present invention more thorough and complete, and to fully convey the scope of the present invention to those skilled in the art.

[0049] The method provided by the present invention uses a structured machine learning architecture to automatically evaluate and visually test the user interface, and can automatically compare and evaluate the design elements of the static design with the target graphical user interface design data object to generate a prediction report of the error estimate, and in some embodiments, infer it as an improved user interface rendering code, so as to be deployed for dynamic use on various user interfaces. The graphical user interface design used in the present invention can be provided in the form of, for example, a wireframe data object, a picture rendering data object, or a data object format used by other interface design applications. During the initial design and preparation process, the graphical user interface design data object is configured for original design, and the design data object can include a vector graphic representation, including static placeholders for dynamic interactive user interface control elements.

[0050] The technical solution provided by the present invention helps improve the efficiency of computing and resource utilization when performing large-scale operations. In at least one aspect, the technical solution of automatic comparison and evaluation proposed by the present invention overcomes the complex challenges to computing resources caused by large-scale graphical interface development and improves the efficiency of related computing and resource utilization. By automating the evaluation and visual testing of graphical user interface design, the technical solution of the present invention effectively connects user interface design and mobile application code development, thereby improving production efficiency and quality.

[0051] The technical solution provided by the present invention uses an image processing scoring algorithm and a machine learning computer architecture to automatically compare graphical user interface designs, thereby minimizing the human resources invested in user interface design testing while ensuring the accuracy of the test.

[0052] Figure 1 is a flow chart of an example system for automatically evaluating and visually testing test graphical user interface images and graphical reference user interface design images according to some embodiments of the present invention.

[0053] The system 100 includes an automated process configured to first convert a GUI design data object into a plurality of pixel clusters. Subsequently, a series of chained image processing is performed on the pixel clusters to obtain similarity scores representing the similarity levels of various aspects between the GUI design data objects, including pure text similarity, text region segment similarity, graphic region segment similarity, image semantic similarity, graphic structure similarity, graphic features and key point similarity, etc. The generated similarity scores are then input into a test module, which may further include a rule module and a machine learning computer architecture to scan the differences between the GUI design data objects based on the similarity scores. The test module then generates and outputs a GUI test error report. The report may include a list of estimated error predictions based on the underlying distribution of the GUI design data objects, which records the different OCR areas between the test GUI image and the reference GUI design image, including the location of each different OCR area.

[0054] System 100 uses various algorithms and calculations to perform image conversion on GUI design data objects. Table 1 shows different algorithms that can be used to obtain similarity scores based on different types of features.

[0055]

[0056] Table 1

[0057] Optical Character Recognition (OCR) is an algorithm that converts text in a graphic into a machine-readable text format. OCR can compare text, characters, and layout features between two images by recognizing and extracting text from the images. OCR is reliable for comparing various fonts and typefaces, and is suitable for both real-time and offline processing. Generally speaking, OCR is the first conversion and processing applied to the graphical user interface design object in the system 100.

[0058] Segmentation is an algorithm that divides a digital image into multiple image segments, also called image regions or image objects (pixel sets or pixel clusters). Segmentation uses object masks to separate an image into image segments. In system 100, the GUI design object can be segmented after the OCR process (e.g., Figure 2 ).

[0059] Object masking techniques are used to isolate specific objects or portions of an image. Object masking can remove unwanted portions of an image by locating the boundaries of the unwanted portions or objects and placing the object mask along those boundaries. In some embodiments, the object mask can be a raster mask consisting of a grid of pixels, where each pixel can be set to be opaque or transparent. In some embodiments, the object mask can be a vector mask consisting of a series of points, lines, and curves that can be combined to create a mask with a complex shape. In some embodiments, the object mask can be a bitmap mask consisting of a series of bits that determine whether a pixel is black or white.

[0060] In some embodiments, the system segments the OCR processed design object, and all text in the original GUI design object is separated separately. For example, the bitmap object mask can mark the text area of ​​the OCR processed object as white pixels, while keeping all other areas of the OCR processed object as their original pixels.

[0061] In some embodiments, the system segments the design object after OCR processing, and all icons in the original GUI design object are separated separately.

[0062] Scale-invariant feature transform (SIFT) is a computer vision technique for detecting, describing, and matching local features in images. SIFT can provide a comparison of local feature points and descriptors (e.g., key points, edges, etc.) between two images or GUI designs. SIFT is invariant to size, rotation, and illumination, and is reliable regardless of size, rotation, or illumination changes. SIFT has a higher computational complexity and is therefore more suitable for offline computation.

[0063] Contrastive language-image pre-training (CLIP) is a computational scheme for comparing image features, such as objects, scenes, semantics, colors, shapes, etc. CLIP determines the similarity between two images by calculating the cosine similarity score between the vectors (an arithmetic representation) of the two images. CLIP is based on multimodal image and text joint training, which has a high degree of semantic understanding and natural language learning capabilities.

[0064] The color histogram algorithm can be used to measure the similarity of color distribution in an image. A color histogram is a graphical representation of the color distribution in an image phenomenon, which can be used for statistical analysis to determine the distribution of colors in different regions. Color histograms have the characteristics of low computational complexity and are suitable for both real-time and offline processing.

[0065] Template matching is a digital image processing technology used to find the portion of an image (e.g., a test GUI image) that matches a template image (e.g., a graphical reference UI design image data object). Template matching can locate the portion of the test GUI image that matches the graphical reference UI design image data object by means of a sliding window search.

[0066] Structural Similarity Index Measure (SSIM) is an algorithm for determining the overall structural similarity between two images, which is used to determine the similarity of image brightness, contrast, and structure. SSIM compares the structural features of two images by comparing the similarity of pixels (i.e., whether the pixels of the two images are aligned or have similar pixel density values).

[0067] The inventors selected the algorithm solutions listed in Table 1 because each of them can provide comparison and similarity scores for different parameters (eg, structure, color, pixel, etc.) of the same image.

[0068] A reference embodiment will be described below to illustrate how some embodiments of the technical solutions provided by the present invention apply the algorithm solutions in Table 1. First, a candidate test user interface image screenshot data object (as a "test data object") and a reference user interface design image data object (as a "reference data object") are received from a user. The OCR module then uses OCR technology to convert the two data objects into a test machine-encoded text object and a reference machine-encoded text object, respectively, thereby comparing the text and layout features of the two images to generate a first similarity score. The OCR module then uses an object mask to divide the test machine-encoded text object and the reference machine-encoded text object into a test pixel cluster set and a reference pixel cluster set.

[0069] Subsequently, the test module can generate its corresponding color histogram, embedded vector, pixel cluster subset and descriptor object set based on the test pixel cluster set and the reference pixel cluster set.

[0070] The testing module may generate a color histogram based on the distribution of pixels of the testing pixel cluster and the reference pixel cluster to compare the color distribution of the two images, and then generate a second similarity score.

[0071] The test module can also generate an embedding vector by performing image embedding through the CLIP algorithm based on the test pixel cluster and the reference pixel cluster, so that the cosine similarity score between the test pixel cluster and the reference pixel cluster can be calculated as the third similarity score. The test module can also generate a fourth similarity score by locating a subset of the test pixel cluster and the reference pixel cluster to perform template matching.

[0072] The testing module can also calculate and extract descriptor objects based on the local intensity extrema of the test pixel cluster and the reference pixel cluster, and then use the SIFT algorithm to compare them, including comparing each descriptor object in the test pixel cluster with each descriptor object in the reference pixel cluster, thereby generating a fifth similarity score.

[0073] The test module can then convert the test pixel cluster and the reference pixel cluster into a grayscale format to compare the brightness and contrast between each pixel in the test pixel cluster and the reference pixel cluster using the SSIM algorithm to generate a sixth similarity score. The test module can then provide the first, second, third, fourth, fifth and sixth similarity scores to a trained machine learning model data architecture to compare these similarity scores with respective thresholds. The test module then generates a user interface test report based on the results of the comparison.

[0074] The user interface test report includes a series of ordered data pairs, each of which includes a test pixel cluster in a test pixel cluster set and a reference pixel cluster in a corresponding reference pixel cluster set, and the pair of test pixel clusters and reference pixel clusters are test data objects and reference data objects that have differences detected by the machine learning model. The user can then use the user interface test report to correct the differences between the candidate test user interface image data objects and the reference user interface images.

[0075] Figure 2 2 is a flowchart of an exemplary OCR module of a system for automatically evaluating and visually testing candidate test GUI image data objects and reference UI design images according to some embodiments of the present invention. The flowchart 200 describes the OCR conversion steps of the system 100 .

[0076] First, the reference UI design image data object and the test UI screenshot image data object are input into the OCR module and the test module of the system 100. The OCR module performs OCR on the reference UI design image data object and the test UI screenshot image to convert them into machine-encoded text objects for segmentation.

[0077] In some embodiments, the OCR module uses OCR to identify and extract plain text from the image. If the plain text recognized by OCR in the reference user interface design image data object and the test user interface screenshot image data object is the same, the OCR plain text is further provided to the test module as input for similarity analysis and scoring. If the plain text in the reference user interface design image data object and the test user interface screenshot image data object is different, the plain text recognized by OCR will be included in the comparison error report and marked as a difference or error.

[0078] In some embodiments, the OCR module can use OCR to divide the entire image (reference user interface design image data object and test user interface screenshot image data object) into individual OCR small areas. In some embodiments, the icons in the OCR small areas can be separated out after OCR processing. After all OCR text and icons are separated, the image will be provided to the test module as input for similarity analysis and scoring. If the icons divided in the two images are the same, the icon will be provided to the test module as input for further similarity analysis and scoring. If the icons divided in the two images are different, the icon will be included in the comparison error report and marked as a difference or error.

[0079] In some embodiments, the OCR module is further configured to apply an object mask to the text objects of the candidate test user interface screenshot image data object and the reference user interface design image data object, replace the text objects in the test user interface image data object and the reference user interface design image data object with one or more white pixel clusters, and then the OCR module segments the candidate test user interface image data object and the reference user interface design image data object.

[0080] In some embodiments, the OCR module is further configured to apply an object mask to the machine-encoded text objects converted from the candidate test user interface image data objects and the reference user interface design image data objects, replace the graphic symbol objects and icons of the candidate test user interface image data objects and the reference user interface design image data objects with one or more white pixel clusters, and then segment the machine-encoded text objects converted from the candidate test user interface image data objects and the reference user interface design image data objects.

[0081] After the test module generates a score for the image, the score is forwarded to the rules module to determine which specific items to include in the comparison error report.

[0082] Figure 3 is a flowchart diagram illustrating an example testing module of a system for automatically evaluating and visually testing an overall test GUI image and a GUI reference design image data object according to some embodiments of the present invention.

[0083] Flow 300 shows the process of similarity analysis and scoring of the overall test GUI image and the GUI test image. The input provided to the test module can be either the reference UI design image data object and the test UI screenshot image in the original format, or the reference UI design image data object and the test UI screenshot image with OCR text and icons segmented out.

[0084] In the testing module, various algorithms and transformations are applied to the input image. Figure 3 As shown in , in some embodiments, images can be processed using CLIP, color histogram, SSIM, SIFT+Robust, and template matching algorithms, each of which generates a similarity score that is provided to the rule module as input.

[0085] The rule module sets a minimum score for each similarity score and checkpoint / conditional logic to verify whether the similarity score of each type meets the minimum similarity requirement. If the similarity score is below the respective threshold, the feature and similarity score will be recorded as an error in the comparison error report.

[0086] In some embodiments, the test module also applies conditional combinations of various algorithm schemes to the image to calculate a combined similarity score. This combined similarity score is useful when the test UI screenshot image is highly similar to the reference UI design image and it is difficult to find small differences between the two images. For example, if the test UI screenshot image scores 0.92 in all single algorithm scheme scores, its combined similarity score may be a lower 0.9, which is more representative of potential differences or errors in the image.

[0087] The test module is implemented through a machine learning computing architecture, which is trained with a data set consisting of application page screenshots. For example, taking the banking mobile application of the iPhone 14 device as an example, the training set of the machine learning computing architecture can be 200 screenshots of the banking mobile application captured on the iPhone 14 device. As another example, the machine learning computing architecture can also include a separate training data set consisting of 200 screenshots of the banking mobile application on the Samsung Galaxy S22 device.

[0088] For other industry applications, such as e-commerce applications, stock market websites, etc., the business purposes and web pages of such applications or websites may be more complex than those of banking applications. This requires a larger training data set consisting of more images to train the machine learning computing architecture.

[0089] The general training process is that the training data set is provided to the test module (scoring module) as input to obtain the score of each test screenshot page in the training data set. The machine learning computing framework can be trained with these training data sets. By comparing the obtained scores with the thresholds corresponding to each algorithm, the machine learning computing framework then classifies these scores as pass or fail.

[0090] Figure 4 4 is a process block diagram of an example test module of a system for automatically evaluating and visually testing segmented OCR text regions of a test GUI and a graphical reference UI design image according to some embodiments of the present invention. Process 400 illustrates a process for similarity analysis and scoring of segmented icons or OCR text regions of a test GUI image relative to a graphical reference UI design image. Identical icons or OCR text regions in two images may be provided to the test module as input.

[0091] In the testing module, various algorithmic schemes and transformations are applied to the input image segments. Figure 4As shown in , in some embodiments, the image segments can be processed using CLIP, color histogram, SSIM, and SIFT+Robust algorithms, each of which generates a similarity score that is provided to the rule module as input.

[0092] The rule module sets a minimum score for each similarity score and checkpoint / conditional logic to verify whether the similarity score of each type meets the minimum similarity requirement. If the similarity score is below the respective threshold, the feature and similarity score will be recorded as an error in the comparison error report.

[0093] In some embodiments, the test module also applies conditional combinations of various algorithm schemes to the image segments to calculate a combined similarity score. This combined similarity score is useful when the icons of the test UI screenshot image are highly similar to the icons of the reference UI design image, and it is difficult to find small differences between the icons of the two images. For example, if the icon of the test UI screenshot image scores 0.92 in all single algorithm scheme scores, its combined similarity score is likely to be a lower 0.9, which is more representative of the potential differences or errors between the icons in the images.

[0094] Figure 5 An example comparison result between a system for automatically evaluating and visually testing a test GUI image and a GUI reference design image according to some embodiments of the present invention and technical solutions currently available in the market is shown.

[0095] Table 2 shows the results of the comparison between the technical solution of the present invention and the Percy solution.

[0096]

[0097]

[0098] Table 2

[0099] Compared with the Percy scheme, the technical solution proposed by the present invention can compare more attributes of the image. For example, the Percy scheme does not compare local feature points and descriptors or color distribution features when performing pixel-level comparison. Compared with the technical solution of the present invention, the Percy scheme also consumes more computing resources and has a higher computing complexity, using more CPU and memory space.

[0100] Figure 6 An example user interface for a user to input a graphical user interface design and a test graphical user interface image to a system for automatic evaluation and visual testing of input images according to some embodiments of the present invention is shown.

[0101] Figure 6 The technical solution of the present invention shown in provides three levels of similarity testing. The first is the "basic test". At this level, the system will use loose standards and loose thresholds when detecting whether the images match and scoring the similarity. This solution is more suitable for cross-device and cross-platform comparisons. Because, when using loose standards, the differences between two devices that differ by more than 2 can be ignored. For example, this solution can be applied to compare a page captured on an iPhone 15 device with a page captured on an iPhone 8plus device.

[0102] The second method is "Balanced Test". In this level, the system will apply standard benchmarks that are sensitive to different rendering methods to detect acceptable matches and similarity scores between images. This solution is suitable for devices with a generation difference of less than 2 generations on the same platform. For example, this solution can be used to compare images captured on an iPhone 11Pro device with images captured on an iPhone 13 device.

[0103] The third mode is "strict testing". At this level, the system applies strict criteria to perform pixel-level comparisons. This scheme uses strict thresholds to find near-perfect matches and only supports images of the same size for comparison. This scheme is suitable for the same device model. For example, this scheme can be used to compare two images captured on an iPhone 12 device.

[0104] Figure 7 An example test graphical user interface image and an example graphical reference user interface design image are shown for comparison according to some embodiments of the present invention.

[0105] exist Figure 7 In the example, the reference GUI design image data object 702 and the test GUI image data object 704 both show the home page of the example mobile banking application. The test GUI image 704 has Chinese text "Hello Hello" in white font color and red background; and Chinese text "Today's weather is really good" in red font color and white background (e.g. Figure 7 ), while these texts do not exist in the reference GUI design image data object 702. This difference is the type of difference between GUI images that the technical solution provided by the present invention is intended to detect and analyze.

[0106] Various embodiments described herein are configured to handle several different types of differences between a test GUI image and a reference GUI design image. For example, the difference between the test GUI image and the reference GUI design image is the exact location of certain text or icons, the displacement of graphic patterns or text, the overlap between graphic groups, the two images being captured on different types of devices, the absence of features such as graphic patterns or text, the color difference or stretching of the entire image or part of it, etc.

[0107] Figure 8 An example graphical user interface test report generated by comparing an example test graphical user interface image data object with an example graphical reference user interface design image data object according to some embodiments of the present invention is shown. Specifically, Figure 8 The corresponding Figure 7 A graphical user interface test report showing a comparison between the two images shown in FIG. 1 (ie, the comparison error report mentioned above).

[0108] like Figure 8 As shown, the generated test report includes both the original images and the segmented images or mask images (i.e., Figure 3 The global comparison result of the original images shows whether the scores generated by the test module for the original test GUI image and the reference GUI design image exceed the threshold set by the rule module. Similarly, the comparison result between the mask images shows whether the similarity scores generated by the test module for the original test GUI image and the reference GUI design image after the OCR text and icons are segmented out exceed the threshold set by the rule module. The set threshold varies with the level of similarity test selected by the user (such as Figure 6 ), which also changes the difficulty of achieving a "pass" grade in the GUI test report.

[0109] exist Figure 8 In the , the global comparison results between the original images and the comparison results between the mask images are both "passed". This means Figure 7 The test GUI image data object and the GUI reference design image data object are similar enough to pass the threshold set by the test module. The GUI test report further displays a "Different OCR(s)based on Test Image" section, which lists the different OCR regions between the test GUI image and the reference GUI design image. The result also shows the location of each different OCR region. Figure 8As shown in the report, the test GUI image has differences in the Chinese text in two areas. First, the image has white "Hello Hello" text with a red background; second, the image has red "Today's weather is really good" text with a white background. These two places are different from the corresponding parts of the GUI design image that only have pure red and pure white.

[0110] In some embodiments, the output GUI test report can be used to generate a recommended text instruction set to improve the rendering of a reference GUI design image. This embodiment is achieved by inputting the reference GUI design image data object and the generated test report into a system that automatically generates user interface rendering code to generate the recommended text instruction set. The generated recommended text instruction set can be transmitted to a development test environment to be packaged and executed to generate rendered user interface visual elements for the user.

[0111] In some embodiments, the rule module can obtain a set of similarity scores for the improved reference graphical user interface design image; set a first threshold for the similarity scores; determine whether the set of similarity scores reaches or exceeds the first threshold; if the set of similarity scores reaches or exceeds the first threshold, automatically transfer the generated recommended text instruction set to the development test environment for packaging and execution to generate rendered user interface visual elements for the user.

[0112] In some embodiments, the rule module may also set a second threshold to determine whether the group similarity score reaches or exceeds the second threshold; if the group similarity score reaches or exceeds the second threshold, the generated recommended text instruction set is automatically deployed to a production server accessible to multiple users.

[0113] In some embodiments, the generated recommendation text instruction set may be encapsulated into a machine language instruction set, and the machine language instruction set may be output to the user to re-implement the user interface visual elements.

[0114] In some embodiments, the machine language instruction set may be further linked to an executable binary file, wherein the executable binary file is an aggregation of the machine language instruction set and may be provided to a user to re-implement the user interface visual elements.

[0115] In some embodiments, the rule module may set a third threshold; determine whether the group similarity score reaches the third threshold, and automatically discard the generated recommended text instruction if the group similarity score does not reach the third threshold.

[0116] Users of the system can use the results in the test report to efficiently identify where errors or differences may exist between the test graphical user interface image and the graphical reference user interface design drawing, and correct these errors if necessary to obtain the correct user interface image.

[0117] In some embodiments, the technical solution proposed by the present invention can be further configured to generate a set of recommended text instructions or codes to improve the rendering of the graphical user interface design image by inputting the reference graphical user interface design image and the generated test report into the system for automatically generating the user interface rendering code. The generated recommended text instructions or codes can be verified and approved by a manual reviewer to be added to the development and testing environment for compilation and execution, and finally output to the user.

[0118] Fig. 9 Schematic diagram of the structure of a computer device according to one aspect of the present invention. Fig. 9 As shown, the computing device may include: at least one processor ( Fig. 9 Only one is shown in the figure), a memory, a computer-readable storage medium, and a computer program stored on the computer-readable storage medium in the memory and executable on at least one processor, and when the processor executes the computer program, various steps in the above-mentioned embodiment of the method for automatically visually testing a graphical user interface design are implemented. It can be understood by those skilled in the art that Fig. 9 This is merely an example of a computer device and does not constitute a limitation on the computer device. The computer device may include more or fewer components than those shown in the figure, or a combination of certain components, or different components. For example, it may also include a network interface, a display screen, and an input device.

[0119] Those skilled in the art will appreciate that the above description and examples are for demonstration purposes only. Different technologies and techniques may be used to represent information and signals. For example, data, instructions, commands, information, signals, bits, symbols, and chips that may be mentioned in the above description may be represented by voltage, current, electromagnetic waves, magnetic fields or particles, light fields or particles, or a combination thereof.

[0120] The functional modules and units described in the present invention may include processors, electronic devices, hardware devices, electronic components, logic circuits, memories, software codes, firmware codes, etc., or any combination thereof. In addition, the technical features discussed herein may be implemented by dedicated processor circuits, or by executable instructions, and / or a combination thereof.

[0121] As will be understood by those of ordinary skill in the art, various terms are used only to describe specific embodiments and are not intended to limit the embodiments. For example, ordinal numbers used herein (such as "first", "second", "third", etc.) are used to modify an element, such as a structure, assembly, operation, etc., and do not themselves indicate the priority or order of the element relative to another element, but are only used to distinguish it from another element of the same name (without these ordinal numbers, the elements of the same name will not be distinguishable). The term "coupled" is defined as connected, although not necessarily directly connected or mechanically connected; the two items of "coupled" may be one. "One" is defined as one or more, unless the present disclosure explicitly requires other meanings. The term "substantially" means that the specified standard is met in the vast majority of cases, but not completely consistent-it covers the specified standard; for example, "substantially 90 degrees" includes 90 degrees, and "substantially parallel" includes parallel. In any disclosed embodiment, the term "substantially" can be replaced by "within a certain percentage", where the percentage includes 0.1%, 1%, 5% and 10%. The term "almost" can be replaced by "within ten percent". In any disclosed embodiment, "and / or" means in conjunction with or. For example, A, B, and / or C includes: A alone, B alone, C alone, a combination of A and B, a combination of A and C, a combination of B and C, or a combination of A, B, and C. In other words, "and / or" acts as an inclusive but not limited "or". In addition, "A, B, C, or a combination thereof" or "A, B, C, or any combination thereof" includes: A alone, B alone, C alone, a combination of A and B, a combination of A and C, a combination of B and C, or a combination of A, B, and C.

[0122] It should be noted that, in this article, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, device, article or method including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, device, article or method. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the presence of other identical elements in the process, device, article or method including the element. The implementation of any device, system and method may consist of or be substantially composed of the steps, elements and (or) features described, rather than just containing / including / having these steps, elements and (or) features.

[0123] Furthermore, a device or system configured in a certain way is configured in at least that way, but it may also be configured in other ways not specifically described. Some aspects of one example may be applied to other examples even if not described or illustrated, unless this is expressly prohibited by the disclosure or the nature of the particular example.

[0124] Those skilled in the art will further understand that the various illustrative logic blocks, modules, circuits and algorithm steps associated with the present disclosure can be implemented as electronic hardware, computer software or a combination of the two. In order to clearly illustrate this interchangeability of hardware and software, the above-mentioned illustrative components, blocks, modules, circuits and steps are generally described by their functions. Whether this function is implemented as hardware or software depends on the design limitations imposed on the specific application and the overall system. Those skilled in the art may implement the described functions in different ways for each specific application, but these implementation decisions should not be understood as making the technical solution out of the scope of this article. Those skilled in the art will also easily recognize that the order or combination of components, methods or interactions described here are only examples, and the components, methods or interactions of various aspects of this article may be combined or executed in a manner different from that illustrated and described herein.

[0125] The various illustrative logical blocks, modules, and circuits mentioned herein may be implemented or executed by a processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or a combination of these designs to implement the functions described in the present disclosure. The processor may be a microprocessor, but as an alternative, the processor may also be another form of processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, for example, a combination of a DSP and a microprocessor, a plurality of microprocessors, a combination of one or more microprocessors and a DSP core, or any other such configuration.

[0126] The steps of the method or algorithm described in the present invention may be directly embodied in hardware, or may be embodied by a software module executed by a processor, or may be embodied by a combination of the two. The software module may be stored in a random access memory (RAM), a flash memory, a read-only memory (ROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a register, a hard disk, a removable disk, or a CD-ROM. An exemplary storage medium is coupled to a processor so that the processor can read information from the storage medium and write information to the storage medium. Alternatively, the storage medium may be an integral part of the processor. The processor and the storage medium may be integrated in an application-specific integrated circuit (ASIC). The ASIC may be located in a user terminal. Alternatively, the processor and the storage medium may be located in a user terminal as separate components.

[0127] The functions described in one or more exemplary designs may be implemented by hardware, software, firmware, or any combination thereof. If implemented by software, these functions may be stored or transmitted as one or more instructions or codes on a computer-readable medium. Computer-readable media include all media that facilitate the transfer of computer programs from one location to another, including both computer storage media and communication media. Computer-readable storage media may be any medium that can be accessed by a computer. By way of example and not limitation, these computer-readable media may include random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), optical disk-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to carry or store the required program code means, in the form of instructions or data structures, and can be accessed by a computer or processor. In addition, a connection may also be appropriately referred to as a computer-readable medium. For example, if the software is transmitted via a website, server, or other remote source using a coaxial cable, fiber optic cable, twisted pair, or digital subscriber line (DSL), then the coaxial cable, fiber optic cable, twisted pair, or DSL is also included in the definition of the medium. As used herein, "disk" and "disc" include compact disk (CD), laser disk, optical disk, digital versatile disk (DVD), hard disk, solid state disk and Blu-ray disk, where disks generally reproduce data magnetically, while discs reproduce data optically using lasers. Combinations of the above should also be included within the scope of computer-readable media.

[0128] The above description and examples provide a complete description of the structure and use of illustrative embodiments. Although some of the above examples are described in a particular level of detail, or with reference to one or more separate examples, those skilled in the art may make many modifications to the disclosed embodiments without departing from the scope of the invention. Therefore, the various illustrative embodiments of the method and system are not limited to the specific forms disclosed. Instead, they include all modifications and alternatives that fall within the scope of the claims, and other examples may include some or all of the features of the illustrated examples. For example, some elements may be omitted, or they may be combined into a unit structure, and / or the connecting parts may be replaced. In addition, when appropriate, some aspects of any of the examples described above may be combined with some aspects of any of the other examples described to form further examples with similar or different properties and / or functions, and to solve the same or different problems. Similarly, it should be understood that the benefits and advantages described above may be related to one embodiment or to several embodiments.

[0129] Although various aspects of the present disclosure and their advantages have been described in detail, it should be understood that changes, substitutions and modifications may be made herein without departing from the spirit of the disclosure as defined by the appended claims. In addition, the scope of the present application is not intended to be limited to the specific embodiments of the processes, machines, manufactures, material compositions, means, methods and steps described in the specification. As will be readily understood by those of ordinary skill in the art from this disclosure, processes, machines, manufactures, material compositions, means, methods or steps currently existing or developed in the future that perform substantially the same functions or achieve substantially the same results as the corresponding embodiments described herein may be utilized in accordance with the present disclosure. Therefore, the appended claims are intended to include such processes, machines, manufactures, material compositions, means, methods or steps within their scope.

Claims

1. A method for automatic visual testing of a graphical user interface design, characterized in that: include: receiving a candidate data object and a reference data object, each of said data objects representing a graphical user interface design; Using optical character recognition, converting the candidate data object and the reference data object into a candidate machine-encoded text object and a reference machine-encoded text object, respectively, performing text and layout feature comparison, and generating a first similarity score; Using an object mask to segment the candidate machine-encoded text object and the reference machine-encoded text object into a candidate pixel cluster set and a reference pixel cluster set, respectively; Generating a candidate image color histogram and a reference image color histogram based on the pixel distribution of the candidate pixel cluster set and the reference pixel cluster set, respectively, comparing the color distribution between the candidate image color histogram and the reference image color histogram, and generating a second similarity score; Generate a candidate embedding vector and a reference embedding vector based on the candidate pixel cluster set and the reference pixel cluster set by an image embedding technique to calculate a third similarity score, wherein the third similarity score is a cosine similarity score between the candidate embedding vector and the reference embedding vector; Locating the candidate set of pixel clusters and sub-clusters of the reference set of pixel clusters, performing template matching between the candidate data object and the reference data object, and generating a fourth similarity score; Calculating and extracting a set of candidate descriptor objects and a set of reference descriptor objects based on local intensity extreme values ​​of the candidate pixel cluster set and the reference pixel cluster set for comparison and generating a fifth similarity score, each of the descriptor objects representing a feature point in the data object, the comparison comprising comparing each descriptor object of the candidate descriptor object set with each descriptor object of the reference descriptor object set; converting the candidate pixel cluster set and the reference pixel cluster set into grayscale format to compare brightness and contrast between each pixel of the candidate pixel cluster set and each pixel of the reference pixel cluster set to generate a sixth similarity score; providing the first, second, third, fourth, fifth, and sixth similarity scores to a trained machine learning model data architecture for scanning and comparison to a set of passing thresholds; as well as An output structure data object is generated based on the results of comparison between the first, second, third, fourth, fifth and sixth similarity scores and the passed threshold set, wherein the output structure data object includes a list of ordered data pairs, wherein each ordered data pair includes a candidate pixel cluster in the candidate pixel cluster set and a reference pixel cluster in the corresponding reference pixel cluster set, wherein the candidate pixel cluster and its corresponding reference pixel cluster are detected by the trained machine learning model data structure as differences between the candidate data object and the reference data object.

2. The method according to claim 1, characterized in that The method further comprises: inputting the reference data object and the output structure data object into a system for automatically generating user interface rendering code to generate a recommended text instruction set to generate an improved reference data object; The recommended text instruction set is transmitted to the development and testing environment for packaging and execution to generate user interface visual elements.

3. The method according to claim 2, characterized in that The method further comprises: Obtaining a similarity score set of the improved reference data objects; Setting a first threshold for the set of similarity scores; determining whether a similarity score in the set of similarity scores reaches or exceeds the first threshold; and If the similarity score in the similarity score set reaches or exceeds the first threshold, the recommended text instruction set is automatically transmitted to a development and testing environment to be packaged and executed to generate a user interface visual element.

4. The method according to claim 3, characterized in that The method further comprises: Setting a second threshold for the set of similarity scores; determining whether a similarity score in the set of similarity scores reaches or exceeds the second threshold; and If the similarity score in the similarity score set reaches or exceeds the second threshold, the recommended text instruction set is automatically deployed to a production server accessible to multiple users.

5. The method according to claim 2, characterized in that The method further comprises: Encapsulating the recommended text instruction set into a machine language instruction set; and The set of machine language instructions is output to a user to re-implement the user interface visual elements.

6. The method according to claim 5, characterized in that The method further comprises: linking the set of machine language instructions into an executable binary file, wherein the executable binary file is an aggregation of the set of machine language instructions; and The executable binary is output to a user to re-implement the user interface visual elements.

7. The method according to claim 6, characterized in that The method further comprises: Running the executable binary file to render a graphical user interface; and Output the rendered graphical user interface to the user to re-implement the user interface visual elements.

8. The method according to claim 4, characterized in that The method further comprises: Setting a third threshold for the similarity score set; determining whether a similarity score in the similarity score set reaches the third threshold; and If the similarity score in the similarity score set does not reach the third threshold, the recommended text instruction set is automatically discarded.

9. The method according to claim 1, characterized in that The using the object mask to segment the candidate machine-encoded text object and the reference machine-encoded text object into a candidate pixel cluster set and a reference pixel cluster set respectively comprises: Applying an object mask to the candidate machine-encoded text object and the reference machine-encoded text object, the text objects in the candidate data object and the reference data object are replaced with one or more white pixel clusters.

10. The method according to claim 9, characterized in that The using the object mask to segment the candidate machine-encoded text object and the reference machine-encoded text object into a candidate pixel cluster set and a reference pixel cluster set respectively further comprises: Applying an object mask to the candidate machine-encoded text object and the reference machine-encoded text object, replacing the graphic symbol objects in the candidate data object and the reference data object with one or more white pixel clusters.

11. A system for automatic visual testing of a graphical user interface design, characterized in that: include: Test module, which is configured as: receiving a candidate data object and a reference data object, each of said data objects representing a graphical user interface design; An optical character recognition module configured to: receiving the candidate data object and the reference data object; Using optical character recognition, converting the candidate data object and the reference data object into a candidate machine-encoded text object and a reference machine-encoded text object, respectively, performing text and layout feature comparison, and generating a first similarity score; Using an object mask to segment the candidate machine-encoded text object and the reference machine-encoded text object into a candidate pixel cluster set and a reference pixel cluster set, respectively; The test module is further configured to: Generating a candidate image color histogram and a reference image color histogram based on the pixel distribution of the candidate pixel cluster set and the reference pixel cluster set, respectively, comparing the color distribution between the candidate image color histogram and the reference image color histogram, and generating a second similarity score; Generate a candidate embedding vector and a reference embedding vector based on the candidate pixel cluster set and the reference pixel cluster set by an image embedding technique to calculate a third similarity score, wherein the third similarity score is a cosine similarity score between the candidate embedding vector and the reference embedding vector; Locating the candidate set of pixel clusters and sub-clusters of the reference set of pixel clusters, performing template matching between the candidate data object and the reference data object, and generating a fourth similarity score; Calculating and extracting a set of candidate descriptor objects and a set of reference descriptor objects based on local intensity extreme values ​​of the candidate pixel cluster set and the reference pixel cluster set for comparison and generating a fifth similarity score, each of the descriptor objects representing a feature point in the data object, the comparison comprising comparing each descriptor object of the candidate descriptor object set with each descriptor object of the reference descriptor object set; converting the candidate pixel cluster set and the reference pixel cluster set into grayscale format to compare brightness and contrast between each pixel of the candidate pixel cluster set and each pixel of the reference pixel cluster set to generate a sixth similarity score; providing the first, second, third, fourth, fifth and sixth similarity scores to a rules module for scanning and comparison with a set of passing thresholds; as well as The rules module is configured as follows: An output structure data object is generated based on the results of comparison between the first, second, third, fourth, fifth and sixth similarity scores and the passed threshold set, and the output structure data object includes a list of ordered data pairs, wherein each of the ordered data pairs includes a candidate pixel cluster in the candidate pixel cluster set and a reference pixel cluster in the corresponding reference pixel cluster set, wherein the candidate pixel cluster and its corresponding reference pixel cluster are detected by the trained machine learning model data structure as differences between the candidate data object and the reference data object.

12. The system according to claim 11, characterized in that The rule module is further configured to: inputting the reference data object and the output structure data object into a system for automatically generating user interface rendering code to generate a recommended text instruction set to generate an improved reference data object; The recommended text instruction set is transmitted to the development and testing environment for packaging and execution to generate user interface visual elements.

13. The system of claim 12, wherein: The rule module is further configured to: Obtaining a similarity score set of the improved reference data objects; Setting a first threshold for the set of similarity scores; determining whether a similarity score in the similarity score set reaches or exceeds the first threshold; as well as If the similarity score in the similarity score set reaches or exceeds the first threshold, the recommended text instruction set is automatically transmitted to a development and testing environment to be packaged and executed to generate a user interface visual element.

14. The system of claim 13, wherein: The rule module is further configured to: Setting a second threshold for the set of similarity scores; determining whether a similarity score in the set of similarity scores reaches or exceeds the second threshold; and If the similarity score in the similarity score set reaches or exceeds the second threshold, the recommended text instruction set is automatically deployed to a production server accessible to multiple users.

15. The system of claim 12, wherein: The rule module is further configured to: Encapsulating the recommended text instruction set into a machine language instruction set; and The set of machine language instructions is output to a user to re-implement the user interface visual elements.

16. The system of claim 15, wherein: The rule module is further configured to: linking the set of machine language instructions into an executable binary file, wherein the executable binary file is an aggregation of the set of machine language instructions; and The executable binary is output to a user to re-implement the user interface visual elements.

17. The system of claim 16, wherein: The rule module is further configured to: Running the executable binary file to render a graphical user interface; and Output the rendered graphical user interface to the user to re-implement the user interface visual elements.

18. The system of claim 14, wherein: The rule module is further configured to: Setting a third threshold for the similarity score set; determining whether a similarity score in the similarity score set reaches the third threshold; and If the similarity score in the similarity score set does not reach the third threshold, the recommended text instruction set is automatically discarded.

19. The system of claim 11, wherein: The operation of the optical character recognition module using the object mask to segment the candidate machine-encoded text object and the reference machine-encoded text object into a candidate pixel cluster set and a reference pixel cluster set respectively further comprises: Applying an object mask to the candidate machine-encoded text object and the reference machine-encoded text object, the text objects in the candidate data object and the reference data object are replaced with one or more white pixel clusters.

20. The system of claim 19, wherein: The operation of the optical character recognition module using the object mask to segment the candidate machine-encoded text object and the reference machine-encoded text object into a candidate pixel cluster set and a reference pixel cluster set respectively further comprises: Applying an object mask to the candidate machine-encoded text object and the reference machine-encoded text object, replacing the graphic symbol objects in the candidate data object and the reference data object with one or more white pixel clusters.

21. A computer device comprising a memory, a processor and a computer program stored therein and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method for automatically visually testing a graphical user interface design as claimed in any one of claims 1 to 10 are implemented.

22. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method for automatic visual testing of a graphical user interface design according to any one of claims 1 to 10 are implemented.

Images