User interface acceptance method and apparatus

Abstract

The present application relates to the field of computer, and provide a kind of user interface acceptance method and device.Method includes: obtaining the user interface image to be processed and the template image corresponding to the user interface to be processed;The user interface image to be processed and the template image are input into difference comparison model, and the difference annotation image of the user interface image to be processed and the template image is obtained;Based on the difference annotation image, acceptance is carried out, and the acceptance result of the user interface to be processed is obtained.Through difference comparison model, pixel-level matching of the user interface image to be processed and the template image is realized, difference annotation image is output and automatic acceptance is carried out, and the automation of UI acceptance whole process is realized.

Description

Technical Field

[0001] This invention relates to the field of computers, and more particularly to a method and apparatus for user interface acceptance. Background Technology

[0002] With the deepening of enterprise digitalization and the rapid iteration of application development, UI acceptance has become a key step in the application development process in order to ensure that the developed user interface (UI) is as close as possible to the UI design scheme given in the design draft.

[0003] UI acceptance involves comparing the sample artwork with the developed web UI. Inconsistencies often arise in page layout, font size, line quality, and color. Failure to promptly identify these differences negatively impacts user experience. Traditional manual comparison is inefficient and prone to missing discrepancies. This has led to the development of automated UI acceptance platforms that support large-scale intelligent acceptance. However, due to the diverse styles of sample artworks, making them difficult to exhaustively list, and the presence of similar or repetitive styles between sample artworks and web UIs, existing UI acceptance platforms struggle to effectively locate abnormal data and characterize common features between the two images, easily leading to mismatches. Summary of the Invention

[0004] This invention aims to at least solve one of the technical problems existing in related technologies. To this end, this invention proposes a user interface acceptance method, which achieves pixel-level matching between the user interface image to be processed and the template image through a difference comparison model, outputs a difference-annotated image, and performs automated acceptance, thereby automating the entire UI acceptance process.

[0005] The present invention also proposes a user interface acceptance device.

[0006] The present invention also proposes an electronic device.

[0007] The present invention also proposes a non-transitory computer-readable storage medium.

[0008] The present invention also proposes a computer program product.

[0009] A user interface acceptance method according to a first aspect of the present invention includes:

[0010] Obtain the user interface image to be processed and the template image corresponding to the user interface to be processed;

[0011] The user interface image to be processed and the template image are input into the difference comparison model to obtain the difference-annotated image between the user interface image to be processed and the template image.

[0012] Acceptance is performed based on the difference-annotated image to obtain the acceptance result of the user interface to be processed.

[0013] The user interface acceptance method according to embodiments of the present invention can automatically acquire user interface images and template images to be processed, input them into a difference comparison model to obtain feature groups, and perform difference localization. Compared with the existing technology that matches based on the similarity of feature descriptors, this method takes into account the overall offset and scaling of the sample and the web page UI, and can find common features between the two images, improving the matching accuracy when the pages have similar or repetitive styles. Furthermore, after difference annotation based on the matching results, subsequent automated acceptance and the acquisition of acceptance results can be facilitated.

[0014] According to an embodiment of the present invention, the difference comparison model includes a feature extraction module, a feature matching module, a difference localization module, and a labeling module; the step of inputting the user interface image to be processed and the template image into the difference comparison model to obtain a difference-labeled image between the user interface image to be processed and the template image includes:

[0015] The user interface image to be processed and the template image are input into the feature extraction module. The feature extraction module performs feature extraction and feature classification on the user interface image to be processed and the template image respectively to obtain the first feature group and the second feature group.

[0016] The first feature group and the second feature group are input into the feature matching module. Through the preset attention mechanism graph neural network in the feature matching module, the transformation matrix of the user interface image to be processed and the template image and the registration image of the user interface image to be processed are obtained.

[0017] The registered image and the template image are input into the difference localization module for comparison to obtain the difference coordinates;

[0018] The transformation matrix and the difference coordinates are passed to the annotation module to obtain the difference-annotated image.

[0019] According to an embodiment of the present invention, the step of performing feature extraction and feature classification on the user interface image to be processed and the template image respectively to obtain the first feature group and the second feature group includes:

[0020] The user interface image to be processed and the template image are used to extract features through a shared coding structure to obtain the original feature map of the user interface image to be processed and the original feature map of the template image.

[0021] The original feature map of the user interface image to be processed is classified into features through the feature point detection branch and the descriptor detection branch to obtain the first feature group. The original feature map of the template image is classified into features through the feature point detection branch and the descriptor detection branch to obtain the second feature group.

[0022] According to an embodiment of the present invention, obtaining the transformation matrix of the user interface image to be processed and the template image, as well as the registration image of the user interface image to be processed, through a preset attention mechanism graph neural network in the feature matching module, includes:

[0023] Encode the first feature group and the second feature group respectively to obtain the corresponding first feature matching vector and second feature matching vector;

[0024] The first feature matching vector and the second feature matching vector are input into the preset attention mechanism graph neural network to obtain the target feature matching matrix;

[0025] Based on the target feature matching matrix, the transformation matrix of the user interface image to be processed and the template image are obtained, as well as the registration image of the user interface image to be processed. (Template image)

[0026] According to an embodiment of the present invention, the step of obtaining the difference-annotated image by transferring the transformation matrix and the difference coordinates to the annotation module includes:

[0027] Based on the transformation matrix, the overall difference between the user interface image to be processed and the template image is obtained;

[0028] The difference coordinates and the overall difference amount are labeled on the registered image to obtain the difference-labeled image.

[0029] According to one embodiment of the present invention, obtaining the acceptance result of the user interface to be processed based on the difference-annotated image includes:

[0030] Based on the difference-annotated image, the defect score of the user interface image to be processed is calculated according to preset rules, and an acceptance report of the user interface image to be processed is generated.

[0031] Based on the acceptance report of the user interface image to be processed, the acceptance result of the user interface to be processed is determined.

[0032] According to an embodiment of the present invention, the step of obtaining the user interface image to be processed and the template image corresponding to the user interface to be processed further includes:

[0033] Obtain the initial user interface image to be processed and the initial template image corresponding to the initial user interface to be processed;

[0034] Overlap verification is performed on the initial user interface image to be processed and the initial template image to obtain the overlap deviation of the initial user interface image to be processed relative to the initial template image;

[0035] The initial overlap deviation is repaired to obtain the user interface image to be processed and the template image.

[0036] A user interface acceptance device according to a second aspect of the present invention includes:

[0037] The acquisition module is used to acquire the user interface image to be processed and the template image corresponding to the user interface to be processed;

[0038] The difference annotation module is used to input the user interface image to be processed and the template image into the difference comparison model to obtain the difference-annotated image between the user interface image to be processed and the template image.

[0039] The acceptance module is used to perform acceptance based on the difference-annotated image and obtain the acceptance result of the user interface to be processed.

[0040] The user interface acceptance device according to embodiments of the present invention can automatically acquire user interface images and template images to be processed, input them into a difference comparison model to obtain feature groups, and perform difference localization. Compared with the existing technology that matches based on the similarity of feature descriptors, this device takes into account the overall offset and scaling of the sample and the web page UI, and can find common features between the two images, thus improving the accuracy of matching when the pages have similar or repetitive styles. Furthermore, after difference annotation based on the matching results, subsequent automated acceptance and the acquisition of acceptance results can be facilitated.

[0041] An electronic device according to a third aspect of the present invention includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the user interface acceptance method described above.

[0042] According to a fourth aspect of the present invention, a non-transitory computer-readable storage medium is provided thereon storing a computer program that, when executed by a processor, implements the steps of the user interface acceptance method described above.

[0043] A computer program product according to a fifth aspect of the present invention includes a computer program that, when executed by a processor, implements the steps of the above-described user interface acceptance method.

[0044] The above-mentioned one or more technical solutions in the embodiments of the present invention have at least one of the following technical effects: the difference comparison model can automatically perform AI acceptance analysis on the differences between the user interface to be processed and the design draft online, and the generated difference annotation image can mark the places where the web page element position, font size, size and color of the user interface to be processed are deviated, generating a detailed difference annotation image, making various UI defects clear at a glance.

[0045] Furthermore, pre-acceptance was achieved by overlapping verification of the user interface to be processed and the design draft. This self-testing method can identify obvious UI defects in advance and fix them in a timely manner before conducting accurate UI acceptance of the delivery difference comparison model.

[0046] Furthermore, due to the diverse and exhaustive nature of template images, compared to the shortcomings of positive sample algorithms in existing technologies, the difference comparison model of this invention can effectively identify differences between different user interfaces to be processed and corresponding samples without collecting a large amount of sample data, and has strong generalization ability.

[0047] Furthermore, when accepting images with discrepancies, the system can intelligently analyze the level of UI defects based on preset rules for different web pages, classify and score the defects, and generate a detailed UI acceptance report.

[0048] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0049] To more clearly illustrate the technical solutions in the embodiments of the present invention or related technologies, the drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0050] Figure 1 This is a flowchart illustrating the user interface acceptance method provided in an embodiment of the present invention;

[0051] Figure 2 This is a schematic diagram of the structure of the difference comparison model provided in the embodiment of the present invention;

[0052] Figure 3 This is a schematic diagram of the feature extraction module provided in an embodiment of the present invention;

[0053] Figure 4 This is a schematic diagram of the feature matching module provided in an embodiment of the present invention;

[0054] Figure 5 This is a schematic diagram of the pre-acceptance process provided in an embodiment of the present invention;

[0055] Figure 6 This is a flowchart illustrating the fully automated user interface acceptance method provided in this embodiment of the invention.

[0056] Figure 7 This is a schematic diagram of the user interface acceptance device provided in an embodiment of the present invention;

[0057] Figure 8 This is a schematic diagram of a mobile terminal for differentially annotated images provided in an embodiment of the present invention;

[0058] Figure 9 This is a schematic diagram of the structure of the electronic device provided in an embodiment of the present invention. Detailed Implementation

[0059] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0060] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the embodiments of this application. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Furthermore, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0061] The traditional manual UI acceptance process typically involves: visual analysis -> screenshotting -> screenshot upload -> screenshot annotation -> Excel file with screenshot descriptions of UI defects -> sending to developers via chat. This process is extremely cumbersome and time-consuming. To address the current issues of low efficiency and high cost in UI acceptance, this invention proposes a user interface acceptance method that achieves fully online, intelligent, and refined intelligent inspection. It performs pixel-level reconstruction of differences between the UI to be processed and the sample, such as... Figure 1 As shown, the method includes at least the following steps:

[0062] Step 101: Obtain the user interface image to be processed and the template image corresponding to the user interface to be processed;

[0063] Step 102: Input the user interface image to be processed and the template image into the difference comparison model to obtain the difference annotation image between the user interface image to be processed and the template image;

[0064] Step 103 of the template image: Perform acceptance based on the difference-annotated image to obtain the acceptance result of the user interface to be processed.

[0065] Regarding step 101, it should be noted that the user interface image to be processed in this embodiment of the invention, i.e., the web page UI to be processed on the terminal, can be a mobile UI such as a mobile application interface, or a PC web page interface. The terminal can be, for example, a personal computer, mobile phone, tablet computer, laptop computer, e-book reader, smart voice interaction device, smart home appliance, or in-vehicle terminal. The template image is the original design drawing of each user interface, created by the designer on the design side and stored on the development side.

[0066] Regarding step 102, it should be noted that the difference comparison model is a neural network model trained with a certain amount of sample data, possessing the functions of feature extraction, feature matching, difference localization, and annotation. This model can automatically extract and analyze the differences between the webpage to be inspected and the design draft online, output difference-annotated images, and perform AI acceptance analysis, thus solving the problem of difference identification in UI acceptance.

[0067] Specifically, during feature extraction, the user interface image to be processed and the template image need to be extracted separately. The first and second feature groups extracted contain various types of features of the two images, such as the layout, size, shape, color difference and brightness of the images. After feature extraction, the feature points of the input image and a complete descriptor can be obtained.

[0068] During feature matching, the input consists of feature points and complete descriptors in the template image and the user interface image to be processed, resulting in a transformation matrix and the transformed registered image, i.e., the target image.

[0069] During difference localization, the input is a template image and a registration image. By comparing them, the mask of the difference can be found, and the difference coordinates of each difference can be obtained accordingly.

[0070] When performing the final difference annotation, the relative pose of the two images can be calculated based on the transformation matrix to obtain the overall transformation amount of the user interface image to be processed relative to the template image. Then, the transformation position of the details can be obtained based on the difference coordinates. Therefore, the final output difference-annotated image can be obtained based on these two feature matching results and difference localization results.

[0071] Regarding step 103, it should be noted that the difference-annotated image includes the differences between the user interface image to be processed and its corresponding template image, such as... Figure 8 As shown, direct acceptance refers to visually describing the defects of the current user interface image to be processed through methods such as classification or scoring, and further obtaining acceptance results. These results can also be fed back to the relevant developers, allowing them to fix the defects based on the acceptance findings and then reapply for UI acceptance.

[0072] The user interface acceptance method of this invention, compared with traditional methods, integrates the analysis, annotation and description processes into the difference comparison model. Through AI algorithms, it can save a lot of acceptance time for acceptance personnel, effectively improve the efficiency of developers and acceptance personnel, enhance UI fidelity, and achieve automated inspection.

[0073] It is understandable that, such as Figure 2 As shown, the difference comparison model includes a feature extraction module, a feature matching module, a difference localization module, and an annotation module. The user interface image to be processed and the template image are input into the difference comparison model to obtain a difference-annotated image between the user interface image to be processed and the template image, including:

[0074] The user interface image to be processed and the template image are input into the feature extraction module. The feature extraction module performs feature extraction and feature classification on the user interface image to be processed and the template image respectively to obtain the first feature group and the second feature group.

[0075] The first feature group and the second feature group are input into the feature matching module. Through the preset attention mechanism graph neural network in the feature matching module, the transformation matrix of the user interface image to be processed and the template image and the registration image of the user interface image to be processed are obtained.

[0076] The registered image and the template image are input into the difference localization module for comparison to obtain the difference coordinates;

[0077] The transformation matrix and the difference coordinate annotation module are used to obtain the difference-annotated image.

[0078] Template image template image template image It should be noted that, as Figure 2As shown, the difference comparison model in this embodiment of the invention is pre-trained using user interface images of samples and corresponding template images. During training, a loss function corresponding to the sample question-and-answer corpus is constructed based on the differences between the sample's webpage UI and the template, and the model parameters of the difference comparison model are updated based on the loss function. The training stops when the updated difference comparison model converges, or when the number of updates reaches a preset number. Since the difference comparison model in this embodiment of the invention trains the network by extracting similar features from two images rather than labeled features, this model has strong generalization ability without requiring manual annotation of a large amount of sample data in the early stage.

[0079] Specifically, the process of the difference comparison model processing and outputting difference-annotated images includes:

[0080] Step 201: Input the user interface image to be processed (web page UI image) and the sample into the feature extraction module to obtain the first feature group of the web page UI image and the second feature group of the sample, respectively. Each feature group includes the corresponding feature map and feature descriptor.

[0081] Step 202: Input the first feature group and the second feature group into the feature matching module and combine them with the attention algorithm to perform feature matching, so as to obtain the registration image and the transformation matrix;

[0082] Step 203: Input the registered image and sample into the difference localization module, and output the difference coordinate positions;

[0083] Step 204: Input the difference coordinates and transformation matrix into the annotation module to annotate the difference type at each difference coordinate and obtain the difference-annotated image.

[0084] It is understandable that feature extraction and feature classification are performed on the user interface image to be processed and the template image respectively to obtain the first feature group and the second feature group, including:

[0085] The user interface image to be processed and the template image are used to extract features through a shared coding structure to obtain the original feature map of the user interface image to be processed and the original feature map of the template image.

[0086] The original feature map of the user interface image to be processed is classified into features through the feature point detection branch and the descriptor detection branch to obtain the first feature group. The original feature map of the template image is classified into features through the feature point detection branch and the descriptor detection branch to obtain the second feature group.

[0087] It should be noted that, as Figure 3As shown, the feature extraction module includes a shared coding structure, a feature point detection branch, and a descriptor detection branch. The shared coding structure can reduce the dimensionality of the input image and extract features; this structure includes convolutional layers and pooling layers. Specifically, by setting three max-pooling layers, an 8x downsampled feature map is obtained.

[0088] The feature map output from the shared coding structure is processed by a feature point detection branch and a descriptor detection branch. The feature point detection branch obtains the probability that a pixel is a feature point and then maps it to a feature map of the same size as the input image, obtaining the feature distribution of interest. The output feature map can include block-like attribute features such as image layout, size, and shape. The feature descriptor detection branch performs bicubic interpolation to obtain a complete descriptor and uses L2 normalization to obtain the feature descriptor for each unit image. The feature descriptor includes attribute features such as color difference, color temperature, and brightness. All feature maps and feature descriptors of the user interface image to be processed constitute the first feature group, and all feature maps and feature descriptors of the template image constitute the second feature group.

[0089] Understandably, the transformation matrix of the user interface image to be processed and the template image, as well as the registration image and template image of the user interface image to be processed, are obtained through the pre-set attention mechanism graph neural network in the feature matching module, including:

[0090] Encode the first feature group and the second feature group respectively to obtain the corresponding first feature matching vector and second feature matching vector;

[0091] The first feature matching vector and the second feature matching vector are input into a preset attention mechanism graph neural network to obtain the target feature matching matrix;

[0092] Based on the target feature matching matrix, the transformation matrix of the user interface image to be processed and the template image, as well as the registration image of the user interface image to be processed, are obtained.

[0093] It should be noted that, as Figure 4 As shown, the feature matching module takes as input the feature maps and complete feature descriptors of the sample template and the target web page UI image, and outputs the affine transformation matrix and the transformed registration image. The structure of the feature matching module includes a graph neural network with an attention mechanism and an optimization matching layer.

[0094] The pre-defined attention mechanism of the graph neural network can encode the feature map and feature descriptor corresponding to each image into a feature matching vector, and then feed it into a neural network based on self-attention and cross-attention. Through attention aggregation, the feature points of the sample and the web page UI image are constructed into a complete graph, whose nodes are the feature points of the two images. The attention mechanism is used to enhance the matching performance.

[0095] Assuming the user interface image to be processed is image A, and the template image is image B, the workflow after the feature matching vectors corresponding to images A and B are input into the feature matching module includes the following steps:

[0096] Step 301: Calculate the i-th element of image A in the n-th layer of the graph neural network, represented as... Then, the feature point representation of image A at layer n+1 is... As shown in Equation 1:

[0097]

[0098] Where MLP represents a multilayer perceptron, || represents a cascade operation, and p ε→i This represents the aggregated weights obtained by weighting self-attention and cross-attention, with the self-attention and cross-attention weights being updated alternately.

[0099] Step 302: Obtain the score matrix S by calculating the inner product of the matching descriptors of image A and image B. i,j ,like

[0100] As shown in Equation 2:

[0101]

[0102] in, and These are the matching descriptors for the i-th element of image A and the j-th element of image B, respectively. T is the optimal feature matching matrix.

[0103] Step 303: Maximize the overall score ∑ i,j S i,j W i,j Calculate the target feature matching matrix W i,j .

[0104] Step 304: Based on the target feature matching matrix W i,j Finally, the relative poses of the sample image and the web page UI image are calculated to obtain the transformation matrix and the registered image after registration with the sample image.

[0105] It is understandable that by combining the transformation matrix and the difference coordinate annotation module, a difference-annotated image is obtained, including:

[0106] Based on the transformation matrix, the overall difference between the user interface image to be processed and the template image is obtained;

[0107] The difference coordinates and overall difference values ​​are labeled on the registered image to obtain a difference-labeled image.

[0108] Regarding the template image, it's important to note that the transformation matrix provides the overall difference between the webpage UI image and the sample image, including the overall relative translation, scaling, and rotation. Simultaneously, in the difference localization module, calculating the difference coordinates involves performing a bitwise subtraction between the template image and the registered image to obtain a mask of the difference. Then, the coordinates of the minimum bounding rectangle and the contour coordinates of the difference points can be calculated from the mask, thus locating the positions of the difference points, i.e., the difference coordinates. This difference localization module can identify differences caused by variations in component styles, font sizes, line thickness, etc. Finally, the difference-annotated image can also display areas where the positions, font sizes, dimensions, and colors of different webpage elements deviate from the template image.

[0109] Understandably, acceptance testing is conducted based on the difference-annotated images, resulting in the acceptance results for the user interface to be processed, including:

[0110] Based on the difference-annotated images, the defect score of the user interface image to be processed is calculated according to preset rules, and an acceptance report of the user interface image to be processed is generated.

[0111] Based on the acceptance report of the user interface image to be processed, determine the acceptance result of the user interface to be processed.

[0112] It's important to note that the preset rules are a rule engine built upon the correspondence between the types and degrees of differences. Different differences on the difference-annotated image correspond to different UI defects. By setting specific rule engines for different web pages, the engine can intelligently analyze the level of UI defects, classify and score them, and generate a detailed UI acceptance report. For web pages with low acceptance report scores, the corresponding developers are notified online, and the acceptance report is automatically sent. After fixing the defects based on the report, the developers reapply for UI acceptance. The entire process achieves closed-loop acceptance, ensuring full automation.

[0113] Specifically, the rules engine can set two threshold values. When the report score is higher than the first threshold, it indicates that the report is compliant and is directly fed back to the user. The first threshold can be set to 80 points. When the report score is lower than the first threshold but higher than the second threshold, it indicates that there are some issues in the report. In this case, an alert should be triggered to the user, and the two compared images should be checked for matching. The second threshold can be set to 60 points. When the report score is lower than the second threshold, it indicates that the report is abnormal and unqualified, and the file needs to be re-uploaded for evaluation.

[0114] It is understandable that after obtaining the user interface image to be processed and the corresponding template image, the process also includes:

[0115] Overlap verification is performed on the user interface image to be processed and the template image to obtain the overlap deviation of the user interface image to be processed relative to the template image;

[0116] After correcting the overlap deviation, the corrected user interface image to be processed and the template image are input into the difference comparison model to obtain the difference-annotated image between the user interface image to be processed and the template image.

[0117] It should be noted that this embodiment of the invention can integrate a pre-acceptance analysis tool on the development side, using a simplified neural network model to perform overlap verification between the design draft and the webpage. Pre-acceptance allows for the early detection and repair of obvious UI defects through self-testing before formal UI acceptance is delivered. Specifically, the pre-acceptance analysis tool can use canvas graphic matching for overlap verification, and can also employ other methods such as... Figure 5 The pre-acceptance tool shown performs a pre-acceptance process. To improve computational efficiency, it can initially and efficiently determine the similarity between two images and obtain the approximate location of the differences. This tool includes a size adjustment module, a grayscale conversion module, a sliding window module, and a structural similarity calculation module.

[0118] The workflow of the pre-acceptance analysis tool includes:

[0119] The input template image and the image to be tested are used. First, the image to be tested is adjusted using the size adjustment module to make the size of the image to be consistent with that of the template image. Then, the grayscale module converts both the image to grayscale to obtain their corresponding grayscale images. A Gaussian weighted function is used as a weighted sliding window. Assuming the w*h image is divided into n*m parts, sub-images with a width of w / n and a height of h / m are obtained, with a horizontal step of w / n and a vertical step of h / m. Using the structural similarity calculation module, the corresponding sub-images of the two images are calculated, including their brightness, contrast, and structural parameters. Brightness is estimated using the average grayscale of the image, contrast is estimated using the standard deviation of the image, and covariance is used as a measure of structural similarity to obtain the local SSIM index results, thus revealing the difference information of each sub-image. Based on the local SSIM results of the n*m ​​sub-images, the average SSIM index is calculated as the structural similarity result for the entire image. Simultaneously, image differences are calculated to highlight the transformed parts of the image, and the differences are initially located through contour extraction and minimum bounding rectangle, as well as the overlap deviation of the user interface image to be processed relative to the template image.

[0120] It is understandable that, such as Figure 6 As shown, this embodiment of the invention also provides a fully automated user interface acceptance method, including:

[0121] Step a: Upload the UI source files of the user interface image to be processed and the template image;

[0122] Step b: Determine if there are any discrepancy annotations on the template image;

[0123] Step c: If there are discrepancies, determine whether the user interface image to be processed and the template image match.

[0124] Step d: If a match is found, AI calculations are performed using the difference comparison model to output a difference-annotated image to the rule engine.

[0125] Step e: The rules engine will calculate the report score of the current difference-annotated image. When the report score is in different score ranges, it will trigger an alert, report an anomaly to the development team, or send a compliance report to the user.

[0126] Step f: If the user interface image to be processed and the template image do not match, report an error and return to step a;

[0127] Step g: If there are no discrepancies, perform overlap verification directly through pre-acceptance and return the corrected image to step d for judgment.

[0128] This invention improves the accuracy of difference identification by adding a pre-acceptance step, which allows for the identification of significant differences in template images even when they are unlabeled, correcting overlap deviations before sending them to the difference comparison model for identification.

[0129] The user interface acceptance device provided by the present invention is described below. The user interface acceptance device described below can be referred to in correspondence with the user interface acceptance method described above. Figure 7 As shown, the application publishing device includes:

[0130] The acquisition module 701 is used to acquire the user interface image to be processed and the template image corresponding to the user interface to be processed.

[0131] The difference annotation module 702 is used to input the user interface image to be processed and the template image into the difference comparison model to obtain the difference annotation image between the user interface image to be processed and the template image.

[0132] The acceptance module 703 is used to perform acceptance based on the difference-annotated image and obtain the acceptance result of the user interface to be processed.

[0133] The user interface acceptance device of this invention can automatically acquire user interface images and template images to be processed, input them into a difference comparison model to obtain feature groups, and perform difference localization. Compared with the existing technology that matches based on the similarity of feature descriptors, this device takes into account the overall offset and scaling of the sample and the web page UI, and can find common features between the two images, thus improving the accuracy of matching when the pages have similar or repetitive styles. Furthermore, after difference annotation based on the matching results, subsequent automated acceptance and the acquisition of acceptance results can be facilitated.

[0134] It is understandable that the difference comparison model includes a feature extraction module, a feature matching module, a difference localization module, and a labeling module;

[0135] The feature extraction module is used to extract and classify features from the user interface image and the template image to be processed, respectively, to obtain the first feature group and the second feature group;

[0136] The feature matching module is used to input the first feature group and the second feature group into a preset attention mechanism graph neural network to obtain the transformation matrix of the user interface image to be processed and the template image, as well as the registration image of the user interface image to be processed.

[0137] The difference localization module is used to compare the registered image with the template image to obtain the difference coordinates;

[0138] The annotation module is used to obtain a differentially annotated image based on the transformation matrix and the difference coordinates.

[0139] It is understandable that feature extraction and feature classification are performed on the user interface image to be processed and the template image respectively to obtain the first feature group and the second feature group, including:

[0140] The user interface image to be processed and the template image are used to extract features through a shared coding structure to obtain the original feature map of the user interface image to be processed and the original feature map of the template image.

[0141] The original feature map of the user interface image to be processed is classified into features through the feature point detection branch and the descriptor detection branch to obtain the first feature group. The original feature map of the template image is classified into features through the feature point detection branch and the descriptor detection branch to obtain the second feature group.

[0142] Understandably, the first and second feature sets are input into a pre-defined attention mechanism graph neural network to obtain the transformation matrix of the user interface image to be processed and the template image, as well as the registration image of the user interface image to be processed, including:

[0143] The first feature group and the second feature group are encoded respectively to obtain the corresponding first feature matching vector and second feature matching vector;

[0144] The first feature matching vector and the second feature matching vector are input into a preset attention mechanism graph neural network to obtain the target feature matching matrix;

[0145] Based on the target feature matching matrix, the transformation matrix of the user interface image to be processed and the template image, as well as the registration image of the user interface image to be processed, are obtained.

[0146] It is understandable that, based on the transformation matrix and difference coordinates, a difference-annotated image is obtained, including:

[0147] Based on the transformation matrix output by the feature matching module, the overall difference between the user interface image to be processed and the template image is obtained;

[0148] The difference coordinates and overall difference values ​​are labeled on the registered image to obtain a difference-labeled image.

[0149] Understandably, acceptance testing is conducted based on the difference-annotated images, resulting in the acceptance results for the user interface to be processed, including:

[0150] Based on the difference-annotated images, the defect score of the user interface image to be processed is calculated according to preset rules, and an acceptance report of the user interface image to be processed is generated.

[0151] Based on the acceptance report of the user interface image to be processed, determine the acceptance result of the user interface to be processed.

[0152] Understandably, the device also includes a pre-acceptance module, which is used for:

[0153] Overlap verification is performed on the user interface image to be processed and the template image to obtain the overlap deviation of the user interface image to be processed relative to the template image;

[0154] After correcting the overlap deviation, the corrected user interface image to be processed and the template image are input into the difference comparison model to obtain the difference-annotated image between the user interface image to be processed and the template image.

[0155] Figure 9 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 9 As shown, the electronic device may include a processor 810, a communications interface 820, a memory 830, and a communication bus 840, wherein the processor 810, the communications interface 820, and the memory 830 communicate with each other via the communication bus 840. The processor 810 can call logical instructions in the memory 830 to execute the following methods:

[0156] Obtain the user interface image to be processed and the corresponding template image of the user interface to be processed;

[0157] Input the user interface image to be processed and the template image into the difference comparison model to obtain the difference-annotated image between the user interface image to be processed and the template image;

[0158] Acceptance is performed based on the differentially labeled images to obtain the acceptance results of the user interface to be processed.

[0159] Furthermore, the logical instructions in the aforementioned memory 830 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to related technologies, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0160] On the other hand, embodiments of the present invention disclose a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium. The computer program includes program instructions, and when the program instructions are executed by a computer, the computer is able to perform the methods provided in the above-described method embodiments, such as including:

[0161] Obtain the user interface image to be processed and the corresponding template image of the user interface to be processed;

[0162] Input the user interface image to be processed and the template image into the difference comparison model to obtain the difference-annotated image between the user interface image to be processed and the template image;

[0163] Acceptance is performed based on the difference-annotated image to obtain the acceptance result of the user interface to be processed. Furthermore, embodiments of the present invention also provide a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, this computer program is implemented to perform the transmission methods provided in the above embodiments, including, for example:

[0164] Obtain the user interface image to be processed and the corresponding template image of the user interface to be processed;

[0165] Input the user interface image to be processed and the template image into the difference comparison model to obtain the difference-annotated image between the user interface image to be processed and the template image;

[0166] Acceptance is performed based on the differentially labeled images to obtain the acceptance results of the user interface to be processed.

[0167] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0168] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the parts that contribute to the related technology, can be embodied in the form of software products. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods of various embodiments or some parts of embodiments.

[0169] Finally, it should be noted that the above embodiments are only for illustrating the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the embodiments, those skilled in the art should understand that various combinations, modifications, or equivalent substitutions of the technical solutions of the present invention do not depart from the spirit and scope of the technical solutions of the present invention and should all be covered within the scope of the present invention.

Claims

1. A user interface acceptance method, characterized in that, include: Obtain the user interface image to be processed and the template image corresponding to the user interface to be processed; The user interface image to be processed and the template image are input into the difference comparison model to obtain the difference-annotated image between the user interface image to be processed and the template image. Based on the difference-annotated image, the acceptance result of the user interface to be processed is obtained; The step of obtaining the user interface image to be processed and the template image corresponding to the user interface to be processed also includes: Obtain the initial user interface image to be processed and the initial template image corresponding to the initial user interface to be processed; Overlap verification is performed on the initial user interface image to be processed and the initial template image to obtain the overlap deviation of the initial user interface image to be processed relative to the initial template image; The overlap deviation is repaired to obtain the user interface image to be processed and the template image; The overlap verification includes adjusting the size of the initial user interface image to be processed and the initial template image to be the same, converting the initial user interface image to be processed and the initial template image to grayscale to obtain two corresponding grayscale images, dividing the two grayscale images into multiple sub-images based on a sliding window, calculating the local SSIM index between the corresponding sub-images to obtain the structural similarity between the two grayscale images, and calculating the image difference and obtaining the overlap deviation by contour extraction and minimum bounding rectangle positioning difference.

2. The user interface acceptance method according to claim 1, characterized in that, The difference comparison model includes a feature extraction module, a feature matching module, a difference localization module, and an annotation module; the step of inputting the user interface image to be processed and the template image into the difference comparison model to obtain a difference-annotated image between the user interface image to be processed and the template image includes: The user interface image to be processed and the template image are input into the feature extraction module. The feature extraction module performs feature extraction and feature classification on the user interface image to be processed and the template image respectively to obtain a first feature group and a second feature group. The first feature group and the second feature group are input into the feature matching module. Through the preset attention mechanism graph neural network in the feature matching module, the transformation matrix of the user interface image to be processed and the template image and the registration image of the user interface image to be processed are obtained. The registered image and the template image are input into the difference localization module for comparison to obtain the difference coordinates; The transformation matrix and the difference coordinates are input into the annotation module to obtain the difference-annotated image.

3. The user interface acceptance method according to claim 2, characterized in that, The step of performing feature extraction and feature classification on the user interface image to be processed and the template image respectively to obtain a first feature group and a second feature group includes: The user interface image to be processed and the template image are used to extract features through a shared coding structure to obtain the original feature map of the user interface image to be processed and the original feature map of the template image. The original feature map of the user interface image to be processed is classified into features through the feature point detection branch and the descriptor detection branch to obtain the first feature group. The original feature map of the template image is classified into features through the feature point detection branch and the descriptor detection branch to obtain the second feature group.

4. The user interface acceptance method according to claim 2, characterized in that, The step of obtaining the transformation matrix of the user interface image to be processed and the template image, as well as the registration image of the user interface image to be processed, through the preset attention mechanism graph neural network in the feature matching module, includes: Encode the first feature group and the second feature group respectively to obtain the corresponding first feature matching vector and second feature matching vector; The first feature matching vector and the second feature matching vector are input into the preset attention mechanism graph neural network to obtain the target feature matching matrix; Based on the target feature matching matrix, the transformation matrix of the user interface image to be processed and the template image, as well as the registration image of the user interface image to be processed, are obtained.

5. The user interface acceptance method according to claim 2, characterized in that, The step of inputting the transformation matrix and the difference coordinates into the annotation module to obtain the difference-annotated image includes: Based on the transformation matrix, the overall difference between the user interface image to be processed and the template image is obtained; The difference coordinates and the overall difference amount are labeled on the registered image to obtain the difference-labeled image.

6. The user interface acceptance method according to any one of claims 1 to 5, characterized in that, The process of obtaining the acceptance result of the user interface to be processed based on the difference-annotated image includes: Based on the difference-annotated image, the defect score of the user interface image to be processed is calculated according to preset rules, and an acceptance report of the user interface image to be processed is generated. Based on the acceptance report of the user interface image to be processed, the acceptance result of the user interface to be processed is determined.

7. A user interface acceptance device, characterized in that, include: The acquisition module is used to acquire the user interface image to be processed and the template image corresponding to the user interface to be processed; The step of obtaining the user interface image to be processed and the template image corresponding to the user interface to be processed further includes: obtaining the initial user interface image to be processed and the initial template image corresponding to the initial user interface to be processed. The difference annotation module is used to input the user interface image to be processed and the template image into the difference comparison model to obtain the difference-annotated image between the user interface image to be processed and the template image. The acceptance module is used to obtain the acceptance result of the user interface to be processed based on the difference-annotated image; The pre-acceptance module is used to perform overlap verification on the initial user interface image to be processed and the initial template image, obtain the overlap deviation of the initial user interface image to be processed relative to the initial template image, repair the overlap deviation, and obtain the user interface image to be processed and the template image. The overlap verification includes adjusting the size of the initial user interface image to be processed and the initial template image to be the same, converting the initial user interface image to be processed and the initial template image to grayscale to obtain two corresponding grayscale images, dividing the two grayscale images into multiple sub-images based on a sliding window, calculating the local SSIM index between the corresponding sub-images to obtain the structural similarity between the two grayscale images, and calculating the image difference and obtaining the overlap deviation by contour extraction and minimum bounding rectangle positioning difference.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the user interface acceptance method as described in any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the user interface acceptance method as described in any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the user interface acceptance method as described in any one of claims 1 to 6.

Images