User interface identification intelligent testing method, device, terminal and storage medium
By combining image processing and differential testing technology in user interface testing and user interface recognition based on multi-dimensional features, the problem of detecting the display differences of different hardware devices in the prior art is solved, and more efficient and accurate test results are achieved.
Patent Information
- Application Number
- CN202210528320.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-05-16
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2042-05-16
AI Technical Summary
The prior art is difficult to effectively detect the display differences of software on different hardware devices in user interface testing, and relying on manual judgments leads to inadequate objective and accurate test results.
Using intelligent user interface recognition testing method based on multi-dimensional features, combining image processing technology and differential testing, customizing differential detection methods through block segmentation, edge detection and image understanding of user interface screenshots, generating similarity reports, and reducing manual labor Intervention.
It improves the accuracy and objectivity of user interface testing, can effectively detect complex user interface differences, reduce manual errors, and improve testing efficiency.
Smart Images

Figure CN114863142B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of software testing technology, and is particularly suitable for the field of application software testing with a user interface. In particular, it relates to an intelligent user interface recognition testing method, device, terminal and storage medium that can more quickly detect differences in user interfaces displayed on different hardware devices when the same software is deployed, and help software testers improve software testing efficiency and ensure software quality. Background Art
[0002] In recent years, the Internet industry has developed rapidly, and various applications have emerged, which are related to all aspects of social life. At the same time, the testing technology for software applications has become increasingly important. At present, many industries do not provide source code for testing due to confidentiality reasons. Testers need to perform black box testing on the characteristics of the software and write test cases for the data returned by the server. However, in terms of user interfaces, whether in Web development or client development, there are many types of drawing software (such as CAD drawing software) on the market, and the data format cannot be unified. And because of the particularity of user interface data, it is difficult for testers to simply test the specific state of the user interface through data. They can only judge the correctness of the data, but cannot guarantee the rationality of the final display. Therefore, new testing technologies for user interface features have become an important hot issue in academia and industry.
[0003] In the testing of applications involving interactive interfaces, user interface recognition technology has become the mainstream method. This is because of the particularity of the user interface. It is difficult to conduct truly effective testing from the data level. Testers are therefore more concerned with understanding the user interface of the software through image recognition. At present, most user interface testing technologies are tests that combine image recognition and machine learning. Through human participation in annotation, combined with machine learning, the trained model is finally used for the recognition and processing of graphics in the user interface. However, because machine learning requires a large amount of data, the human input in this regard is very high. At the same time, people have subjective judgment problems. For example, a slight deviation in the angle may mark an ordinary triangle as a right triangle due to human observation problems; for example, the parallel relationship between two line segments, when the line segments are far apart, it is easy for people to subjectively regard non-parallel line segments as parallel line segments. Therefore, human subjective problems will cause problems in some graphic categories with high similarity. The most common ones are deviations in the judgment of right angles, parallel relationships, and length relationships. Human deviations will cause problems in the training data, affect the model, and finally the test results of plane geometry will have problems, especially at the critical points of different types of graphics, such as equilateral triangles and isosceles triangles. Therefore, we hope to study a method to get rid of human involvement as much as possible, improve the objectivity of the test, and thus achieve the goal of improving accuracy.
[0004] In the area of differential testing, some scholars have done some preliminary work, but few have applied it to user interface display testing, and only a few have proposed using differential testing for compiler testing. However, the above differential testing cannot meet the requirements for detecting differences in complex user interfaces. Summary of the invention
[0005] In order to overcome the shortcomings of the prior art software testing methods for user interfaces, the purpose of the present invention is to provide an intelligent user interface recognition testing method, device, terminal and storage medium, which is based on the multi-dimensional characteristics of the user interface and draws on existing mature image processing technology to achieve feature recognition and extraction of the user interface, improve accuracy, get rid of dependence on manual identification, and customize differential detection methods to meet the detection of complex user interfaces, and finally provide a differential test report based on similarity.
[0006] The present invention is implemented by the following technical solutions:
[0007] A user interface recognition intelligent testing method, comprising:
[0008] Get a screenshot of the application software's user interface;
[0009] Get the difference threshold;
[0010] By combining differential testing technology with image recognition technology, interface features are extracted and interface similarity is measured for user interface screenshots. By comparing interface similarity and difference thresholds, it is determined whether there are display differences for the same application software on different devices, and the test results are generated into a differential test report.
[0011] Optionally, the interface feature extraction includes the following steps:
[0012] Divide the user interface screenshot into blocks and obtain a block set;
[0013] Construct a block association relationship diagram based on the block stacking relationship;
[0014] Get the controls in all blocks based on edge detection;
[0015] Identify control intent based on image understanding and intent recognition technology.
[0016] Further optionally, the step of dividing the user interface screenshot into blocks and obtaining a block set specifically includes:
[0017] First, Gaussian filtering is applied to smooth the image and remove some noise. Then the intensity gradient of the image is found and non-maximum suppression technology is applied to eliminate false detection of edges. Next, double thresholds are used to determine possible boundaries. The partitions are segmented according to the correlation of the control types in the same partition. After segmentation, the controls in each partition are of the same type, while the controls in different partitions are of different types.
[0018] Optionally, a point in the block association relationship graph represents a partition, and an edge represents an association relationship between two partitions.
[0019] Optionally, the step of acquiring controls in all blocks according to edge detection specifically includes:
[0020] Graphic edge detection technology is used to detect the boundaries of each control, and then the types and quantities of controls are counted.
[0021] Optionally, the interface similarity measurement includes the following steps:
[0022] Design corresponding similarity measurement methods according to the different dimensional characteristics of the user interface;
[0023] A fusion algorithm is used to fuse the similarities of multiple dimensions into a single value to represent the similarity of user interfaces.
[0024] Further optionally, the fusion algorithm adopts the following formula:
[0025]
[0026] in, Indicates interface 1 and interface 2, Indicates the similarity between two user interfaces. Indicates the similarity of controls. Indicates the similarity of the control's intent, Represents the similarity of the association graph, Represents the fusion function.
[0027] In addition, to achieve the above-mentioned purpose, the present invention also provides a user interface recognition intelligent testing device comprising:
[0028] The upload module is used to obtain the screenshot of the user interface of the application software;
[0029] A setting module is used to obtain a difference threshold;
[0030] The differential testing module is used to extract interface features and measure interface similarity from user interface screenshots by combining differential testing technology with image recognition technology. By comparing interface similarity with difference thresholds, it is determined whether there are display differences for the same application software on different devices, and the test results are generated into a differential testing report.
[0031] In addition, to achieve the above-mentioned purpose, the present invention also provides a terminal, which includes a memory, a processor, and a user interface recognition intelligent test program stored in the memory and executable on the processor, wherein: the user interface recognition intelligent test program implements the steps of the above-mentioned user interface recognition intelligent test method when executed by the processor.
[0032] In addition, to achieve the above-mentioned purpose, the present invention also provides a computer-readable storage medium, on which a user interface recognition intelligent test program is stored. When the user interface recognition intelligent test program is executed by a processor, the steps of the above-mentioned user interface recognition intelligent test method are implemented.
[0033] Compared with the prior art, the present invention has the following beneficial effects:
[0034] The user interface recognition intelligent testing method, device, terminal and storage medium of the present invention solve the problem that the prior art software testing field still has insufficient testing methods for user interfaces. It is based on the multi-dimensional characteristics of the user interface and draws on existing mature image processing technology to achieve feature recognition and extraction of the user interface, improve accuracy, get rid of dependence on manual identification, and customize differential detection methods to meet the detection of complex user interfaces, and finally provide a differential test report based on similarity.
[0035] The user interface recognition intelligent testing method of the present invention introduces differential testing into the user interface display testing field for the first time, combines image processing technology with differential testing for the first time, completes user interface recognition intelligent testing, and for the first time frees user interface display error detection from dependence on human judgment, thereby improving the objectivity of the test and achieving the purpose of improving accuracy. BRIEF DESCRIPTION OF THE DRAWINGS
[0036] Figure 1 This is a flowchart of the implementation of the intelligent testing method for user interface recognition of the present invention;
[0037] Figure 2 for Figure 1 The flowchart of the user interface feature extraction module shown;
[0038] Figure 3 for Figure 1 The flowchart of the user interface feature similarity calculation module is shown. DETAILED DESCRIPTION
[0039] The present invention is further described below in conjunction with specific implementation methods. It should be noted that, under the premise of no conflict, the various embodiments or technical features described below can be arbitrarily combined to form a new embodiment.
[0040] Figure 1 is an implementation flow chart of the user interface recognition intelligent testing method of the present invention, such as Figure 1 As shown, the user interface recognition intelligent testing method includes at least the following steps:
[0041] S10, obtaining a screenshot of the user interface of the application software;
[0042] S20, obtaining a difference threshold;
[0043] S30, through the differential testing technology combined with image recognition technology, the interface features are extracted and the interface similarity is measured for the user interface screenshots. By comparing the interface similarity with the difference threshold, it is determined whether there are display differences for the same application software on different devices, and the test results are generated into a differential test report.
[0044] In this embodiment, the object of the user interface recognition intelligent test can be application software that can be installed on terminal devices of different platforms, so as to test the screenshots obtained from the same application software installed on terminal devices of different platforms through the same user interface recognition intelligent test method. For example, the application software can be life practical software, social software, efficient office software, music video software, shopping software, travel software, etc., the terminal device can be a mobile phone, a tablet computer, a car computer, a wearable device, etc., and the platform of the terminal device can include one of the following: Android platform, iOS platform, Windows platform and Linux platform, etc. The embodiment of this application does not limit the type of application software, the type of terminal device and the type of platform of the terminal device.
[0045] The present invention solves the problem that the testing methods for user interfaces in the field of software testing in the prior art are still insufficient. Based on the multi-dimensional characteristics of the user interface, the present invention draws on the existing mature image processing technology to realize feature recognition and extraction of the user interface, improve accuracy, get rid of the dependence on manual identification, and customize the differential detection method to meet the detection of complex user interfaces, and finally provide a differential test report based on the similarity.
[0046] In this embodiment, the user interface screenshots provided by the user are used to test whether the same application software has display differences on different platforms based on the differential testing principle, and a user interface similarity report is generated. The method includes the following three modules:
[0047] 1) User interface feature extraction module: Perform multi-dimensional feature extraction on the user interface. First, image recognition technology is used to identify the partition blocks in the user interface, and the relative positions and associations of the partition blocks are captured. Next, edge detection technology is used to extract the edge information of the control graphics in the partition blocks, that is, the edges that make up the control. Then, image summarization technology is used to predict the intent of the control, that is, the text description, based on the type and content of the control. The association features, control features, and intent features constitute the feature space of the user interface.
[0048] 2) User interface feature similarity measurement module: This module predefines the feature similarity measurement method and designs different similarity measurement algorithms for different features. This section first measures the control similarity. Considering that the display may be different on different devices or operating systems, we do not consider the size of the control, but only the type and shape of the control. When measuring the intent similarity, considering that the intent is the semantics of the natural language text description, we use a mature language model to measure the semantic similarity of the text. When measuring the association similarity, we first build an association relationship graph between partition blocks. Each block is regarded as a vertex in the graph and connected from top to bottom using directed edges. Then use the graph edit distance to measure the similarity between the association graphs of two blocks. After obtaining the control, intent, and association similarity, these three dimensions are further integrated to obtain the user interface similarity.
[0049] 3) Difference judgment module: This module determines whether there are differences between the two compared user interfaces based on the user interface similarity and the manually set difference threshold, thereby determining whether the application software has display errors. It should be noted that the setting of this difference threshold takes into account the accuracy error of image recognition and understanding technology. If the difference threshold is exceeded, it is considered that the user interface has a display error and has not passed the difference test. If it fails, a report that it has not passed the difference test will be given; if it passes, a report that it has passed the difference test will be given.
[0050] In a preferred embodiment, the interface feature extraction comprises the following steps:
[0051] Divide the user interface screenshot into blocks and obtain a block set;
[0052] Construct a block association relationship diagram based on the block stacking relationship;
[0053] Get the controls in all blocks based on edge detection;
[0054] Identify control intent based on image understanding and intent recognition technology.
[0055] Specifically, Figure 2 As shown, the user interface feature extraction module includes the following steps:
[0056] 1. User interface partition extraction:
[0057] In the present invention, for the user interface pictures provided by the user, we first preprocess them to eliminate the interference of platform-related features. Then, image recognition technology is used to identify valid partitions in the interface. Specifically, edge detection technology is used to perform edge detection. First, Gaussian filtering is applied to smooth the image and remove some noise. Then the intensity gradient of the image is found, and non-maximum suppression technology is applied to eliminate false detection of edges. Then, a double threshold is used to determine possible boundaries. The partitions are segmented according to the correlation of the control types in the same partition. After segmentation, the controls in each partition are of the same type, while the controls in different partitions are of different types.
[0058] 2. Partition association extraction
[0059] In the present invention, the user interface is composed of multiple different partitions, and these partitions are not randomly stacked. Therefore, even if the same application software runs on different platforms, the association relationship between the partitions of the user interface should be the same. We use an association relationship graph to describe the association relationship between different partitions, where a point in the graph represents a partition, and an edge of the graph represents the association relationship between two partitions, including sequential and parallel relationships.
[0060] 3. Control extraction
[0061] In the present invention, a control is a block with the smallest granularity in the user interface, and each partition may contain multiple controls. Even if the same application software runs on different platforms, the controls on the user interface should be the same, including the number and type. We regard the control as an image with smaller partitions, and use the graphic edge detection technology to detect the boundary of each control. Then we count the types and number of controls.
[0062] 4. Control Intent Recognition
[0063] In the present invention, the type of control determines the operations that the control can perform, such as the submission of a button, but this operation is universal and has nothing to do with specific business logic. For example, the texts in the two submit buttons are "Submit Form" and "Submit Report", respectively. Therefore, from the perspective of the control type alone, both buttons perform the submission operation, but the intentions of the two buttons are different. After all, "form" and "report" are semantically different. Therefore, it is necessary for us to pay attention to the intention of the control. Specifically, we identify the intention of the control based on image understanding and intention recognition technology.
[0064] like Figure 3 As shown, the user interface feature similarity calculation module includes the following steps:
[0065] 1. Control Similarity Metrics
[0066] In the present invention, the control similarity metric is used to measure the similarity of the smallest granularity. We comprehensively measure the similarity of the metric space from two perspectives: control type and control intent. Specifically, we first measure whether the control types are consistent. If the control types are consistent, we perform the control intent similarity metric. If they are inconsistent, there is no need to perform subsequent intent similarity metric. Regarding the similarity of control intent, considering the performance of image understanding and intent recognition technology, the accuracy of such technology usually cannot reach 100%. Therefore, we allow errors in intent similarity. When such errors are less than the set difference threshold, it is judged as control intent. Figure 1 To.
[0067] 2. Association Similarity Measures
[0068] In the present invention, the association similarity metric is used to measure the similarity of the association graph. We use the graph edit distance to measure the similarity between two block association graphs. The operations involved in the edit distance calculation include adding and deleting operations. The modification operation can be regarded as deleting a node first and then inserting a new node. The edit distance is the ratio of the sum of the edit operations to the maximum number of nodes in the two graphs.
[0069] 3. User Interface Similarity Metrics
[0070] In the present invention, the user interface similarity measure is used to uniquely represent the differences between users. We use the fusion result of similarities in multiple dimensions as the user interface similarity.
[0071]
[0072] The above formula is the calculation method of the fusion algorithm, where Indicates interface 1 and interface 2, Indicates the similarity between two user interfaces. Indicates the similarity of controls. Indicates the similarity of the control's intent, Represents the similarity of the association graph, Represents the fusion function, which can be the mean or the weighted mean.
[0073] The user interface recognition intelligent testing method of the present invention introduces differential testing into the user interface display testing field for the first time, combines image processing technology with differential testing for the first time, completes user interface recognition intelligent testing, and for the first time frees user interface display error detection from dependence on human judgment, thereby improving the objectivity of the test and achieving the purpose of improving accuracy.
[0074] In addition, the present invention also provides a user interface recognition intelligent testing device comprising:
[0075] The upload module is used to obtain the screenshot of the user interface of the application software;
[0076] A setting module is used to obtain a difference threshold; the difference threshold setting module can be set manually or automatically by an algorithm, and no matter which setting method is used, the accuracy error of the image recognition and understanding technology needs to be considered;
[0077] The differential testing module is used to extract interface features and measure interface similarity from user interface screenshots by combining differential testing technology with image recognition technology. By comparing interface similarity with difference thresholds, it is determined whether there are display differences for the same application software on different devices, and the test results are generated into a differential testing report.
[0078] Optionally, the differential test module includes:
[0079] 1. User interface feature extraction module: Perform multi-dimensional feature extraction on the user interface, and analyze the partitions, partition associations, controls, and control intent in the user interface through image recognition technology to improve accuracy.
[0080] 2. User interface feature similarity measurement module: used to predefine feature similarity measurement methods and design different similarity measurement algorithms for different features
[0081] 3. Difference judgment module: used to judge whether there are differences between the two compared user interfaces based on the user interface similarity and the manually set difference threshold, so as to judge whether there are display errors in the application software.
[0082] In addition, the present invention also provides a terminal, which includes a memory, a processor, and a user interface recognition intelligent test program stored in the memory and executable on the processor, wherein: when the user interface recognition intelligent test program is executed by the processor, the steps of the above-mentioned user interface recognition intelligent test method are implemented.
[0083] In addition, the present invention also provides a computer-readable storage medium, on which a user interface recognition intelligent test program is stored. When the user interface recognition intelligent test program is executed by a processor, the steps of the above-mentioned user interface recognition intelligent test method are implemented.
[0084] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.
[0085] One or more embodiments of the present invention may be described in the general context of computer executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. One or more embodiments of the present application may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.
[0086] Each embodiment of the present invention is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.
[0087] The above description is only an embodiment of this document and is not intended to limit this document. For those skilled in the art, this document may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of this document should be included in the scope of the claims of this document.
Claims
1. A user interface recognition intelligent testing method, characterized in that: include: Get a screenshot of the application software's user interface; Get the difference threshold; By combining differential testing technology with image recognition technology, interface features are extracted and interface similarity is measured for user interface screenshots. By comparing interface similarity with difference thresholds, it is determined whether the same application software has display differences on different devices, and the test results are generated into a differential test report. The feature space extracted from the interface features includes association features, control features and intention features; The interface feature extraction comprises the following steps: Divide the user interface screenshot into blocks and obtain a block set; Construct a block association relationship diagram based on the block stacking relationship; Get the controls in all blocks based on edge detection; Identify control intent based on image understanding and intent recognition technology; The measurement aspects of the interface similarity include control similarity, intention similarity and association similarity; The interface similarity measurement comprises the following steps: Design corresponding similarity measurement methods according to the different dimensional characteristics of the user interface; A fusion algorithm is used to fuse the similarities of multiple dimensions into a single value to represent the similarity of the user interfaces; the fusion algorithm adopts the following formula: Sim(I1,I2)=f(S W ,S I ,S R ) Among them, I1, I2 represent interface 1 and interface 2, Sim(I1, I2) represents the similarity between the two user interfaces, and S W Indicates the similarity of controls, S I Indicates the similarity of control intentions, S R represents the similarity of the association graph, and f(·) represents the fusion function.
2. The user interface recognition intelligent testing method according to claim 1, characterized in that: The step of dividing the user interface screenshot into blocks and obtaining a block set specifically includes: First, Gaussian filtering is applied to smooth the image and remove some noise. Then the intensity gradient of the image is found and non-maximum suppression technology is applied to eliminate false detection of edges. Next, double thresholds are used to determine possible boundaries. The partitions are segmented according to the correlation of the control types in the same partition. After segmentation, the controls in each partition are of the same type, while the controls in different partitions are of different types.
3. The user interface recognition intelligent testing method according to claim 1, characterized in that: A point in the block association relationship graph represents a partition, and an edge represents the association relationship between two partitions.
4. The user interface recognition intelligent testing method according to claim 1, characterized in that: The step of obtaining controls in all blocks according to edge detection specifically includes: Graphic edge detection technology is used to detect the boundaries of each control, and then the types and quantities of controls are counted.
5. A user interface recognition intelligent testing device, characterized in that: include: The upload module is used to obtain the screenshot of the user interface of the application software; A setting module is used to obtain a difference threshold; The differential test module is used to extract interface features and measure interface similarity from user interface screenshots by combining differential test technology with image recognition technology. By comparing interface similarity with difference thresholds, it is determined whether there are display differences between the same application software on different devices, and the test results are used to generate differential test reports. The feature space extracted from the interface features includes association features, control features and intention features; The interface feature extraction comprises the following steps: Divide the user interface screenshot into blocks and obtain a block set; Construct a block association relationship diagram based on the block stacking relationship; Get the controls in all blocks based on edge detection; Identify control intent based on image understanding and intent recognition technology; The measurement aspects of the interface similarity include control similarity, intention similarity and association similarity; The interface similarity measurement comprises the following steps: Design corresponding similarity measurement methods according to the different dimensional characteristics of the user interface; A fusion algorithm is used to fuse the similarities of multiple dimensions into a single value to represent the similarity of the user interfaces; the fusion algorithm adopts the following formula: Sim(I1,I2)=f(S W ,S I ,S R ) Among them, I1, I2 represent interface 1 and interface 2, Sim(I1, I2) represents the similarity between the two user interfaces, and S W Indicates the similarity of controls, S I Indicates the similarity of control intentions, S R represents the similarity of the association graph, and f(·) represents the fusion function.
6. A terminal, characterized in that: The terminal includes a memory, a processor, and a user interface recognition intelligent test program stored in the memory and executable on the processor, wherein: when the user interface recognition intelligent test program is executed by the processor, the steps of the user interface recognition intelligent test method as described in any one of claims 1 to 4 are implemented.
7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a user interface recognition intelligent test program, and when the user interface recognition intelligent test program is executed by the processor, the steps of the user interface recognition intelligent test method according to any one of claims 1 to 4 are implemented.
Citation Information
Patent Citations
Black box regression test method
CN107193739A
Cross-platform user interface automatic test method and device, equipment and storage medium
CN112231234A
Method and apparatus for performing automated user-interface layout testing
US9213625B1
User interface automated testing method and apparatus, and computer-readable storage medium
WO2020233331A1
