Test method, device, computer storage medium and electronic equipment of user interface

Abstract

The application discloses a kind of user interface test method, device, computer storage medium and electronic equipment, it is related to the field of financial technology, wherein, the verification method includes: obtaining the design specification file of user interface and design draft;Extract the layer position information and layer style information of each element in design draft, determine the component in design draft by element in design draft, and determine the type of component according to the layer position information and layer style information of element in component;Determine the target specification content matched with design draft in design specification file, and respectively determine component ID according to the type of each component, extract the standard style associated with each component ID from target specification content;Compare the layer style information of each component with standard style, obtain comparison result, and show comparison result, the present application solves the technical problems of low efficiency and low accuracy in related art by comparing and detecting components in artificial way.

Description

Technical Field

[0001] This invention relates to the field of financial technology, and more specifically, to a method, apparatus, computer storage medium, and electronic device for testing user interfaces. Background Technology

[0002] The user interface (UI) of a software application plays a crucial role in establishing a positive user impression and remains indispensable from the initial stages of software design and development. Therefore, UI testing is highly valued in the software industry. UI testing includes testing user-friendliness, human-centered design, and ease of use. During the software design phase, if the UI design differs from industry-standard design specifications, modifications can be made directly based on the identified issues, thus helping development teams reduce trial-and-error costs and shorten the development cycle.

[0003] However, the compliance inspection of user interface design drafts currently relies mainly on manual methods. The design drafts are compared with the design specifications by visual inspection. Based on the component information in the design drafts, the corresponding design standards are found in the design specifications, and then manual annotations are made. Finally, the inspection is carried out based on the annotated component categories.

[0004] There is currently no effective solution to the problem of low efficiency and low accuracy in manually comparing and testing components in related technologies. Summary of the Invention

[0005] This invention provides a method, apparatus, computer storage medium, and electronic device for testing user interfaces, to at least solve the technical problems of low efficiency and low accuracy in comparing and testing components manually in related technologies.

[0006] According to one aspect of the embodiments of this application, a method for testing a user interface is provided. The method includes: obtaining a design specification file for the user interface and opening a design draft of the user interface stored on a server, wherein the design draft is composed of multiple components, each component containing at least one element, the design specification file containing multiple design specification contents, each design specification content containing component IDs of multiple types of components and standard styles for each component; extracting layer position information and layer style information of each element in the design draft, determining components in the design draft from the elements in the design draft, and determining the type of the component based on the layer position information and layer style information of the elements in the component; determining target specification content matching the design draft in the design specification file, and determining component IDs according to the type of each component in the design draft, extracting the standard style associated with each component ID from the target specification content; comparing the layer style information of each component with the standard style to obtain a comparison result, and displaying the comparison result on a browser on the server.

[0007] Optionally, obtaining the user interface design specification file and opening the user interface design draft stored on the server includes: obtaining the design draft compressed package stored on the server, decompressing the design draft compressed package to obtain the design draft; executing a preset startup command on the server to load the design draft and opening the design draft, wherein the preset startup command is used to call the browser driver to load the design draft into the browser.

[0008] Optionally, extracting the layer position information and layer style information of each element in the design draft includes: traversing the design draft, obtaining the outline position coordinates of the layers of each element in the design draft, and obtaining the layer position information of each layer; for an element containing a single layer, triggering the layer, obtaining the layer style information of the element, and determining the layer position information of the layer as the layer position information of the element, wherein the layer style information includes at least one of the following style information: the size of the layer, color, and text on the element; for an element containing multiple layers, establishing the association relationship between multiple layers of the same element, and triggering multiple layers of the same element respectively, obtaining multiple layer style information, the multiple layer style information constitutes the layer style information of the element, and the layer position information of multiple layers of the same element constitutes the layer position information of the element, wherein the element containing multiple layers is determined in the following way: determining at least two layers with an inclusion relationship based on the layer position information of each layer, and determining the at least two layers with an inclusion relationship as layers of the same element.

[0009] Optionally, determining the components in the design draft based on the elements in the design draft, and determining the type of the component based on the layer position information and layer style information of the elements in the component, includes: determining the type of a single component based on the layer position information and layer style information of the elements in the design draft; combining single components and local features not identified as components into a combined component, and determining the type of the combined component.

[0010] Optionally, determining the type of a single component based on the layer position information and layer style information of elements in the design draft includes: slicing the design draft according to the layer position information of the elements to obtain multiple sub-images, where each sub-image contains at least one element; for each sub-image, determining whether the sub-image conforms to the component construction rules; if the sub-image does not conform to the component construction rules, determining that the sub-image does not contain a component; if the sub-image conforms to the component construction rules, analyzing the layer style information of the elements in the sub-image according to the component construction rules to obtain the components in the sub-image and the first probability of the components belonging to each type; inputting the layer style information of the elements in the sub-image into a first image recognition model to identify the components in the sub-image and the second probability of the components belonging to each type; determining the weight associated with the component construction rules and the weight associated with the first image recognition model, weighted summing the first and second probabilities of each component belonging to each type to obtain multiple probabilities, and determining the type indicated by the highest probability among the multiple probabilities as the type of the component.

[0011] Optionally, combining single components and local features not identified as components into combined components, and determining the type of combined components includes: combining and analyzing local features in the design draft and identified single components according to component combination rules to obtain multiple combined components and a third probability that each combined component belongs to each type; inputting the combined components in the design draft into a second image recognition model to identify a fourth probability that the combined components belong to each type; determining the weights associated with the component combination rules and the weights associated with the second image recognition model, and performing a weighted summation of the third and fourth probabilities that each combined component belongs to each type to obtain multiple probabilities, and determining the type indicated by the highest probability among the multiple probabilities as the type of the combined component.

[0012] Optionally, displaying the comparison results on the server's browser includes: constructing a table based on the comparison results of each component, the component type, the component's layer style information, and the component's standard style information, and displaying the constructed table in the browser; or packaging the comparison results, component type, component's layer style information, and component's standard style information for each component separately to obtain packaged information, associating the packaged information with the corresponding component, and displaying the packaged information associated with the component after receiving a click event executed on the component.

[0013] According to another aspect of the embodiments of this application, a user interface testing apparatus is provided. The apparatus includes: an acquisition unit, configured to acquire a user interface design specification file and open a user interface design draft stored on a server, wherein the design draft is composed of multiple components, each component containing at least one element, the design specification file containing multiple design specification contents, each design specification content containing component IDs of multiple types of components and standard styles for each component; an extraction unit, configured to extract layer position information and layer style information of each element in the design draft, determine components in the design draft from the elements in the design draft, and determine the type of component based on the layer position information and layer style information of the elements in the component; a determination unit, configured to determine the target specification content matching the design draft in the design specification file, determine the component ID according to the type of each component in the design draft, and extract the standard style associated with each component ID from the target specification content; and a comparison unit, configured to compare the layer style information of each component with the standard style, obtain a comparison result, and display the comparison result on a browser on the server.

[0014] According to another aspect of the present invention, a computer-readable storage medium is also provided, the computer-readable storage medium including a stored computer program, wherein the test method for controlling the device where the computer-readable storage medium is located to execute the above-described user interface is executed when the computer program is running.

[0015] According to another aspect of the present invention, an electronic device is also provided, including one or more processors and a memory, the memory being used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the above-described test method for the user interface.

[0016] This application employs the following steps: obtaining the user interface design specification file and opening the user interface design draft stored on the server. The design draft consists of multiple components, each containing at least one element. The design specification file contains multiple design specification contents, each containing component IDs for multiple component types and standard styles for each component. The layer position information and layer style information of each element in the design draft are extracted. The components in the design draft are determined from the elements, and the component type is determined based on the layer position information and layer style information of the elements within the components. The target specification content that matches the design draft is determined in the design specification file. Each component ID is determined based on its type in the design draft, and the standard style associated with each component ID is extracted from the target specification. The layer style information of each component is compared with the standard style, and the comparison results are displayed on the server's browser. This solves the problems of low efficiency and low accuracy in manually comparing and testing components in related technologies. By using an image recognition model to distinguish the types of components in the design draft, and then extracting the standard styles of the corresponding types of components from the design specification, the layer styles of the components are compared with the standard styles, thereby achieving accurate and efficient testing of the components in the design draft. Attached Figure Description

[0017] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings:

[0018] Figure 1 This is a flowchart of a user interface testing method provided according to an embodiment of this application;

[0019] Figure 2 This is a schematic diagram illustrating the standard style of a component as provided in the embodiments of this application;

[0020] Figure 3 This is a flowchart illustrating the determination of component types according to embodiments of this application;

[0021] Figure 4 This is a timing diagram of an optional user interface testing method provided according to an embodiment of this application;

[0022] Figure 5 This is a schematic diagram of a testing apparatus with a user interface provided according to an embodiment of this application;

[0023] Figure 6 This is a schematic diagram of an electronic device provided according to an embodiment of this application. Detailed Implementation

[0024] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0025] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0026] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate for the embodiments of this application described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0027] It should be noted that all information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for display, data used for analysis, etc.) involved in this disclosure are information and data authorized by the user or fully authorized by all parties.

[0028] According to an embodiment of this application, a method for testing a user interface is provided.

[0029] Figure 1 This is a flowchart of a user interface testing method provided according to an embodiment of this application, such as... Figure 1 As shown, the method includes the following steps:

[0030] Step S102: Obtain the user interface design specification file and open the user interface design draft stored on the server. The design draft is composed of multiple components, each component contains at least one element, and the design specification file contains multiple design specification contents. Each design specification contents contains the component ID of multiple types of components and the standard style of each component.

[0031] Specifically, when performing specification checks on the design draft of the user interface, it is first necessary to obtain the design draft and the specification document for the user interface design, that is, the design specification document. Among them, the component refers to the visual graphic element placed on the design draft page. It completes the corresponding response by running the code instructions. The types of components can include buttons, drop-down boxes, navigation bars and dialog boxes, etc. A complete design draft can be formed by using different arrangements or combinations of multiple components.

[0032] It should be noted that in the design draft, a component is composed of one or more elements. Each element can contain one or more layers, and a component can contain one or more elements. That is, the elements and the relevant information of the layers contained in the elements together form a component. Here, a layer refers to a "film" containing elements such as text or graphics, which are stacked together in order. Finally, multiple layers are combined to form an image, which is the presentation effect of the page. Layers accurately position the elements on the page. Text, images, plugins, or nested layers can be added to layers, that is, text layers, image layers, etc.

[0033] Step S104: Extract the layer position information and layer style information of each element in the design draft, determine the components in the design draft based on the elements in the design draft, and determine the type of the component based on the layer position information and layer style information of the elements in the component.

[0034] To find the corresponding design standards from the specification documents, it is necessary to determine the type of each component in the design draft. The component type requires the layer position and layer style information of the component elements. Specifically, retrieval software is used to identify the layers in the design draft. If the layer is rectangular, the coordinates of its four points are calculated and used as the layer position information. Then, programming software is used to simulate clicking the center position of the component element based on the layer position information, thereby obtaining the layer's color, size, and other layer style information.

[0035] Based on the relationship between elements and components, the component to which an element belongs is determined, or a single element is treated as a component. Furthermore, the obtained layer position information and layer style information are used as inputs and fed into the trained image recognition model to finally determine the type of component composed of elements.

[0036] Step S106: Determine the target specification content that the design draft matches in the design specification file, and determine the component ID according to the type of each component in the design draft. Extract the standard style associated with each component ID from the target specification content.

[0037] For example, design specification documents may include design specification content for mobile devices and design specification content for desktop devices. Each design specification content includes the component IDs of multiple types of components involved in the design drafts for the corresponding device, as well as the standard styles of the components.

[0038] Therefore, during the comparative testing, the target specification content for matching the design draft type to be tested is first determined in the design specification file. Then, the corresponding component ID is assigned in the specification content according to the component type in the design draft, and the corresponding standard style is found in the design specification content according to the component ID. Figure 2 This is a schematic diagram illustrating the standard style of a component determined according to an embodiment of this application. Figure 2 As shown, for example, when the design draft to be inspected is a mobile user interface design draft, it is necessary to find the standard style of the dialog box. First, the design specification content of the mobile terminal is obtained from the design specification file, and the corresponding component ID is assigned in the specification content according to the component type in the design draft. Then, the corresponding standard style is found in the design specification content according to the component ID. That is, the component ID assigned to the dialog box is 102. Then, according to the component ID and the ID of the mobile terminal specification, the component ID: 100102 in the specification content is found, which means the corresponding standard style can be found.

[0039] Step S108: Compare the layer style information of each component with the standard style to obtain the comparison result, and display the comparison result on the browser of the server.

[0040] Specifically, the names of standard styles are represented using standard Cascading Style Sheets (CSS) styles. The standard values ​​under the obtained standard style names are compared with the component's layer style information, and the comparison results are displayed, allowing designers to intuitively understand whether the components in the design draft conform to design specifications.

[0041] For example, the standard size of the navigation bar should be 19 pixels. The current size of the navigation bar in the detected layer style information is 18 pixels. By collecting and storing the two values, the comparison result of the navigation bar can be obtained.

[0042] The user interface testing method provided in this application involves obtaining a user interface design specification file and opening a user interface design draft stored on a server. The design draft consists of multiple components, each containing at least one element. The design specification file contains multiple design specification contents, each containing component IDs for multiple types of components and standard styles for each component. The method extracts layer position information and layer style information from each element in the design draft. Components in the design draft are determined from the elements, and the component type is determined based on the layer position information and layer style information of the elements within the components. Finally, a target specification matching the design draft is determined in the design specification file. The system identifies the component ID for each component in the design draft based on its type, extracts the standard style associated with each component ID from the target specification, compares the layer style information of each component with the standard style, obtains the comparison results, and displays the comparison results on the server's browser. This solves the problems of low efficiency and low accuracy in manually comparing and testing components in related technologies. By using an image recognition model to distinguish the types of components in the design draft, extracting the standard styles of the corresponding types of components from the design specification, and comparing the layer styles of the components with the standard styles, the system achieves accurate and efficient testing of the components in the design draft.

[0043] Optionally, in the user interface testing method provided in this application embodiment, obtaining the user interface design draft in the server includes: obtaining the design draft compressed package stored in the server, decompressing the design draft compressed package to obtain the design draft; executing a preset startup command on the server to load the design draft, and opening the design draft, wherein the preset startup command is used to call the browser driver to load the design draft into the browser.

[0044] Specifically, to obtain the user interface design drafts, firstly, the compressed design draft package stored in the design software's repository is decompressed, resulting in multiple page design drafts. Secondly, a preset startup command is executed on the current server to invoke the browser driver and load the design drafts into a simulated browser. The design draft homepage is then opened using boundless mode, which means directly loading and opening the design draft page to be tested without opening a browser page.

[0045] Optionally, in the user interface testing method provided in this application embodiment, extracting the layer position information and layer style information of each element in the design draft includes: traversing the design draft, obtaining the outline position coordinates of the layers of each element in the design draft, and obtaining the layer position information of each layer; for an element containing a single layer, triggering the layer, obtaining the layer style information of the element, and determining the layer position information of the layer as the layer position information of the element, wherein the layer style information includes at least one of the following style information: the size of the layer, the color, and the text on the element; for an element containing multiple layers, establishing the association relationship of multiple layers of the same element, and triggering multiple layers of the same element respectively, obtaining multiple layer style information, the multiple layer style information constitutes the layer style information of the element, and the layer position information of multiple layers of the same element constitutes the layer position information of the element, wherein the element containing multiple layers is determined by: determining at least two layers with an inclusion relationship based on the layer position information of each layer, and determining the at least two layers with an inclusion relationship as the layers of the same element.

[0046] Specifically, the software is used to traverse all the design draft pages, identify the layers contained in the design draft, and extract the position coordinates of the layers. Based on the obtained position coordinates and the length and width of the layers, the coordinates of the four positions of the layer (top left, top right, bottom left, and bottom right) are calculated, and the coordinates of the four positions are recorded as the layer position information.

[0047] Since a component can contain one or more elements, when retrieving layer style information for an element containing a single layer, the user clicks on the corresponding layer in the design draft, and the style information for that layer pops up. This style information can include the layer's color, size, etc. After extracting the pop-up style information, the layer's position information is determined as the element's layer position information, and the style information is determined as the element's layer style information. These are then saved together with the layer position information.

[0048] For elements containing multiple layers, it is necessary to obtain the layer style information for each element separately. For example, a button component contains multiple button elements, each containing an outer large layer and an inner text layer, etc. After obtaining the four-point coordinates of the multiple layers of the button element, the relationship between the layers is determined based on the coordinate data. That is, if the inner text layer is contained within the coordinates of the outer large layer, the relationship between the two is determined. When the button component is clicked, multiple layers are triggered, and the style information of multiple layers is extracted. This information together constitutes the layer style information of the button element.

[0049] Optionally, in the user interface testing method provided in this application embodiment, determining the components in the design draft from the elements in the design draft, and determining the type of the component based on the layer position information and layer style information of the elements in the component includes: determining the type of a single component based on the layer position information and layer style information of the elements in the design draft; combining the single component and local features not identified as components into a combined component, and determining the type of the combined component.

[0050] Specifically, components in the design draft can include single components and composite components. Composite components consist of multiple single components or local features that were not identified as components. The methods for determining the types of single components and composite components are different. Based on the obtained layer position information and layer style information of the elements, the type of a single component can be obtained by inputting the information into an image recognition model. Combining single components with local features yields composite components, and inputting these composite components into an image recognition model yields the type of the composite component. Here, local features refer to elements that were not identified as part of a complete component.

[0051] Optionally, in the user interface testing method provided in this application embodiment, determining the type of a single component based on the layer position information and layer style information of elements in the design draft includes: slicing the design draft according to the layer position information of the elements to obtain multiple sub-images, wherein each sub-image contains at least one element; for each sub-image, determining whether the sub-image conforms to the component construction rules; if the sub-image does not conform to the component construction rules, determining that the sub-image does not contain a component; if the sub-image conforms to the component construction rules, analyzing the element layer style information in the sub-image according to the component construction rules to obtain the components in the sub-image and the first probability that the components belong to each type; inputting the element layer style information in the sub-image into a first image recognition model to identify the components in the sub-image and the second probability that the components belong to each type; determining the weight associated with the component construction rules and the weight associated with the first image recognition model, weighted summing the first probability and the second probability that each component belongs to each type to obtain multiple probabilities, and determining the type indicated by the highest probability among the multiple probabilities as the type of the component.

[0052] Specifically, when identifying the type of a component, it is necessary to determine the images in the design draft based on the obtained layer position information and layer style information, and then slice the design draft to obtain a design draft containing multiple sub-images. Each sub-image can include one or more elements. For example, when slicing a text box, if the text box is a sub-image containing multiple elements, then the elements are further sliced ​​to obtain the most refined sub-image.

[0053] After obtaining the sub-image, the first step is to use interference rules to judge all layers in the sub-image and remove the layers that are judged to be interference layers. The interference rules can include the layer's transparency index, the size of the layer, the relative position and overlapping relationship of the layers, and whether there are gradients or color changes at the edges of the layers. For example, if a certain layer has high transparency, it is judged to be an interference layer and needs to be removed.

[0054] Then, on the one hand, it is determined whether the subgraph conforms to the component building rules. The component building rules can contain multiple sub-rules, each of which defines the characteristics of a type of component. A subgraph conforms to a sub-rule if the elements in the subgraph match the characteristics of the type of component defined by the sub-rule. For example, a sub-rule indicates that if a component contains two lines with slope information and connectivity, then the component is an arrow component.

[0055] When it is determined that the subgraph conforms to any subrule in the component construction rules, the probability value of the subgraph belonging to the corresponding type of component is obtained based on the analysis of the subrule, which is also the first probability.

[0056] On the other hand, the layer style information of the sub-image is used as input and fed into the first recognition model to determine the probability value of belonging to certain types, which is the second probability.

[0057] Furthermore, based on the determined weights of the two, the weights and probability values ​​are weighted and calculated. For example, for a certain component, the probability of identifying two models, A and B, can be obtained by inputting the component construction rule and the first recognition model, respectively. The probability of belonging to type A in the component construction rule is a1, and the probability of belonging to type B is b1; the probability of belonging to type A in the first recognition model is a2, and the probability of belonging to type B is b2. The weights of the first recognition model and the component construction rule are M and N, respectively. Then, the final probability of the component belonging to type A is N*a1+M*a2, and the probability of belonging to type B is N*b1+M*b2. After obtaining the two weighted probability values, the maximum value between the two is the type of the component.

[0058] Optionally, in the user interface testing method provided in this application embodiment, combining single components and local features not identified as components into combined components, and determining the type of combined components includes: combining and analyzing local features in the design draft and identified single components according to component combination rules to obtain multiple combined components and a third probability that each combined component belongs to each type; inputting the combined components in the design draft into a second image recognition model to identify a fourth probability that the combined components belong to each type; determining the weight associated with the component combination rules and the weight associated with the second image recognition model, performing a weighted summation of the third probability and the fourth probability that each combined component belongs to each type to obtain multiple probabilities, and determining the type indicated by the highest probability among the multiple probabilities as the type of the combined component.

[0059] Specifically, the second image recognition model refers to the trained deep learning model; the third probability refers to the probability of belonging to a certain type obtained through component combination rules, where the component combination rules are rules based on the fact that certain small components can be combined into complex components; the fourth probability refers to the probability of belonging to a certain type obtained after applying the second image recognition model, where the component combination rules refer to using the obtained local features to combine small components and local features into complex components through methods such as connection and overlay. Complex components can include index lists, step bars, pagination, etc. For example, a dropdown box is a complex component, consisting of a border, a search icon, and tooltip text. Single components and local features are combined according to the component combination rules, and the combined component is denoted as a combined component. The combined component is used as input and fed into both the component combination rules and the second image recognition model.

[0060] For example, for a certain component, the probabilities of recognizing two types, A and B, can be obtained by inputting the component combination rule and the second image recognition model, respectively. The probability of belonging to type A in the component combination rule is a1, and the probability of belonging to type B is b1; the probability of belonging to type A in the second image recognition model is a2, and the probability of belonging to type B is b2. The weights of the second image recognition model and the component combination rule are M and N, respectively. Then, the final probability of the component belonging to type A is N*a1+M*a2, and the probability of belonging to type B is N*b1+M*b2. After obtaining the two weighted probability values, the maximum value between the two is the type of the combined component.

[0061] Figure 3 This is a flowchart illustrating the determination of component types according to embodiments of this application. Figure 3As shown, after extracting the layer information from the design draft, interfering layers are removed through methods such as transparency, relative position and overlap, and edge detection. Based on the first image recognition model and component construction rules, small components are identified. The type of each small component is obtained through weighted summation and extreme value taking. Simultaneously, local features that were not initially identified as components are also identified. For example, buttons can be identified as radio buttons or checkboxes, while local features can be arrows or search icons. Small components and local features are combined using component combination rules to form complex components, i.e., composite components. These composite components are then used by the second image recognition model and component combination rules, employing weighted summation and extreme value taking to identify their types, ultimately yielding the types of each component.

[0062] Optionally, in the user interface testing method provided in this application embodiment, displaying the comparison results on the server's browser includes: constructing a table based on the comparison results of each component, the type of the component, the layer style information of the component, and the standard style information of the component, and displaying the constructed table in the browser; or packaging the comparison results of each component, the type of the component, the layer style information of the component, and the standard style information of the component separately to obtain packaging information, associating the packaging information with the corresponding component, and displaying the packaging information associated with the component after receiving a click event executed on the component.

[0063] To present the test results of the design drafts to users, on the one hand, the recorded design draft images, the types of components in the images, the layer position information of the components, the layer style information, and the comparison results are generated in the form of a table. At the same time, suggestions are made on whether the design of the component needs to be modified, and the suggestions are generated into the constructed table. The suggestions include mandatory modification of the specification level and recommended modification of the specification level. The table containing comparison results, type and other information is displayed in the browser.

[0064] On the other hand, in order to present the results more clearly, the information that needs to be displayed is saved and then displayed on the upper layer of the design draft in the form of a transparent box according to the layer position information. That is, each layer of the design draft is bound to packaged information that combines the comparison results of each component, the type of the component, the layer style information of the component, and the standard style information of the component. When the user clicks on the corresponding layer, the system receives the click request and displays the packaged information.

[0065] Figure 4 This is a timing diagram of an optional user interface testing method provided according to an embodiment of this application. Figure 4 As shown, the method includes:

[0066] After the designer submits the completed user interface design to the testing tool for standardized testing, the tester clicks the testing tool to start the inspection.

[0067] The testing tool first scans the submitted design drafts to obtain the design specification documents and establishes a preset comparison result table. Then, it locates the elements in the scanned design draft pages, obtains the layer position information and layer style information of the elements, and inputs the obtained information into a trained image recognition model to obtain the type of the corresponding component. The component type is assigned a component ID, and the component ID is used to search in the design specification documents to match the design specification content of the corresponding component. This achieves the comparison and detection of the component's style information with the design specification. The comparison results are stored in the preset comparison result table, a comparison report is generated, and finally, it is sent to the testers, thus realizing the testing effect of the design drafts.

[0068] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0069] This application also provides a user interface testing apparatus. It should be noted that the user interface testing apparatus of this application can be used to execute the user interface testing method provided in this application. The user interface testing apparatus provided in this application will be described below.

[0070] Figure 5 This is a schematic diagram of a testing apparatus with a user interface provided according to an embodiment of this application, such as... Figure 5 As shown, the device includes: an acquisition unit 50, an extraction unit 51, a determination unit 52, and a comparison unit 53.

[0071] The acquisition unit 50 is used to acquire the design specification file of the user interface and open the design draft of the user interface stored on the server. The design draft is a design draft composed of multiple components, each component contains at least one element, and the design specification file contains multiple design specification contents. Each design specification content contains the component ID of multiple types of components and the standard style of each component.

[0072] Extraction unit 51 is used to extract the layer position information and layer style information of each element in the design draft, determine the components in the design draft from the elements in the design draft, and determine the type of the component based on the layer position information and layer style information of the elements in the component.

[0073] Unit 52 is used to determine the target specification content that the design draft matches in the design specification file, and to determine the component ID according to the type of each component in the design draft, and to extract the standard style associated with each component ID from the target specification content.

[0074] The comparison unit 53 is used to compare the layer style information of each component with the standard style, obtain the comparison result, and display the comparison result on the browser of the server.

[0075] Optionally, in the user interface testing device provided in this application embodiment, the acquisition unit 50 includes: an acquisition module, used to acquire a compressed design draft package stored in the server, and decompress the compressed design draft package to obtain the design draft; and an execution module, used to execute a preset startup command on the server to load the design draft and open the design draft, wherein the preset startup command is used to call the browser driver to load the design draft into the browser.

[0076] Optionally, in the user interface testing device provided in this application embodiment, the extraction unit 51 includes: a traversal module, used to traverse the design draft, obtain the outline position coordinates of the layers of each element in the design draft, and obtain the layer position information of each layer; a trigger module, used to trigger the layer for an element containing a single layer, obtain the layer style information of the element, and determine the layer position information of the layer as the layer position information of the element, wherein the layer style information includes at least one of the following style information: the size of the layer, the color, and the text on the element; and an establishment module, used to establish the association relationship of multiple layers of the same element for an element containing multiple layers, and trigger the multiple layers of the same element respectively to obtain multiple layer style information, the multiple layer style information constitutes the layer style information of the element, and the layer position information of the multiple layers of the same element constitutes the layer position information of the element, wherein the element containing multiple layers is determined by: determining at least two layers with an inclusion relationship according to the layer position information of each layer, and determining the at least two layers with an inclusion relationship as the layers of the same element.

[0077] Optionally, in the user interface testing device provided in the embodiments of this application, the determining unit 52 includes: a first determining module, used to determine the type of a single component based on the layer position information and layer style information of the elements in the design draft; and a combining module, used to combine the single component and local features not identified as components into a combined component, and determine the type of the combined component.

[0078] Optionally, in the user interface testing device provided in this application embodiment, the determining unit 52 includes: a slicing module, used to slice the design draft according to the layer position information of the elements to obtain multiple sub-images, wherein each sub-image contains at least one element; a judging module, used to judge whether each sub-image conforms to the component construction rules, and if the sub-image does not conform to the component construction rules, determine that the sub-image does not contain a component; an analysis module, used to analyze the element layer style information in the sub-image according to the component construction rules if the sub-image conforms to the component construction rules, and obtain the components in the sub-image and the first probability that the components belong to each type; a first input module, used to input the layer style information of the elements in the sub-image into a first image recognition model, and identify the components in the sub-image and the second probability that the components belong to each type; and a second determining module, used to determine the weight associated with the component construction rules and the weight associated with the first image recognition model, and to perform a weighted summation of the first probability and the second probability that each component belongs to each type to obtain multiple probabilities, and to determine the type indicated by the highest probability among the multiple probabilities as the type of the component.

[0079] Optionally, in the user interface testing device provided in this application embodiment, the determining unit 52 includes: a combination module, used to combine and analyze the local features in the design draft and the identified single components according to the component combination rules, to obtain multiple combined components and a third probability that each combined component belongs to each type; a second input module, used to input the combined components in the design draft into a second image recognition model, to identify a fourth probability that the combined components belong to each type; and a third determining module, used to determine the weight associated with the component combination rules and the weight associated with the second image recognition model, to perform a weighted summation of the third probability and the fourth probability that each combined component belongs to each type, to obtain multiple probabilities, and to determine the type indicated by the highest probability among the multiple probabilities as the type of the combined component.

[0080] Optionally, in the user interface testing device provided in this application embodiment, the comparison unit 53 includes: a construction module, used to construct a table based on the comparison results of each component, the type of the component, the layer style information of the component, and the standard style information of the component, and display the constructed table in a browser; or a packaging module, used to package the comparison results of each component, the type of the component, the layer style information of the component, and the standard style information of the component respectively to obtain packaging information, associate the packaging information with the corresponding component, and display the packaging information associated with the component after receiving a click event executed on the component.

[0081] The user interface testing apparatus provided in this application embodiment includes an acquisition unit 50, which acquires a user interface design specification file and opens a user interface design draft stored on a server. The design draft consists of multiple components, each containing at least one element. The design specification file contains multiple design specification contents, each containing component IDs for multiple types of components and standard styles for each component. An extraction unit 51 extracts the layer position information and layer style information of each element in the design draft, determines the components in the design draft based on the elements, and determines the component type based on the layer position information and layer style information of the elements in the component. A determination unit 52 determines the target specification content that matches the design draft in the design specification file, determines the component ID based on the type of each component in the design draft, and extracts the standard style associated with each component ID from the target specification content. The comparison unit 53 is used to compare the layer style information of each component with the standard style, obtain the comparison result, and display the comparison result on the browser of the server. This solves the problems of low efficiency and low accuracy of manually comparing and testing components in related technologies. By using an image recognition model to distinguish the types of components in the design draft, and then extracting the standard styles of the corresponding types of components from the design specifications, the layer styles of the components are compared with the standard styles, thereby achieving the effect of accurately and efficiently testing the components in the design draft.

[0082] The aforementioned user interface testing device includes a processor and a memory. The aforementioned acquisition unit 50, extraction unit 51, determination unit 52, and comparison unit 53 are all stored in the memory as program units. The processor executes the aforementioned program units stored in the memory to achieve the corresponding functions.

[0083] The processor contains a kernel, which retrieves the corresponding program units from memory. One or more kernels can be configured, and adjusting kernel parameters can address the inefficiencies and low accuracy of manual component comparison and detection in related technologies.

[0084] The memory may include non-permanent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0085] This application also provides a computer storage medium for storing a program, wherein the program, when running, controls the device where the non-volatile storage medium is located to execute a user interface testing method.

[0086] This application also provides an electronic device. Figure 6 This is a schematic diagram of an electronic device provided according to an embodiment of this application, such as... Figure 6 As shown, the electronic device 60 includes a processor and a memory; the memory stores computer-readable instructions, and the processor executes these computer-readable instructions, wherein the execution of the computer-readable instructions performs a user interface testing method. The electronic device in this document can be a server, PC, PAD, mobile phone, etc.

[0087] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0088] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0089] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0090] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0091] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0092] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0093] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0094] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0095] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A method for testing a user interface, characterized in that, include: Obtain the user interface design specification file and open the user interface design draft stored on the server. The design draft is composed of multiple components, each component contains at least one element, and the design specification file contains multiple design specification contents. Each design specification content contains the component ID of multiple types of components and the standard style of each component. Extract the layer position information and layer style information of each element in the design draft, determine the components in the design draft based on the elements in the design draft, and determine the type of the component based on the layer position information and layer style information of the elements in the component; In the design specification file, the target specification content that the design draft matches is determined, and the component ID is determined according to the type of each component in the design draft. The standard style associated with each component ID is extracted from the target specification content. The layer style information of each component is compared with the standard style to obtain the comparison result, and the comparison result is displayed on the browser of the server. The process of determining the components in the design draft by identifying the elements in the design draft, and determining the type of the component based on the layer position information and layer style information of the elements in the component, includes: The type of a single component is determined based on the layer position information and layer style information of the elements in the design draft; The single component and local features not identified as components are combined into a combined component, and the type of the combined component is determined.

2. The test method according to claim 1, characterized in that, Obtain the user interface design specification file and open the user interface design draft stored on the server, including: Obtain the compressed design file stored on the server, and decompress the compressed design file to obtain the design file; A preset startup command is executed on the server to load the design draft and open the design draft. The preset startup command is used to call the browser driver to load the design draft into the browser.

3. The test method according to claim 1, characterized in that, Extracting the layer position information and layer style information of each element in the design draft includes: Traverse the design draft to obtain the outline position coordinates of the layers of each element in the design draft, and obtain the layer position information of each layer; For an element containing a single layer, the layer is triggered, the layer style information of the element is obtained, and the layer position information of the layer is determined as the layer position information of the element. The layer style information includes at least one of the following style information: the size of the layer, the color, and the text on the element. For an element containing multiple layers, establish the association relationship between the multiple layers of the same element, and trigger the multiple layers of the same element respectively to obtain multiple layer style information. The multiple layer style information constitutes the layer style information of the element, and the layer position information of the multiple layers of the same element constitutes the layer position information of the element. The element containing multiple layers is determined in the following way: determine at least two layers with an inclusion relationship based on the layer position information of each layer, and determine the at least two layers with an inclusion relationship as layers of the same element.

4. The test method according to claim 1, characterized in that, The type of a single component is determined based on the layer position information and layer style information of the elements in the design draft, including: The design draft is sliced ​​according to the layer position information of the element to obtain multiple sub-images, wherein each sub-image contains at least one element; For each subgraph, determine whether the subgraph conforms to the component construction rules. If the subgraph does not conform to the component construction rules, determine that the subgraph does not contain a component. If the subgraph conforms to the component construction rules, the element layer style information in the subgraph is analyzed according to the component construction rules to obtain the components in the subgraph and the first probability of each component belonging to each type; The layer style information of the elements in the sub-image is input into the first image recognition model to identify the components in the sub-image and the second probability of each component belonging to each type; The weights associated with the component construction rules and the weights associated with the first image recognition model are determined. The first probability and the second probability of each component belonging to each type are weighted and summed to obtain multiple probabilities. The type indicated by the highest probability among the multiple probabilities is determined as the type of the component.

5. The test method according to claim 1, characterized in that, Combining the single components and local features not identified as components into a combined component, and determining the type of the combined component includes: Based on the component combination rules, the local features in the design draft and the identified single components are combined and analyzed to obtain multiple combined components and the third probability of each combined component belonging to each type. The combined components in the design draft are input into the second image recognition model to identify the fourth probability of each type of the combined components; The weights associated with the component combination rules and the weights associated with the second image recognition model are determined. The third and fourth probabilities of each combined component belonging to each type are weighted and summed to obtain multiple probabilities. The type indicated by the highest probability among the multiple probabilities is determined as the type of the combined component.

6. The test method according to claim 1, characterized in that, Displaying the comparison results on the browser of the server includes: A table is constructed based on the comparison results of each component, the component type, the component layer style information, and the component standard style information, and the constructed table is displayed in the browser; or The comparison results, component type, component layer style information, and component standard style information of each component are packaged separately to obtain package information. The package information is then associated with the corresponding component, and the package information associated with the component is displayed after a click event is received on the component.

7. A user interface testing device, characterized in that, include: The acquisition unit is used to acquire the design specification file of the user interface and open the design draft of the user interface stored in the server. The design draft is a design draft composed of multiple components, each component contains at least one element, and the design specification file contains multiple design specification contents. Each design specification content contains the component ID of multiple types of components and the standard style of each component. The extraction unit is used to extract the layer position information and layer style information of each element in the design draft, determine the components in the design draft from the elements in the design draft, and determine the type of the component based on the layer position information and layer style information of the elements in the component. The determining unit is used to determine the target specification content that the design draft matches in the design specification file, and to determine the component ID according to the type of each component in the design draft, and to extract the standard style associated with each component ID from the target specification content; The comparison unit is used to compare the layer style information of each component with the standard style, obtain the comparison result, and display the comparison result on the browser of the server. The determining unit includes: a first determining module, used to determine the type of a single component based on the layer position information and layer style information of the elements in the design draft; and a combining module, used to combine the single component and local features not identified as components into a combined component, and determine the type of the combined component.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored computer program, wherein, when the computer program is executed, it controls the device on which the computer-readable storage medium is located to perform the test method for the user interface as described in any one of claims 1 to 6.

9. An electronic device, characterized in that, It includes one or more processors and a memory, the memory being used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the user interface testing method according to any one of claims 1 to 6.

Images