UI element state recognition method and device and UI test method

Abstract

The invention provides a UI element state recognition method and device and a UI test method. The UI element state recognition method comprises the steps that an element image of a to-be-recognized UI element is obtained according to a test case; according to the test case and the element image, obtaining an element type of the UI element to be identified; according to the element type, obtaining a target feature of the UI element to be identified from the element image; and obtaining the state of the UI element to be identified according to the target feature. According to the method, the state of the current UI element in the element image is judged by analyzing the features of different types of UI elements, and as only feature extraction and comparison need to be carried out, analysis and processing of the image can be completed in a relatively short time, and the efficiency is improved; the method does not depend on fixed rules or models any more, analysis is carried out through multi-dimensional features such as connected domains, colors and shapes, the accuracy can be improved, and high flexibility is achieved.

Description

Technical Field

[0001] The present invention relates to the field of software testing technology, and in particular to a UI element state recognition method and device, and a UI testing method. Background Art

[0002] At present, when testing the user interface (UI), the commonly used UI automation test needs to write test case codes according to the user's operation, so as to judge whether the expected and actual results are consistent through the test case codes.

[0003] Taking the in-vehicle central control system as an example, during the testing process of the in-vehicle central control system, it is necessary to confirm the elements and element attributes of the Android page through the solution of the automation framework. During the testing process, it was found that some UI elements cannot be directly confirmed through the automation framework. This is because these UI elements are not original UI elements. Many of them are customized UI elements, so there are no related switch attributes, so it is impossible to obtain the true status of all UI elements only through some attributes of UI elements. For example, the on and off state of a switch, or the level of switch on, or the special state of a button only changes in the UI interface.

[0004] by Figure 1 and Figure 2 For example, from the page structure, the marked positions are all written with the word "High", but the corresponding clicked and unclicked states on the right are exactly the same. In this case, we need to analyze the image more deeply to confirm whether the button is currently clicked or not.

[0005] In order to solve the above problems, the common practice in the prior art is to obtain a UI image and then compare it with a target reference image to obtain a test result. This solution has a slow processing speed and poor flexibility. It requires the preset of a large number of target reference images. For UI interfaces with multiple themes, the number of target reference images will increase exponentially. Moreover, once the UI interface changes, the target reference image needs to be additionally modified, which is another relatively large workload. Summary of the invention

[0006] In view of the above defects of the prior art, the present invention provides a UI element state recognition method and device, and a UI testing method to solve the technical problems of slow UI testing speed and poor flexibility.

[0007] To achieve the above-mentioned purpose and other related purposes, the present invention provides a UI element status identification method, comprising: obtaining an element image of a UI element to be identified according to a test case; obtaining an element type of the UI element to be identified according to the test case and the element image; obtaining a target feature of the UI element to be identified from the element image according to the element type; and obtaining the status of the UI element to be identified according to the target feature.

[0008] In one embodiment of the present invention, an element image of a UI element to be identified is obtained according to a test case, including: obtaining position information of the UI element to be identified according to the test case, and obtaining an image of a corresponding area in the UI interface according to the position information to obtain an element image of the UI element to be identified.

[0009] In one embodiment of the present invention, the element type of the UI element to be identified is obtained according to the test case and the element image, including: obtaining the test content according to the test case, and determining whether the test content contains level information: if yes, the type of the UI element to be identified is a multi-state switching button; if not, obtaining the type of the UI element to be identified according to the element image.

[0010] In one embodiment of the present invention, the type of the UI element to be identified is obtained based on the element image, including: obtaining the aspect ratio of the element image based on the element image; judging whether the aspect ratio of the element image is greater than a preset ratio: if yes, the type of the UI element to be identified is a slider; if no, the type of the UI element to be identified is a switch button.

[0011] In one embodiment of the present invention, when the type of the UI element to be identified is a switch button, the target features of the UI element to be identified are obtained from the element image, including: according to the color of each pixel in the element image, obtaining the main color value and the main color proportion in the element image as the target features of the UI element to be identified.

[0012] In one embodiment of the present invention, when the type of the UI element to be identified is a multi-state switch button, the target features of the UI element to be identified are obtained from the element image, including: obtaining a number of connected domains based on the element image; based on the several connected domains, obtaining one or more of the number of connected domains, the shape of the connected domains, the main color value of the connected domains, and the size of the connected domains as the target features of the UI element to be identified.

[0013] In one embodiment of the present invention, a plurality of connected domains are obtained according to the element image, including: performing grayscale conversion and binarization processing on the element image in sequence to obtain a binary image of the identified UI element; and using a connected domain labeling algorithm to process the binary image of the UI element to be identified to obtain a plurality of the connected domains.

[0014] In one embodiment of the present invention, when the type of the UI element to be identified is a sliding bar, the target features of the UI element to be identified are obtained from the element image, including: obtaining the slider color value and track background color of the sliding bar according to the color of each pixel in the element image; and obtaining the slider position of the sliding bar according to the slider color value and the track background color as the target features of the UI element to be identified.

[0015] To achieve the above-mentioned purpose and other related purposes, the present invention also provides a UI element state recognition device, including: an image acquisition unit, used to acquire an element image of a UI element to be recognized according to a test case; a type recognition unit, used to obtain the element type of the UI element to be recognized according to the test case and the element image; a feature extraction unit, used to acquire the target feature of the UI element to be recognized from the element image according to the element type; and a state recognition unit, used to obtain the state of the UI element to be recognized according to the target feature.

[0016] To achieve the above-mentioned purpose and other related purposes, the present invention also provides a UI testing method, comprising: executing test cases to simulate operations on UI elements to be identified; using the above-mentioned UI element state identification method to obtain the actual state of the UI elements to be identified; and obtaining test results based on the actual state and expected state of the UI elements to be identified.

[0017] Beneficial effects of the present invention: The present invention proposes a UI element state recognition method and device, and a UI testing method, wherein the UI element state recognition method determines the state of the current UI element in the element image by flexibly analyzing the features of different types of UI elements. Since only feature extraction and comparison are required, image analysis and processing can be completed in a shorter time, thereby improving efficiency. At the same time, the method no longer relies on fixed rules or models, but analyzes through multi-dimensional features such as connected domains, colors, and shapes. On the one hand, it can improve the recognition accuracy, and on the other hand, it can quickly adjust when the UI interface changes, thereby having strong flexibility. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0019] Figure 1 A schematic diagram of the UI interface and properties when the button "High" is not clicked in the vehicle-mounted central control system provided by an embodiment of the present invention;

[0020] Figure 2 A schematic diagram of the UI interface and properties when the button "High" is clicked in the vehicle-mounted central control system provided by an embodiment of the present invention;

[0021] Figure 3 A flowchart of a method for identifying a UI element state according to an embodiment of the present invention;

[0022] Figure 4 A feature extraction flow chart of a multi-state switch button provided by an embodiment of the present invention;

[0023] Figure 5 A flow chart of connected domain processing provided by an embodiment of the present invention;

[0024] Figure 6 A flow chart of feature extraction of a slider provided by an embodiment of the present invention;

[0025] Figure 7 A schematic diagram of a UI element state recognition device provided by an embodiment of the present invention;

[0026] Figure 8 A flowchart of a UI testing method provided by an embodiment of the present invention.

[0027] Explanation of the reference numerals: 101, image acquisition unit; 102, type recognition unit; 103, feature extraction unit; 104, state recognition unit. DETAILED DESCRIPTION

[0028] The following describes the embodiments of the present invention through specific specific embodiments, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. It should be noted that the following embodiments and the features in the embodiments can be combined with each other without conflict. In addition to the specific methods, equipment, and materials used in the embodiments, according to the technical personnel in this technical field's grasp of the prior art and the records of the present invention, any methods, equipment, and materials of the prior art that are similar or equivalent to the methods, equipment, and materials described in the embodiments of the present invention can also be used to implement the present invention.

[0029] It should be understood that the terms used in the embodiments of the present invention are for describing specific embodiments rather than for limiting the scope of protection of the present invention. Unless otherwise defined, all technical and scientific terms used in the present invention have the same meaning as those generally understood by those skilled in the art.

[0030] In the following description, numerous details are discussed to provide a more thorough explanation of the embodiments of the present invention. However, it is obvious to those skilled in the art that the embodiments of the present invention can be implemented without these specific details. In some of the embodiments, well-known structures and devices are shown in the form of block diagrams rather than in detail to avoid making the embodiments of the present invention difficult to understand.

[0031] The flowchart and block diagram in the accompanying drawings illustrate the system architecture, functions and operations that may be implemented according to the methods and computer program products of various embodiments disclosed in the present invention. In this regard, each box in the flowchart or block diagram can represent a module, a program segment, or a part of a code, and the module, the program segment, or a part of the code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or the flowchart, and the combination of the boxes in the block diagram and / or the flowchart can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0032] See also Figure 3 , Figure 3 A method for identifying a UI element state is provided in one embodiment of the present invention, and includes steps S100 to S400.

[0033] Step S100, obtain the element image of the UI element to be identified according to the test case. Test cases are commonly used in the software industry, and are generally used to check whether the software UI interface, core functions, etc. are running normally. Specifically, by executing the test case, the software performs the corresponding operation, so that the actual response results of the software interface and function can be compared with the expected results to determine whether the software is running normally. Taking the vehicle-mounted central control system as an example, the test case may be to turn on a switch button, adjust the level of a button, and so on. In this embodiment, the test case includes at least some methods for simulating clicking on a UI element in the UI interface. At this time, the element image of the UI element to be identified can be obtained according to the UI element that needs to be simulated clicked in the test case.

[0034] In a specific embodiment of the present invention, step S100 includes: obtaining the position information of the UI element to be identified according to the test case, and obtaining the image of the corresponding area in the UI interface according to the position information to obtain the element image of the UI element to be identified. The test case contains the ID of the UI element that needs to be simulated clicked (i.e., the UI element to be identified), and the position information of the UI element can be obtained based on the ID information. Taking the switch button as an example, the position information can be, for example, the horizontal and vertical coordinates of the upper left corner of the UI element and the width and height of the UI element. After obtaining the position information, the image of the corresponding area can be captured according to the position information to obtain the element image of the UI element to be identified.

[0035] Step S200: Obtain the element type of the UI element to be identified according to the test case and the element image. Since many UI elements are customized by users, and the target features to be extracted for different types of UI elements are also different, in this step, the element type of the UI element to be identified is first obtained.

[0036] In a specific embodiment of the present invention, the element type of the UI element may be, for example, a switch button, a multi-state switch button, a slider, and the like. The following takes the vehicle-mounted central control system as an example to introduce these UI elements in detail. The switch button corresponds to the two states of on and off. For example, the on and off buttons of a function in the UI interface are switch buttons. A multi-state switch button is a user interface element that can switch between multiple preset states, and usually represents different functions or modes with different visual appearances. It corresponds to multiple levels of states. When the button is pressed, the state of the button changes in the following order, for example: off → level one → level two → level three → off. Common multi-state switch buttons include seat massage levels, heating levels, and the like. A slider is a component in the user interface used to select a numerical range or a single value, usually consisting of a track and a slidable slider, such as volume adjustment, brightness adjustment, and the like.

[0037] In a specific embodiment of the present invention, step S200 includes: obtaining test content according to the test case, and determining whether the test content contains level information: if yes, the type of the UI element to be identified is a multi-state switch button; if not, obtaining the type of the UI element to be identified according to the element image. It can be understood that among the three UI elements exemplified above, only the multi-state switch button involves level information. Therefore, in this step, whether the type of the UI element to be identified is a multi-state switch button is determined based on whether the test content contains level information.

[0038] In a specific embodiment of the present invention, the type of the UI element to be identified is obtained according to the element image, including: according to the element image, obtaining the aspect ratio of the element image; judging whether the aspect ratio of the element image is greater than a preset ratio: if yes, the type of the UI element to be identified is a slider; if not, the type of the UI element to be identified is a switch button. The preset ratio in this step can be set according to actual conditions. Generally speaking, the aspect ratio of the slider is larger, and the aspect ratio of the switch button is smaller, so the type of the UI element to be identified can be judged as a slider or a switch button according to the aspect ratio of the element image.

[0039] In addition to judging the type of UI element based on the comparison between the aspect ratio of the element image and the preset ratio, the color type can also be judged. The color type can be judged before or after the aspect ratio of the element image, depending on the actual situation.

[0040] In a specific embodiment of the present invention, Figure 1 As shown, there are only two colors in the slider and the switch button, so the judgment logic can be changed to:

[0041] According to the element image, the color type and the aspect ratio of the element image are obtained;

[0042] Determine whether the number of colors in the element image is equal to the preset number:

[0043] If yes, further determine whether the aspect ratio of the element image is greater than the preset ratio:

[0044] If yes, the type of the UI element to be identified is a slider;

[0045] If not, the type of the UI element to be identified is a switch button.

[0046] In this embodiment, the determination of the color type is performed before the determination of the aspect ratio of the element image.

[0047] In a specific embodiment of the present invention, the slider is a gradient color with many color types, so the judgment logic can be changed to:

[0048] According to the element image, the aspect ratio of the element image is obtained;

[0049] Determine whether the aspect ratio of the element image is greater than the preset ratio:

[0050] If yes, the type of the UI element to be identified is a slider;

[0051] If not, further determine whether the color types in the element image are equal to the preset number:

[0052] If yes, the type of the UI element to be identified is a switch button.

[0053] In this embodiment, the determination of the color type is performed after the determination of the aspect ratio of the element image.

[0054] It should be noted that the above judgment logic is only an illustration, and different judgment logics can be used to obtain the type of the UI element to be identified for different UI designs. Alternatively, the mapping relationship between the ID of the UI element and its type can be saved in a table, and then the type of the UI element to be identified can be obtained by table lookup based on the ID of the UI element to be identified.

[0055] Step S300: Obtain target features of the UI element to be identified from the element image according to the element type. Different target features can be extracted for different types of UI elements. The following will continue to describe the target feature extraction process in detail using a switch button, a multi-state switch button, and a slide bar as examples.

[0056] In a specific embodiment of the present invention, when the type of the UI element to be identified is a switch button, the target feature of the UI element to be identified is obtained from the element image, including: obtaining the main color value and the main color proportion in the element image according to the color of each pixel in the element image, as the target feature of the UI element to be identified. For the switch button, its main feature is the main color. For example, taking the "high" button as an example, in Figure 1 In the , when the button is not pressed, its main color is white. Figure 2 , when the button is pressed, its main color is blue ( Figure 2 In the figure, the image is grayscaled, so it is displayed as gray, but it is actually blue. In addition to the main color value, the target feature also includes the main color ratio, which can increase the number of features to ensure the accuracy of subsequent state recognition.

[0057] See also Figure 4 In a specific embodiment of the present invention, when the type of the UI element to be identified is a multi-state switch button, the target features of the UI element to be identified are obtained from the element image, including steps S311 and S312.

[0058] Step S311, obtain several connected domains according to the element image. Since the multi-state switch button needs to indicate multiple levels in the element image, the UI of the multi-state switch button will be more complicated, generally composed of more icons. Taking the seat massage button as an example, its corresponding element image may include a seat block and three slash blocks. The seat block is used to indicate that the button is used to adjust the seat massage level, and the three slash blocks correspond to the massage levels 1, 2, and 3, respectively. Therefore, the element image of the multi-state switch button will be more complicated. In order to obtain its target features, it is necessary to process the element image first to obtain several connected domains.

[0059] See also Figure 5 In a specific embodiment of the present invention, step S311 includes: S3111, grayscale conversion and binarization processing are performed on the element image in sequence to obtain a binary image for identifying UI elements; S3112, the binary image of the UI element to be identified is processed using a connected domain labeling algorithm to obtain a number of connected domains. The element image is generally a color image in RGB or other formats. After grayscale processing and binarization processing, the final binary image contains only black and white, which reduces the complexity of extracting connected domains. Grayscale processing and binarization processing are both conventional steps in image processing technology, and will not be expanded in detail here.

[0060] The connected domain labeling algorithm is an algorithm used to identify and distinguish connected pixel areas in computer vision and image processing. These algorithms are widely used in image analysis, machine vision, medical imaging and other fields. The connected domain labeling algorithm can be, for example, a Two-Pass algorithm, a seed filling algorithm or a contour tracing algorithm. After being processed by the connected domain labeling algorithm, several connected domains can be obtained. At this time, some connected domains that do not meet the requirements, such as image noise, can be excluded according to the size of the connected domain. Finally, the remaining connected domains are used as the output of step S311.

[0061] Step S312: Based on the connected domains, one or more of the number of connected domains, the shape of the connected domains, the main color value of the connected domains, and the size of the connected domains are obtained as target features of the UI elements to be identified. After the connected domains are obtained, the target features can be obtained based on the parameters of the connected domains.

[0062] The number of connected domains corresponds to the number of connected domains obtained. Take the seat massage button as an example: when the button is in the off state, the number of its connected domains is only 1, that is, 1 seat block; when the button is in gear 1, the number of its connected domains is 2, that is, 1 seat block and 1 slash block; when the button is in gear 2, the number of its connected domains is 3, that is, 1 seat block and 2 slash blocks; when the button is in gear 3, the number of its connected domains is 4, that is, 1 seat block and 3 slash blocks. Therefore, in this embodiment, the state of the UI element can be obtained according to the number of connected domains.

[0063] In addition to the number of connected domains, features such as the shape of the connected domain, the main color value of the connected domain, and the size of the connected domain can all be used as the basis for state judgment. The specific selection of one or several features can be set according to the actual situation. It can be understood that the more target features there are, the more accurate the state judgment will be.

[0064] See also Figure 6In a specific embodiment of the present invention, when the type of the UI element to be identified is a slider, the target feature of the UI element to be identified is obtained from the element image, including: S321, according to the color of each pixel in the element image, the slider color value and the track background color of the slider are obtained; S322, according to the slider color value and the track background color, the slider position of the slider is obtained as the target feature of the UI element to be identified. The slider position corresponds to the opening ratio of the slider, which ranges from 0% to 100%. The slider position can be, for example, 55%, 70%, etc.

[0065] Step S400, according to the target feature, obtain the state of the UI element to be identified. After obtaining the target feature of the UI element to be identified, the state of the UI element to be identified can be determined by simple logical judgment. For example, in the above, we mentioned the seat massage button as an example. When the number of connected domains is 1, 2, 3, and 4, the corresponding seat massage button states are off, gear 1, gear 2, and gear 3, respectively. Alternatively, if the main color value of the UI element "high" is blue and the main color accounts for 90%, then the state of the "high" button is on. It can be seen that when determining the state of the UI element to be identified by the target feature, there is no need to use complex image matching or learning models, only simple logical judgment is required.

[0066] The simple logic described in step S400 may be packaged as a template for subsequent use and adjustment.

[0067] Table 1: Example of state decision logic template

[0068]

[0069] Table 2: Example of execution and results of state decision logic template

[0070]

[0071] Please refer to Table 1 and Table 2. In a specific embodiment of the present invention, the first column numbers 1 to 4 in Table 1 are the usage methods of the templates, and the first column numbers 5 to 8 are the parameters and methods actually used. When using, you only need to configure the corresponding image path and the parameters that need to be confirmed. Table 2 shows the execution status and results of each template in Table 1.

[0072] The following is a brief description of each template: (1) For color_between, it refers to whether there is a color between two execution color areas in the image; (2) For the state of the button after clicking, it means that if the button after clicking is specified as a color range, once the color of the image is within this color range, the button is clicked; (3) For the number of connected domains of color areas, it refers to how many areas of specified colors appear on the image to determine the specific state of the icon.

[0073] In general, through the analysis of the above embodiments, the complete process from image acquisition to state determination is demonstrated. This method has high flexibility, fast processing speed, and high accuracy. Although the above embodiments are mostly explained using the vehicle-mounted central control system as an example, in fact, the above scheme can be applied to all industries that require UI testing. The above method can effectively identify and judge the button status through meticulous preprocessing and precise feature extraction, and then respond quickly. This method can flexibly adapt to different interfaces and button styles, and is easy to optimize and adjust through feedback mechanisms.

[0074] It should be noted that the step division of the above methods is only for clear description. When implemented, they can be combined into one step or some steps can be split and decomposed into multiple steps. As long as they contain the same logical relationship, they are all within the scope of protection of this patent; adding insignificant modifications to the algorithm or process or introducing insignificant designs without changing the core design of the algorithm and process are all within the scope of protection of this patent.

[0075] See also Figure 7 , Figure 7 A UI element state recognition device provided for an embodiment of the present invention includes an image acquisition unit 101, a type recognition unit 102, a feature extraction unit 103 and a state recognition unit 104, wherein the image acquisition unit 101 is used to acquire an element image of a UI element to be recognized according to a test case; the type recognition unit 102 is used to obtain the element type of the UI element to be recognized according to the test case and the element image; the feature extraction unit 103 is used to obtain a target feature of the UI element to be recognized from the element image according to the element type; and the state recognition unit 104 is used to obtain the state of the UI element to be recognized according to the target feature.

[0076] It should be noted that the UI element state recognition device of this embodiment is a device corresponding to the above-mentioned UI element state recognition method, and the functional modules in the UI element state recognition device may correspond to the corresponding steps in the UI element state recognition method. The UI element state recognition device of this embodiment can be implemented in conjunction with the UI element state recognition method, that is, in the absence of conflict, the relevant technical details mentioned in the UI element state recognition method of the above-mentioned embodiment can also be applied to the UI element state recognition device of this embodiment.

[0077] See also Figure 8 , Figure 8 A UI testing method provided for an embodiment of the present invention includes the following steps: S1, executing a test case to simulate the operation of a UI element to be identified; S2, using the above-mentioned UI element state identification method to obtain the actual state of the UI element to be identified; S3, obtaining a test result based on the actual state and expected state of the UI element to be identified. The UI testing method is basically similar to the conventional testing method, except that in step S2, in the prior art, the actual state of the UI element to be identified is mostly obtained by image matching, while this embodiment is different in that it obtains the actual state of the UI element to be identified by extracting features and matching the features.

[0078] The above embodiments are merely illustrative of the principles and effects of the present invention, and are not intended to limit the present invention. Anyone familiar with the art may modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by a person of ordinary skill in the art without departing from the spirit and technical concept disclosed by the present invention shall still be covered by the claims of the present invention.

Claims

1. A method for identifying UI element status, characterized in that: include: Obtain element images of UI elements to be identified according to test cases; Obtaining the element type of the UI element to be identified according to the test case and the element image; According to the element type, acquiring a target feature of the to-be-identified UI element from the element image; According to the target feature, the state of the UI element to be identified is obtained.

2. The UI element state recognition method according to claim 1, characterized in that: Obtain element images of the UI elements to be identified according to the test case, including: The position information of the UI element to be identified is obtained according to the test case, and the image of the corresponding area in the UI interface is obtained according to the position information to obtain the element image of the UI element to be identified.

3. The UI element state recognition method according to claim 1, characterized in that: Obtaining the element type of the UI element to be identified according to the test case and the element image, including: Obtaining test content according to the test case, and determining whether the test content contains level information: If yes, the type of the UI element to be identified is a multi-state switch button; If not, the type of the UI element to be identified is obtained according to the element image.

4. The method for identifying the state of a UI element according to claim 3, characterized in that: Obtaining the type of the UI element to be identified according to the element image includes: According to the element image, obtaining the aspect ratio of the element image; Determine whether the aspect ratio of the element image is greater than a preset ratio: If yes, the type of the UI element to be identified is a slider; If not, the type of the UI element to be identified is a switch button.

5. The method for identifying UI element status according to claim 1, characterized in that: When the type of the UI element to be identified is a switch button, acquiring a target feature of the UI element to be identified from the element image includes: According to the color of each pixel in the element image, the main color value and the main color proportion in the element image are obtained to serve as the target feature of the UI element to be identified.

6. The UI element state recognition method according to claim 1, characterized in that: When the type of the UI element to be identified is a multi-state switch button, acquiring a target feature of the UI element to be identified from the element image includes: Obtaining a plurality of connected domains according to the element image; According to the connected domains, one or more of the number of connected domains, the shape of the connected domains, the main color value of the connected domains, and the size of the connected domains are obtained as target features of the UI elements to be identified.

7. The method for identifying the state of a UI element according to claim 6, characterized in that: According to the element image, several connected domains are obtained, including: Performing grayscale conversion and binarization processing on the element image in sequence to obtain a binary image of the identified UI element; The binary graph of the UI element to be identified is processed using a connected domain labeling algorithm to obtain a plurality of connected domains.

8. The method for identifying the state of a UI element according to claim 1, characterized in that: When the type of the UI element to be identified is a slider, acquiring a target feature of the UI element to be identified from the element image includes: According to the color of each pixel in the element image, the slider color value and the track background color of the slider are obtained; The slider position of the slide bar is obtained according to the slider color value and the track background color to serve as the target feature of the UI element to be identified.

9. A UI element state recognition device, characterized in that: include: An image acquisition unit, used to acquire an element image of a UI element to be identified according to a test case; A type identification unit, used to obtain the element type of the UI element to be identified according to the test case and the element image; A feature extraction unit, configured to obtain a target feature of the UI element to be identified from the element image according to the element type; The state recognition unit is used to obtain the state of the UI element to be recognized according to the target feature.

10. A UI testing method, characterized in that: include: Execute test cases to simulate operations on the UI elements to be identified; Obtaining the actual state of the UI element to be identified by using the UI element state identification method according to any one of claims 1 to 8; A test result is obtained according to the actual state and the expected state of the UI element to be identified.

Images