UI automatic test method and test system, and storage medium

Abstract

The invention provides a UI automatic test method and test system and a storage medium. The method comprises the steps that an optical character recognition result of a target text and a to-be-tested UI is obtained, and the optical character recognition result comprises a plurality of elements and element texts, coordinate frames and visual features of all the elements; obtaining the semantic similarity between the target text and each element text; obtaining attribute similarity between the target text and each element based on the visual features of each element; according to the semantic similarity and the attribute similarity, elements, corresponding to the target text, in the optical character recognition result are determined to serve as target elements; taking the position of the coordinate frame of the target element in the UI as a target position; and generating a presentation image which highlights the target position and / or the target element. Therefore, fuzzy matching based on semantics can be realized, multi-modal features including text semantics and visual features are fused, and the robustness and accuracy of UI element positioning are improved in combination with context sensing.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of UI (User Interface) testing, in particular to a UI automatic testing method and system, and a storage medium. BACKGROUND

[0002] Element positioning is a key item in UI automatic testing, which mainly obtains a target element in a UI to be tested for performing a certain function to simulate user operation on the target element. In existing UI automatic testing, element positioning is mainly achieved by means of precise text matching, fixed coordinates or traditional UI attributes (such as ID, class name, etc.). The precise text matching method mainly determines the position of an element in a UI according to complete matching of a text obtained by OCR (Optical Character Recognition) (also referred to as OCR text) and target text content. However, the precise text matching method lacks semantic understanding ability, and cannot achieve accurate matching of elements when the target text and the OCR text are translated (for example, "login" and "Sign In"), synonymous (for example, "login" and "enter") or variant. The fixed coordinates or traditional UI attribute method is sensitive to UI layout changes, and will easily lead to positioning failure once the coordinates or types of elements are slightly changed. In addition, the existing element positioning method only relies on a single modal feature, such as only relying on text information, without combining other features (such as type, shape and position, etc. visual features) of UI elements for comprehensive judgment, resulting in low positioning accuracy. SUMMARY

[0003] In view of this, the present application provides a UI automatic testing method and system, and a storage medium, which can improve the problem of low element positioning accuracy of existing UI automatic testing.

[0004] The UI automatic testing method provided by the present application comprises: obtaining a target text and an optical character recognition result of a UI to be tested, the optical character recognition result comprising a plurality of elements, and element text, a coordinate frame and visual features of each element; obtaining semantic similarity between the target text and each element text; obtaining attribute similarity between the target text and each element based on the visual features of each element; determining that the target text corresponds to an element in the optical character recognition result as a target element according to the semantic similarity and the attribute similarity; taking a position of the coordinate frame of the target element in the UI as a target position; generate a presentation image highlighting the target position and / or the target element.

[0005] Optionally, the semantic similarity between the target text and each element text is obtained, including: obtaining a semantic vector of the target text; traversing each element of the optical character recognition result, and calculating the semantic similarity between the semantic vector of the target text and the element text of each element; Optionally, the semantic similarity between the target text and each element text is obtained, including: obtaining a semantic vector of the target text and a semantic vector of each element text; traversing each element of the optical character recognition result, and calculating the semantic similarity between the semantic vector of the target text and the semantic vector of each element text.

[0006] Optionally, the attribute similarity between the target text and each element is obtained based on the visual features of each element, including: identifying a keyword contained in the target text; determining a target attribute corresponding to the target text according to the keyword; obtaining the attribute similarity according to the target attribute and the visual features of each element.

[0007] Optionally, the target element is determined according to the semantic similarity and the attribute similarity, including: firstly scoring each element according to the semantic similarity; secondly scoring each element according to the attribute similarity; determining a first weight coefficient of the semantic similarity and a second weight coefficient of the attribute similarity; calculating the sum of the product of the first score and the first weight coefficient and the product of the second score and the second weight coefficient as the total score of each element; taking an element with the highest total score as the target element.

[0008] Optionally, the optical character recognition result further includes a confidence degree of each element; and the target element is determined according to the semantic similarity and the attribute similarity, including: firstly scoring each element according to the semantic similarity; secondly scoring each element according to the attribute similarity; determining a first weight coefficient of the semantic similarity, a second weight coefficient of the attribute similarity, and a third weight coefficient of the confidence degree of each element. computing a sum of a product of the first score and the first weight coefficient, a product of the second score and the second weight coefficient, and a product of the confidence and the third weight coefficient as a total score of each element; taking one element with the highest total score as a target element.

[0009] Optionally, the method further comprises: presetting a reference threshold; judging whether the highest total score is greater than or equal to the reference threshold; if yes, taking one element with the highest total score as a target element; if no, determining that the target element is not found this time.

[0010] Optionally, the first weight coefficient is greater than the second weight coefficient.

[0011] Optionally, the first weight coefficient is greater than the third weight coefficient.

[0012] Optionally, the visual feature comprises a combination of type, shape and position, and the type comprises a combination of a function menu button, an icon button and a label.

[0013] The application provides a test system, comprising a processor and a memory, wherein the memory stores a UI automation test program, and the UI automation test program, when executed by the processor, implements the steps of the UI automation test method according to any one of the above.

[0014] The application provides a storage medium storing a computer program, and the computer program, when executed by a processor, implements the steps of the UI automation test method according to any one of the above.

[0015] As described above, the application comprehensively locates the UI element according to semantic similarity and attribute similarity, and the semantic similarity can realize semantic-based fuzzy matching, surpassing accurate text matching, understanding deep meanings of texts, and improving the positioning success rate in different languages, near-synonyms and other scenarios; through the combination of semantic similarity and attribute similarity, multi-modal features (i.e. text semantics and visual features) are fused, the context perception can be combined, the sensitivity to UI layout changes is reduced, the robustness and accuracy of UI element positioning are improved, and the UI element positioning is more in line with actual interactive scenarios. BRIEF DESCRIPTION OF DRAWINGS

[0016] Figure 1 is a flowchart of a UI automation test method according to an embodiment of the application; Figure 2 is a schematic diagram of an interface of a UI to be tested according to an embodiment of the application; Figure 3 is a flowchart of a target element determination method provided by an embodiment of the present application; Figure 4 is a structural diagram of a test system provided by an embodiment of the present application. DETAILED DESCRIPTION

[0017] To solve the above problems in the prior art, the present application provides a UI automatic test method and test system, and a storage medium. The several protection subjects are based on the same concept, and the principles of solving problems are basically the same or similar. The embodiments of each protection subject can be mutually referred to, and the repeated parts will not be described.

[0018] In the scheme of the present application, UI element positioning is comprehensively performed according to semantic similarity and attribute similarity. Semantic-based fuzzy matching is realized through semantic similarity, which surpasses accurate text matching, understands the deep meaning of text, and improves the positioning success rate in different languages, near-synonyms and other scenarios. Through the combination of semantic similarity and attribute similarity, multi-modal features (i.e. text semantics and visual features) are fused, the context perception is combined, the sensitivity to UI layout changes is reduced, the robustness and accuracy of UI element positioning are improved, and the UI element positioning is more in line with the actual interactive scenario.

[0019] To make the purpose, technical solutions and advantages of the present application clearer, the technical solutions of the present application will be described clearly below in combination with specific embodiments and corresponding drawings. Obviously, the following described embodiments are only part of the embodiments of the present application, not all the embodiments. In the case of no conflict, each of the following embodiments and its technical features can be combined with each other, and also belongs to the technical solutions of the present application.

[0020] Figure 1 is a flowchart of a UI automatic test method of an embodiment of the present application. The UI automatic test method can be simply referred to as "method" or "test method", and the execution subject of each step can be a storage medium, a processor, a controller, etc. with UI automatic test function.

[0021] Please refer to Figure 1 , the method comprises at least the following steps: S11: obtaining a target text and an optical character recognition result of a UI to be tested, the optical character recognition result comprising a plurality of elements, and element text, coordinate frame and visual features of each element.

[0022] An example of the present application can obtain the optical character recognition result of the UI to be tested by an OCR method, and the optical character recognition result is also referred to as an OCR result. The OCR method refers to that an execution subject of the present method detects all characters contained in the UI by an optical method, determines the shape thereof by a dark and bright mode, converts the shape into a black and white dot matrix image file, and then translates the shape into a computer recognizable character by a character recognition method, that is, converts the character in the image into a text format.

[0023] The UI is usually formed by characters and icons, and the icon itself is usually only a decorative pattern and does not contain a character indicating function, so when the UI test refers to testing the functions provided by the UI, the element can refer to the character contained in the UI, which can be specifically manifested as an element text. For example, in combination with the UI shown in Figure 2 The "New Conversation", "Kimi+", "History", "New Project or Topic Start", and "Log in to Sync History Conversation" selected by the solid line frame on the left side can be regarded as corresponding respective elements, and the "New", "Hi! It looks like you might want to start a new project or topic. Can you tell me more details? That way I can better help you!", "Copy", "Retry", "Share", "Search for something", "What do you want to do with your new project?", and "Do you need me to provide ideas?" selected by the solid line frame on the right side can also be regarded as corresponding respective elements.

[0024] The coordinate frame of the element refers to the coordinate range or accurate position of the element in the UI.

[0025] The visual features of the element include but are not limited to the combination of the type, shape, and position of the element.

[0026] The type includes but is not limited to the combination of a function menu button, an icon button, and a label, and can be used to identify the attributes of respective elements. Continue to combine Figure 2The UI shown, the function menu button can be understood as an option that allows to be clicked and realize the corresponding function, for example, the "new" button selected by the solid line frame on the right side of the figure; the icon button can be understood as a pattern with decorative or identifying function, for example, the message icon with "+" shown in the "new session" on the left side of the figure, if the "new session" button does not have the function of directly creating a session, but can only jump to the "new" interface shown on the right side, that is, only has the function of jumping to another interface, then the "new session" button can also be regarded as an icon button. The icon button can also assist in explaining the function of the function menu button, for example, the user can intuitively know that a new session can be created through the "new" button on the right side of the figure by watching the "+" icon displayed next to the "new" button. The label can be understood as an indicative explanation of the element, for example, the text explanation below the "new" button in the figure: "Hi! It looks like you might want to start a new project or topic. Can you tell me more details? That way I can better help you!" Based on the label, the user can more clearly and conveniently execute the function corresponding to the element.

[0027] Unlike the coordinate frame, the position contained in the visual feature can be understood as the relative orientation or approximate position of the element in the entire UI, not the precise position displayed by the coordinate frame.

[0028] S12: Obtain the semantic similarity between the target text and each element text.

[0029] In an example, the target text can be first converted into a semantic vector, for example, by using a custom word vector dictionary, including but not limited to converting the target text into a word vector by the WORD_VECTORS dictionary and calling the Word2Vec model, GloVe model or BERT (Bidirectional Encoder Representations from Transformers) model, each keyword in the target text corresponds to a four-dimensional vector, the word vector can be obtained by the get_word_vector function, and the semantic similarity between the semantic vector (such as word vector) of the target text and the element text of each element of the OCR result is calculated by traversing each element of the OCR result.

[0030] In another example, both the semantic vector of the target text and the semantic vector of each element text can be obtained, and then the semantic similarity between the semantic vector of the target text and the semantic vector of each element text is calculated by traversing each element of the OCR result. The functions and models called to obtain the semantic vector and the semantic similarity can be referred to the above example.

[0031] The semantic similarity between the target text and the element text is usually directly given by the adopted BERT model, and does not need to be obtained by the user or the execution subject of the method through self-defined comparison.

[0032] In step S12, for example, the content of the target text is "login", and the OCR result includes multiple elements, at least four elements with a certain semantic similarity (i.e. a semantic similarity not equal to 0) with "login", namely the first element "login account", the second element "register new user", the third element "login", and the fourth element "Sing In", the element texts of the four elements at least contain the keywords "login account", "register new user", "login", and "Sing In", respectively. The semantic similarity between the target text "login" and the element text "login" of the third element is the highest, the semantic similarity between the target text "login" and the element text "Sing In" of the fourth element is slightly lower, the semantic similarity between the target text "login" and the element text "login account" of the first element is lower, and the semantic similarity between the target text "login" and the element text "register new user" of the second element is the lowest.

[0033] If the semantic similarity is output and represented according to the scoring system, the semantic similarity between the target text and the element text of the third element can be 1.0, i.e. complete match, the semantic similarity between the target text and the element text of the fourth element can be 0.95, i.e. high match, the semantic similarity between the target text and the element text of the first element can be 0.90, i.e. secondary high match, the semantic similarity between the target text and the element text of the second element can be 0.1, i.e. low match, and the semantic similarity between the target text and the element text of other elements (e.g. the fifth element "settings") in the OCR result is 0, i.e. complete mismatch.

[0034] S13: Obtain the attribute similarity between the target text and each element based on the visual features of each element.

[0035] The actual scene can be simulated by a rule-based CNN (Convolutional Neural Network) classifier to determine the types and other visual features of each element in the OCR result according to the text content. For example, the text content containing keywords such as "login" and "Sign In" is identified as a function menu button, the text content containing keywords such as "settings" is identified as an icon button, and the types of other elements are identified as labels.

[0036] In step S13, the types and other visual features of each element and the target text are obtained by calling the corresponding get_element_type function, and compared to obtain the attribute similarity.

[0037] In an example, the target text can first be identified to contain a keyword, and then a target attribute corresponding to the target text can be determined according to the keyword by simulating a CNN classifier, the target attribute can be regarded as a type to which the keyword belongs, that is, a type of visual feature, and then a property similarity between the target text and each element can be obtained according to the target attribute and the visual feature of each element. For example, the content of the target text is "login", the determined target attribute is a function menu button, and the OCR result includes multiple elements, at least four elements having a certain semantic similarity (that is, a semantic similarity not equal to 0) with "login" are included in the elements, which are a first element "login account" with a visual feature type of label, a second element "register new user" with a visual feature type of function menu button, and a fourth element "Sing In" with a visual feature type of function menu button. Therefore, the property similarity between the target text "login" and the fourth element "Sing In" and the second element "register new user" is the highest, and the property similarity between the target text "login" and the first element "login account" is slightly lower.

[0038] If the property similarity is output and represented according to a scoring system, the property similarity between the target text and the fourth element and the second element can be 1.0, that is, complete matching, the property similarity between the target text and the first element can be 0.1, that is, low matching, and the property similarity between the target text and other types of elements (for example, a sixth element "+" with a visual feature type of icon button) in the OCR result is 0, that is, complete mismatching.

[0039] The property similarity between the target text and each element is usually directly given by the adopted CNN classifier, and does not need to be obtained by the user or the execution subject of the method through self-defined comparison.

[0040] S14: determining an element corresponding to the target text in the optical character recognition result as a target element according to the semantic similarity and the property similarity.

[0041] In an example, an element having both a high semantic similarity and a high property similarity can be selected from the OCR result as the target element. It should be noted that the target element in the OCR result does not necessarily have the highest semantic similarity, nor does it necessarily have the highest property similarity, but is the most similar result after weighting the semantic similarity and the property similarity (that is, the maximum value in the result after weighting).

[0042] In a scenario where the semantic similarity and the property similarity are output and represented according to a scoring system, the application can select an element with the highest total score from the OCR result as the target element. For example, the step S14 can include the following sub-steps S141-S145: S141: first scoring each element according to the semantic similarity; S142: second scoring of each element according to attribute similarity; S143: determining a first weight coefficient of semantic similarity and a second weight coefficient of attribute similarity; S144: calculating a sum of a product of the first score and the first weight coefficient and a product of the second score and the second weight coefficient as a total score of each element; S145: taking one element with the highest total score as the target element.

[0043] In other words, the comprehensive scoring algorithm of the present example can be regarded as calculating the total score of each element by using a weighted model, and the corresponding relationship is F score = (F1 x k1) + (F2 x k2), where F score represents the total score of each element, F1 represents the first score, also known as the semantic similarity score, k1 represents the first weight coefficient, F2 represents the second score, also known as the attribute similarity score, and k2 represents the second weight coefficient. The first weight coefficient can be greater than the second weight coefficient, indicating that the weight of semantic similarity is greater than the weight of attribute similarity, and semantic similarity is the most important factor in selecting the target element. For example, the value of the first weight coefficient can be k1 = 0.7, and the value of the second weight coefficient can be k2 = 0.3.

[0044] In actual scenarios, the cosine similarity function can be used to calculate the cosine similarity. First, the word vector of the target text and the word vector of the text of the OCR result are obtained, and then the cosine similarity of the two word vectors is calculated. The result of the cosine similarity is a value between 0 and 1, which is the first score; the attribute similarity score can be obtained by simulating a CNN classifier and assigning different scores according to the relevance of the visual features of the element and the target text, i.e., the second score.

[0045] Taking the target text "login" input by the user as an example, the simulation engine performs OCR operation on the UI to be tested to return the OCR result containing multiple elements such as "username", "Sign In", "Setting", etc. Then, the intelligent_ocr_touch function is called, the target text "login" and the OCR result are input into the algorithm model, the semantic vector of "login" is calculated by, for example, the BERT model, and then each element in the OCR result is traversed to calculate the semantic similarity score (i.e., the first score) of the element text of each element and "login" and the attribute similarity score (i.e., the second score).

[0046] Optionally, the application can also preset a reference threshold, and after step S144 and before step S145, it is judged whether the highest total score (i.e. the maximum value in the total score) is greater than or equal to the reference threshold; if the highest total score is greater than or equal to the reference threshold, the element with the highest total score is taken as the target element; and if the highest total score is less than the reference threshold, it is determined that the target element is not found this time, and the step S11 can be returned, i.e. the target text is reacquired, and the selection of the target element in the next time is performed.

[0047] The setting of the reference threshold can avoid that the selected target element is not the target text of the real demand of the user when the highest total score is too small, i.e. when the highest total score is too small, the selection of the target element is not performed, so as to ensure that the finally selected target element is the target text of the real demand of the user, and the accuracy of the whole UI automatic test is improved. In the actual scene, the reference threshold can be set to 0.7.

[0048] After the target element is determined, the following steps S15 and S16 are performed.

[0049] S15: taking the position of the coordinate frame of the target element in the UI as a target position.

[0050] S16: generating a demonstration image highlighting the target position and / or the target element.

[0051] The target position refers to the position that is suggested or required to be clicked by the user in the current test simulation, and the target position is highlighted, so that the tester can intuitively know which element of the UI is clicked in the simulation, and intuitively know whether the element clicked in the simulation is consistent with the element to be selected by the target text. Similarly, highlighting the target element can also achieve the technical effect.

[0052] The specific forms of highlighting include but are not limited to highlighting, changing the background color, dynamic flashing, etc. For example, the create_demo_image function is called to generate a UI demonstration image, and the target element is highlighted.

[0053] In another example, the application can introduce a confidence to select the target element. Specifically, the optical character recognition result also includes the confidence of each element, i.e. the optical character recognition result in the foregoing step S11 also includes the confidence. In combination with the confidence, the step S14 can include the following sub-steps S241-S245: Figure 3 As shown, the step S14 can include the following sub-steps S241-S245: S241: first scoring each element according to the semantic similarity; S242: second scoring each element according to the attribute similarity; S243: determining a first weight coefficient of semantic similarity, a second weight coefficient of attribute similarity, and a third weight coefficient of the confidence of each element; S244: calculating a sum of a product of the first score and the first weight coefficient, a product of the second score and the second weight coefficient, and a product of the confidence and the third weight coefficient, as a total score of each element; S245: determining whether the highest total score is greater than or equal to a preset reference threshold value; If the highest total score is greater than or equal to the reference threshold value, S246 is performed: taking the element with the highest total score as the target element; and if the highest total score is less than the reference threshold value, S247 is performed: determining that the target element is not found this time, and then returning to the step S11, i.e., reacquiring the target text and performing selection of the target element next time.

[0054] The present example has the same features as the above-described example, and reference can be made to the foregoing, for example, the features of the steps S241 and S242 correspond to the features of the steps S141 and S142, which will not be described again here.

[0055] The confidence indicates the accuracy of the OCR result, and based on the confidence, the present example can take the error of the OCR result into the selection factors of the target element, which can improve the accuracy of the selection of the target element.

[0056] The comprehensive scoring algorithm of the present example can be regarded as using a weighted model to calculate the total score of each element, and the corresponding relationship used in the step S244 is F score = (F1 x k1) + (F2 x k2) + (F3 x k3), where F score , F1, k1, F2, and k2 each represent the meaning described above, F3 represents the confidence of each element, and k3 represents the third weight coefficient. The first weight coefficient can be greater than the second weight coefficient, and the first weight coefficient can also be greater than the third weight coefficient, i.e., the weight of the confidence is less than the weight of the semantic similarity, and the confidence is a secondary factor or the least important factor for selecting the target element. For example, the value of the first weight coefficient can be k1 = 0.6, the value of the second weight coefficient can be k2 = 0.2, and the value of the third weight coefficient can be k3 = 0.2.

[0057] For any example of the foregoing steps S11-S16, the present application comprehensively positions the UI elements according to the semantic similarity and the attribute similarity, and has at least the following four technical effects: 1. Improve positioning accuracy: semantic similarity can achieve semantic-based fuzzy matching, surpassing precise text matching, understanding the deep meaning of the text, and improving the positioning success rate in different languages, synonyms, and other scenarios; by combining semantic similarity and attribute similarity, it is equivalent to fusing multi-modal features (i.e. text semantics and visual features), which can combine context awareness, reduce sensitivity to UI layout changes, improve the accuracy of UI element positioning, and better meet the actual interaction scenario.

[0058] 2. Enhance the robustness of UI element positioning: equivalent to fusing multi-modal features, which can combine context awareness, reduce sensitivity and dependence on UI layout changes, and even if the element coordinates or text form changes slightly, as long as the semantics and, for example, the type are consistent, it can still be positioned correctly.

[0059] 3. Improve the efficiency of automated testing: reduce the test case failure rate caused by UI element positioning failure, reduce manual debugging time, and improve the efficiency and stability of UI automated testing.

[0060] 4. Support multi-language and variant scenarios: can handle UI element positioning in different languages (such as Chinese and English), synonyms, and other scenarios, expanding the scope of automated testing.

[0061] The embodiments of the present application also provide a test system, as shown in Figure 4 The test system 40 includes: An OCR module 41 for obtaining optical character recognition results of a target text and a UI to be tested, the optical character recognition results including a plurality of elements, and element text, coordinate frame and visual features of each element; A semantic understanding module 42 for obtaining semantic similarity between the target text and each element text; A visual feature recognition module 43 for obtaining attribute similarity between the target text and each element based on the visual features of each element; An intelligent positioning and scoring module 44 for determining that the target text corresponds to an element in the optical character recognition results as a target element according to the semantic similarity and the attribute similarity; and, The position of the coordinate frame of the target element in the UI is taken as the target position; A simulation generation module 45 for generating a demonstration image highlighting the target position and / or the target element.

[0062] The above-mentioned various modules of the test system 40 cooperate with each other to execute the steps of the UI automated testing method of any of the preceding examples, so as to achieve the beneficial effects of the UI automated testing method of any of the preceding embodiments. For details, see the preceding embodiments, which will not be repeated here.

[0063] The above modules are logical units of the test system 40. In actual scenarios, any module can be embodied as one or more hardware entities according to an actual form of the test system 40, or one or more functions of multiple modules can be executed by one hardware entity, which all fall within the protection scope of the present application. For example, the intelligent positioning and scoring module 44 as a processing and control center of the test system 40 can be embodied as a central processing and control unit (CPU) or multiple chips included in the test system 40. In addition, the test system 40 can execute the functions of the above-mentioned semantic understanding module 42, visual feature recognition module 43 and intelligent positioning and scoring module 44 through three chips respectively.

[0064] The storage medium stores a UI automation test program, which is essentially a computer program. When the UI automation test program is executed by a processor, the steps of the UI automation test method of any example are implemented.

[0065] The storage medium includes, but is not limited to, any one of a read-only memory (ROM), a random access memory (RAM), a magnetic disk and an optical disk.

[0066] The program stored in the storage medium can execute the steps of the UI automation test method of any example provided by the present application, so that the beneficial effects of the UI automation test method of any example can be achieved. For details, refer to the foregoing examples, which will not be repeated here.

[0067] The present application also provides a test device or chip, which includes a memory and a processor. The memory stores a UI automation test program. When the UI automation test program is executed by the processor, the steps of the UI automation test method of any example are implemented. The test device or chip is provided with the storage medium as shown in the above examples. The processor loads the storage medium to execute the steps of the UI automation test method, so that the beneficial effects of the UI automation test method of the corresponding example can be achieved.

[0068] The above only describes some embodiments of the present application, and does not limit the patent scope of the present application. For those skilled in the art, any equivalent structural transformation made by using the content of the present application and the drawings is also included in the patent protection scope of the present application.

[0069] Although the present document uses steps such as S11, S12 and the like as the code of the steps, the purpose is to express the corresponding content more clearly and briefly, and it does not constitute a substantial limitation on the order. Those skilled in the art may perform S31 first and then perform S11, etc. during implementation, but these should be within the protection scope of the present application.

[0070] Although the present document uses the terms "first", "second", and the like to describe various information, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. In addition, the singular forms "a", "an" and "the" are intended to include the plural forms. The terms "or" and "and / or" are interpreted as inclusive, or mean any one or any combination. Only when the combination of elements, functions, steps or operations is inherently mutually exclusive in some way, will there be an exception to this definition.

Claims

1. A UI automation testing method, characterized in that, include: Obtain the optical character recognition results of the target text and the UI to be tested. The optical character recognition results include several elements, as well as the element text, coordinate box and visual features of each element. The semantic similarity between the target text and the text of each element is obtained; The similarity of the target text to the attributes of each element is obtained based on the visual features of each element. Based on the semantic similarity and the attribute similarity, the target text is determined to correspond to an element in the optical character recognition result and is used as the target element; The position of the target element's coordinate frame within the UI is taken as the target position; Generate a presentation image that highlights the target location and / or the target element.

2. The method according to claim 1, characterized in that, The process of obtaining the semantic similarity between the target text and the texts of each element includes: Obtain the semantic vector of the target text; By traversing each element of the optical character recognition result, the semantic similarity between the semantic vector of the target text and the element text of each element is calculated. or, Obtain the semantic vector of the target text and the semantic vector of each element text; By traversing each element of the optical character recognition result, the semantic similarity between the semantic vector of the target text and the semantic vector of each element text is calculated.

3. The method according to claim 1, characterized in that, The method of obtaining the attribute similarity between the target text and each element based on the visual features of each element includes: Identify the keywords contained in the target text; Determine the target attributes corresponding to the target text based on the keywords; The attribute similarity is obtained based on the target attribute and the visual features of each element.

4. The method according to claim 1, characterized in that, The method of determining the target element based on the semantic similarity and the attribute similarity includes: Each element is first scored based on the semantic similarity. A second score is assigned to each element based on the attribute similarity. Determine the first weight coefficient of the semantic similarity and the second weight coefficient of the attribute similarity; The sum of the product of the first score and the first weight coefficient and the product of the second score and the second weight coefficient is calculated as the total score of each element. Choose the element with the highest total score as the target element.

5. The method according to claim 1, characterized in that, The optical character recognition results also include the confidence level of each element; The method of determining the target element based on the semantic similarity and the attribute similarity includes: Each element is first scored based on the semantic similarity. A second score is assigned to each element based on the attribute similarity. Determine the first weight coefficient of the semantic similarity, the second weight coefficient of the attribute similarity, and the third weight coefficient of the confidence of each element; The sum of the product of the first score and the first weight coefficient, the product of the second score and the second weight coefficient, and the product of the confidence level and the third weight coefficient is calculated as the total score of each element. Choose the element with the highest total score as the target element.

6. The method according to claim 4 or 5, characterized in that, The method further includes: A preset baseline threshold is established; Determine whether the highest total score is greater than or equal to the benchmark threshold; If so, then the element with the highest total score will be the target element; If not, then the target element was not found this time.

7. The method according to claim 4 or 5, characterized in that, The first weighting coefficient is greater than the second weighting coefficient.

8. The method according to claim 1, characterized in that, The visual features include combinations of type, shape, and position, and the type includes combinations of function menu buttons, icon buttons, and labels.

9. A testing system, characterized in that, It includes a processor and a memory, wherein the memory stores a UI automation test program, and when the UI automation test program is executed by the processor, it implements the steps of the UI automation test method as described in any one of claims 1 to 8.

10. A storage medium, characterized in that, The device contains a computer program that, when executed by a processor, implements the steps of the UI automated testing method according to any one of claims 1 to 8.

Images