Software verification method, device, electronic device and storage medium

Abstract

The present invention discloses a software verification method, device, electronic device, and storage medium, which are used to solve the technical problem that traditional software UI automatic evaluation technology must develop automatic evaluation code for different forms of software separately, and the automatic evaluation code maintenance workload is large. The present invention comprises the following steps: determining an element to be operated on a software UI of a preset software; obtaining a screen sub-region containing the element on the software UI, and obtaining region information of the screen sub-region; obtaining screenshot data of the screen sub-region based on the region information; converting the screenshot data into a grayscale image; obtaining the target coordinates of the element on the grayscale image; obtaining the actual coordinates of the element on the software UI based on the target coordinates and the region information; performing a preset operation on the element based on the actual coordinates to generate a UI display result; and verifying the UI display result using the expected UI display result.

Description

Technical Field

[0001] The present invention relates to the field of software verification technology, and in particular to a software verification method, device, electronic equipment and storage medium. Background Art

[0002] Verifying and monitoring the normal operation of various functions based on the software UI has long been the main quality assessment method used by people. The principle is to first drive the software to run, such as entering text according to the prompts on the UI, selecting single or multiple selection boxes, clicking or double-clicking buttons or links and other UI elements, and then checking whether the UI effect meets expectations.

[0003] Because the human eye can easily determine if text and images on a UI have issues, visual evaluation can easily assess the quality of software of any form or technical solution. However, due to limitations in software and hardware development, current mainstream automated evaluation techniques primarily focus on the internal structure of UI elements, which presents a number of shortcomings in practical applications.

[0004] Existing UI automation technology strictly relies on specific software platforms, such as Android, Chrome, Net, or services based on custom protocols. After obtaining the UI elements to be operated and driving the software to run through the platform's technology, the technology provided by the platform is used to obtain updates to the element content and compare it with expectations to verify the operating effect.

[0005] However, these automation solutions require support from a foundational platform. While different forms of terminal software within the same system may share similar business requirements, the automated verification code formats and technical architectures often differ significantly. For example, the Selenium framework and its code used for Web UI automation have little in common or reusability with UIAutomator for Android clients and UIAutomation for iOS clients. This presents challenges such as high technical barriers to in-depth development, difficulty maintaining test scripts in a timely manner, and the inability to fully meet business needs. It can even lead to software based on a particular platform technology being unable to be automatically tested. Summary of the Invention

[0006] The present invention provides a software verification method, device, electronic device and storage medium, which are used to solve the technical problem that traditional software UI automatic evaluation technology must develop automatic evaluation code for different forms of software separately, and the automatic evaluation code maintenance workload is large.

[0007] The present invention provides a software verification method, which is applied to a client; the method comprises:

[0008] Determine the elements to be operated on the software UI of the preset software;

[0009] Acquire a screen sub-area containing the element on the software UI, and acquire area information of the screen sub-area;

[0010] Acquire screenshot data of the screen sub-area according to the area information;

[0011] Converting the screenshot data into a grayscale image;

[0012] Obtaining the target coordinates of the element on the grayscale image;

[0013] Acquire the actual coordinates of the element on the software UI according to the target coordinates and the area information;

[0014] Perform a preset operation on the element according to the actual coordinates to generate a UI display result;

[0015] The UI display result is verified using the expected UI display result.

[0016] Optionally, the area information includes width and height; and the step of obtaining a screen sub-area containing the element on the software UI and obtaining area information of the screen sub-area includes:

[0017] Determine a reference object on the software UI, select a starting point coordinate on the reference object, and save the starting point coordinate as a position offset;

[0018] determining the relative orientation of the element to the reference object;

[0019] Taking the starting point coordinates as a corner point and based on the relative direction, a screen sub-region including the element is intercepted on the software UI, and the width and height of the screen sub-region are obtained.

[0020] Optionally, the step of obtaining the target coordinates of the element on the grayscale image includes:

[0021] When the reference object is text, identifying the text in the grayscale image through text recognition, and obtaining the text coordinates of each text in the grayscale image;

[0022] Get the target text corresponding to the element;

[0023] The target text is matched in the text to obtain the target coordinates corresponding to the element.

[0024] Optionally, the step of obtaining the target coordinates of the element on the grayscale image includes:

[0025] When the type of the reference object is an image, obtaining the image to be searched corresponding to the element;

[0026] Obtaining the image content edge of the image to be searched;

[0027] Using image recognition to identify the image on the grayscale image;

[0028] Scaling the image on the grayscale image using a plurality of preset scaling factors to obtain a scaled image;

[0029] Obtaining a zoomed image content edge of the zoomed image;

[0030] matching the image content edge with the scaled image content edge of the scaled image to determine the target image corresponding to the element;

[0031] Acquiring a relative positional relationship between the target image and the reference object;

[0032] The target coordinates of the element on the grayscale image are determined according to the relative position relationship and the starting point coordinates.

[0033] Optionally, it also includes:

[0034] When the target text cannot be matched in the text, calculating the similarity between the target text and each text on the grayscale image, and determining the maximum similarity;

[0035] Determining whether the maximum similarity is greater than a preset similarity threshold;

[0036] If so, the text coordinates of the text corresponding to the maximum similarity are determined as the target coordinates of the element.

[0037] Optionally, it also includes:

[0038] When the maximum similarity is not greater than the preset similarity threshold, obtaining the confidence and context information of each of the texts;

[0039] Correcting the text according to the confidence level and the context information to obtain a corrected text;

[0040] Calculating the corrected similarity between the similar text and the target text, and determining the maximum corrected similarity;

[0041] When the maximum corrected similarity is greater than the preset similarity threshold, the text coordinates of the corrected text corresponding to the maximum corrected similarity are determined as the target coordinates of the element.

[0042] Optionally, the preset operation includes at least one of adding, deleting, modifying and querying.

[0043] The present invention provides a software verification device, comprising:

[0044] An element determination module is used to determine the element to be operated on the software UI of the preset software;

[0045] A screen sub-area acquisition module, configured to acquire a screen sub-area containing the element on the software UI and acquire area information of the screen sub-area;

[0046] A screenshot data acquisition module, configured to acquire screenshot data of the screen sub-region according to the region information;

[0047] A grayscale image conversion module, used to convert the screenshot data into a grayscale image;

[0048] A target coordinate acquisition module, used to acquire the target coordinates of the element on the grayscale image;

[0049] An actual coordinate acquisition module, configured to acquire the actual coordinates of the element on the software UI according to the target coordinates and the area information;

[0050] A UI display result generating module, configured to perform a preset operation on the element according to the actual coordinates to generate a UI display result;

[0051] The verification module is used to verify the UI display result using the expected UI display result.

[0052] The present invention provides an electronic device, comprising a processor and a memory.

[0053] The memory is used to store program code and transmit the program code to the processor;

[0054] The processor is configured to execute any one of the above software UI verification methods according to the instructions in the program code.

[0055] The present invention provides a computer-readable storage medium, wherein the computer-readable storage medium is used to store program code, and the program code is used to execute the software UI verification method as described in any one of the above items.

[0056] It can be seen from the above technical solution that the present invention has the following advantages: the present invention obtains the UI image of the running software and performs preliminary processing such as grayscale conversion, then identifies the elements and obtains the target coordinates of the elements on the grayscale image to determine the actual coordinates of the elements on the software UI, thereby accurately locating the elements. Further, by operating the located elements, the corresponding UI display results are obtained, and the UI display results are verified by the expected UI display results, thereby realizing automatic evaluation of the software UI. At the same time, there is no need to develop automatic evaluation code for different forms of software, which reduces the workload of code maintenance. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0058] Figure 1 A flowchart of a software verification method provided by an embodiment of the present invention;

[0059] Figure 2 A schematic diagram of a software UI display provided by an embodiment of the present invention;

[0060] Figure 3 A flowchart of a software verification method provided by an embodiment of the present invention;

[0061] Figure 4 This is a structural block diagram of a software verification device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0062] The embodiments of the present invention provide a software verification method, device, electronic device and storage medium, which are used to solve the technical problem that traditional software UI automatic evaluation technology must develop automatic evaluation code for different forms of software respectively, and the automatic evaluation code maintenance workload is large.

[0063] In order to make the purpose, features, and advantages of the present invention more obvious and easy to understand, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described below are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0064] See also Figure 1 , Figure 1 A flowchart of the steps of a software verification method provided by an embodiment of the present invention.

[0065] The present invention provides a software verification method, which may specifically include the following steps:

[0066] Step 101, determining the elements to be operated on the software UI of the preset software;

[0067] Software UI elements primarily consist of text, controls, and icons. These elements are often arranged in clusters, evenly spaced and aligned vertically and horizontally, and maintain a consistent UI style. Furthermore, different software programs have a limited number of control types and UI layout styles, but they share many commonalities. For example, they all use common controls like text boxes and buttons. This minimizes the learning curve for users when switching between different programs. These elements are crucial components of UI design guidelines.

[0068] In the embodiment of the present invention, according to the needs of the evaluation work, it is first necessary to determine the elements to be operated on the UI, such as Figure 2 When entering data in the scene of adding or editing data, generally select the first element in the upper left corner to start input; you can also randomly select any element from the UI.

[0069] Step 102: obtaining a screen sub-region containing an element on the software UI, and obtaining region information of the screen sub-region;

[0070] After the element to be operated is determined, a screen sub-region containing the element to be operated may be selected, and region information of the screen sub-region may be obtained.

[0071] In one example, the region information may include width and height; the step of obtaining a screen sub-region containing an element on a software UI and obtaining region information of the screen sub-region may include:

[0072] S21, determining a reference object on the software UI, selecting a starting point coordinate based on the reference object, and saving the starting point coordinate as a position offset;

[0073] S22, determining the relative orientation of the element and the reference object;

[0074] S23 , using the coordinates of the starting point as a corner point and based on the relative direction, intercepting a screen sub-region containing the element on the software UI, and obtaining the width and height of the screen sub-region.

[0075] In a specific implementation, a screen sub-area containing the element to be operated is selected. The height and width of the area do not need to be strictly restricted. It only needs to contain the selected element, have a certain amount of blank space around it, but not contain other irrelevant elements. To do this, it is necessary to first select a reference object and obtain a starting point coordinate based on the reference object, such as the coordinates of a corner point of the reference object. Then, based on the starting point coordinates, a screen sub-area is generated. Among them, the width and height of the screen sub-area can be the width and height obtained by circling a certain area in eight directions of the screen, namely, top, bottom, left, right, lower right, upper left, lower left or upper right, from the starting point corresponding to the starting point coordinates, or it can be a specified value. The specific direction to be selected can be determined based on the reference object selected on the UI (which can be text, control or icon) and the direction of the reference object in which the element to be operated is located. In this way, even if some components fail to be recognized, these characteristics can be used to calculate the coordinates of the operable pixel points of the element to be operated on the entire UI, so that the entire evaluation process can be driven to continue running even if the recognition success rate cannot reach 100%. For example Figure 2 In the example, we take advantage of the horizontal and vertical alignment and uniform layout of UI components. When OCR successfully recognizes the "success number", we can select its upper left corner coordinate (x1, y1) as the starting point. Assuming its upper right corner coordinate is (x2, y1), the difference between x2 and x1 is divided by 3 to get the width of each character. Reducing x1 by a certain value can get the position of the "execution number" (to the left of the starting point). Reducing x1 and y1 by a certain value can get the position of the "evaluation description" (upper left of the starting point). The certain value here is fixed on a certain UI due to the uniform layout of controls. It can be obtained by calculating the spacing between adjacent controls or texts that have been successfully recognized. For example, Figure 2 In the left column starting with "Product Name," as long as any one of the text can be successfully located by OCR, the difference between its x1 and the x coordinate of the upper right corner will horizontally cover the text box to the left of the starting point. While covering the object to be located and operated on, avoid selecting the entire remaining area. Also, minimize the number of irrelevant objects included in the selected area without compromising the accuracy of the result.

[0076] Step 103, obtaining screenshot data of the screen sub-region according to the region information;

[0077] In an embodiment of the present invention, after determining the screen sub-area, the screenshot() method of PyAutoGUI can be called and the area information of the screen sub-area obtained above can be passed in to obtain the corresponding screenshot data. All subsequent area screenshots used for identification are only operated in the memory and not saved to the local disk.

[0078] Among them, PyAutoGUI is a software tool developed in Python that can work on Windows, Linux and Mac systems. It is specifically used to control the mouse and keyboard to automatically interact with a software system.

[0079] Step 104, converting the screenshot data into a grayscale image;

[0080] After obtaining the screenshot data of the screen sub-area, the screenshot data can be converted from RGB channel data to a grayscale image. For example, the array() method of NumPy or the cvtColor() method of OpenCV can be used to convert the screenshot data into a grayscale image.

[0081] NumPy, or Numerical Python, is an open-source numerical computing extension for Python. This tool can be used to store and manipulate large matrices, much more efficiently than Python's native nested list structure (which can also be used to represent matrices). It supports arrays of many dimensions and matrix operations, and also provides a rich library of mathematical functions for array operations.

[0082] OpenCV is a cross-platform computer vision and machine learning software library released under the Apache 2.0 license (open source), running on Linux, Windows, Android, and Mac OS. It is lightweight and efficient, consisting of a series of C functions and a small number of C++ classes. It also provides interfaces for languages ​​such as Python, Ruby, and MATLAB, implementing many common algorithms for image processing and computer vision.

[0083] Step 105, obtaining the target coordinates of the element on the grayscale image;

[0084] In this embodiment of the present invention, a screen subregion is an area containing an element to be operated, obtained based on the starting point coordinates. Since the starting point coordinates are known, the actual coordinates of the element within the entire software UI can be determined by determining the relative coordinates of the element in the grayscale image relative to the starting point coordinates.

[0085] In one example, the step of obtaining the target coordinates of the element on the grayscale image may include:

[0086] S31, when the reference object is text, identifying the text in the grayscale image through text recognition, and obtaining the text coordinates of each text in the grayscale image;

[0087] S32, obtaining the target text corresponding to the element;

[0088] S33, matching the target text in the text to obtain the target coordinates corresponding to the element.

[0089] In a specific implementation, when the reference object is a string of text, the coordinates of the text can be determined using OCR technology. OCR (optical character recognition) refers to the process by which an electronic device (such as a scanner or digital camera) examines characters printed on paper and uses character recognition methods to translate the shapes into computer text. This involves scanning textual material and then analyzing and processing the image file to obtain text and layout information.

[0090] The specific steps are as follows:

[0091] The text in the grayscale image is recognized by OCR to obtain the text coordinates and confidence of each text in the area; the target text corresponding to the element to be operated is obtained and the corresponding recognition result is queried to obtain the target coordinates of the element in the grayscale image.

[0092] Furthermore, the step of obtaining the target coordinates of the element on the grayscale image may further include:

[0093] S34, when the target text cannot be matched in the text, calculating the similarity between the target text and each text on the grayscale image, and determining the maximum similarity;

[0094] S35, determining whether the maximum similarity is greater than a preset similarity threshold;

[0095] S36: If yes, determine the text coordinates of the text corresponding to the maximum similarity as the target coordinates of the element.

[0096] If the target text cannot be matched in the text, you can use the target text and the text in the grayscale image to construct a SequenceMatcher() instance in Python, and then call its ratio() method to obtain the similarity between the target text and the text in the grayscale image. If the similarity exceeds a certain set similarity threshold, considering the error rate in text recognition, in some UI text recognition scenarios, it can be considered that the two are actually the same string. At this time, the coordinates of the corresponding string in the text obtained by the above recognition are the target coordinates of the target text.

[0097] Furthermore, the step of obtaining the target coordinates of the element on the grayscale image may further include:

[0098] S37, when the maximum similarity is not greater than the preset similarity threshold, obtaining the confidence and context information of each text;

[0099] S38, correcting the text according to the confidence level and the context information to obtain a corrected text;

[0100] S39. Calculate the corrected similarity between the similar text and the target text, and determine the maximum corrected similarity.

[0101] When the maximum corrected similarity is greater than the preset similarity threshold, determine the text coordinates of the corrected text corresponding to the maximum corrected similarity as the target coordinates of the element.

[0102] When the similarity between the text and the target text is lower than the preset similarity threshold, the text recognized from the grayscale image can be checked character by character in order from the character form, the confidence level of the recognition result, and the UI content context to correct the recognition of some special characters. For example, the following characters can be considered the same: 1 (digit) and l (lowercase of L), s and S, 5 and S, B and 8, etc. Which specific characters are allowed to be corrected in this way can be set according to the actual application. Calculate the similarity between the corrected string and the target text. After the similarity exceeds a certain similarity threshold, the two strings can be considered the same, thereby determining the target coordinates of the target text.

[0103] In another case, if only a part of a string is recognized, the order relationship of all strings and their positions can be used. After excluding the correctly recognized strings, search for similar texts. Combining the positional relationship of the text on the grayscale image also helps to determine which UI string the recognition result is. For example Figure 2 If the character "set" in "number of successful sets" is not recognized successfully, the actual character can be determined as "set" according to the situation that its coordinates are紧邻 (should be "immediately adjacent to") below "number of failures" and the other several characters are recognized correctly. After such correction, the accuracy of the recognition result can be improved, and the positioning of more UI elements can be achieved.

[0104] In another example, the steps to obtain the target coordinates of an element on the grayscale image may include:

[0105] S41. When the type of the reference object is an image, obtain the image to be searched corresponding to the element.

[0106] [[ID=​​​​​​​​​​​​​

[0111] S47, obtaining the relative position relationship between the target image and the reference object;

[0112] S48, determining the target coordinates of the element on the grayscale image according to the relative position relationship and the starting point coordinates.

[0113] In a specific implementation, when the UI element selected as the reference object is an icon or an image, its target coordinates can be determined in the selected screen sub-area by dynamic template matching according to the following steps.

[0114] a) Take a screenshot of the element to be searched from the UI of the software to be evaluated and save it as a local file;

[0115] b) Call the imread() method of OpenCV to read the image to be searched corresponding to the element to be operated from the local computer, and then call the cvtColor() method to convert it into a grayscale image;

[0116] c) Call OpenCV's Canny() method to obtain the image content edge of the image to be searched. Furthermore, you can also perform Gaussian blur on the image content edge, use the Sobel operator to calculate the pixel gradient magnitude and direction, use non-maximum suppression (only the largest value is retained) to eliminate the stray effects caused by edge detection, apply double thresholding and suppress weak edges to obtain the final image content edge;

[0117] d) Generate a batch of scaling factors, with the start and end indexes distributed around 1, for example, 0.5 to 1.5. The maximum value can be set by adding a certain amount of redundancy based on the ratio of the current screen width and height to the screen size of the template image. The image is then divided into 20 to 30 parts. In principle, the image after scaling to a certain ratio should be close to the image to be searched.

[0118] e) Circularly scale the image on the grayscale image using the above scaling factor according to the original aspect ratio. If the height or width of the scaled image is less than the corresponding value of the image to be searched, exit the loop; otherwise, continue searching;

[0119] f) Then call OpenCV's matchTemplate() method to scale the image content edge to match the image content edge in a loop;

[0120] g) Call OpenCV's minMaxLoc() method to obtain the maximum value and corresponding position of the matching result in the previous step, and then save these two data together with the corresponding scaling ratio in memory;

[0121] h) If there is already saved data in the previous loop step, compare it with the result of this step and save the maximum value of the two;

[0122] After the loops i), d) to g) are completed, the maximum value, corresponding position and corresponding zoom ratio of the last saved matching result are obtained. The position corresponding to the maximum value is the position of the target image.

[0123] j) Based on the positional relationship between the reference object and the object to be operated, the coordinates of the object to be operated can be obtained. If the image itself is the component to be operated, such as a custom CheckBox, then the coordinates in the previous step are the final result. If the text box to the left of the image is to be operated, subtract a value greater than the distance between the two from the x value and leave the y value unchanged.

[0124] k) If there are multiple images with the same shape to be located in the selected area and one of them needs to be located, the input parameter should be specified as TM_CCOEFF_NORMED when executing matchTemplate() in step e);

[0125] 1) Filter the matching results using NumPy's where() function to obtain the coordinates of matching results that exceed the specified threshold; then perform non-maximum suppression on the resulting values ​​to remove duplicates and obtain the coordinates of the upper left and lower right corners of each matching rectangle before saving them;

[0126] m), and then execute j) and k) on the image scaled according to the original aspect ratio to find images of different sizes in the screenshot. Finally, sort the points in the upper left corner of the result set obtained from step k) from left to right and from top to bottom, and then obtain the position of the image object with the corresponding serial number.

[0127] In another example, when the UI element selected as the reference object is a composite component that contains prompt text and a corresponding image, the coordinates of the rectangle corresponding to the text can be obtained through the above-mentioned text recognition process, and the coordinates of the image can be obtained through the image recognition process. Based on the relative position of the object to be operated next and the two objects on the UI, the coordinates of the object to be operated next can be calculated using the two objects.

[0128] In another example, when the text description of the UI element selected as the reference cannot be identified and there is no image near the element on the UI that can be used for relative positioning, the position of the element to be operated can be determined by calculating the spacing between controls and combining the layout characteristics of the controls, for example Figure 2 If the "Number of Successful Packages" cannot be recognized, since the controls on this UI are evenly distributed vertically and the two controls in each row are centered horizontally, the coordinates of the operable point of the text box in this component can be calculated using the coordinates of the components above or to the right. The corresponding process is as follows:

[0129] The coordinates (x1, y1) of a reference component in the horizontal or vertical direction can be obtained by using the above-mentioned text recognition or image recognition process;

[0130] Using the calculation method from the previous step, calculate the vertical or horizontal distance d between two other components with the same or similar spacing on the UI as needed. If the object to be operated is vertical to the object in the previous step, calculate the vertical distance; if they are horizontal, calculate the horizontal distance;

[0131] If components are not evenly distributed in space, the positions will often differ slightly from those of evenly distributed components. If there are many unevenly distributed components and the differences are large, the placement of components on the UI will be somewhat chaotic and not in line with UI design specifications. In this case, it is necessary to use the above differences to make some estimates.

[0132] In some scenarios, it is necessary to detect whether a component at a certain location has been missed (not detected). In addition to improving the accuracy of image and text recognition (which must be trained and verified on the server first), you can also take a screenshot of the area where the control is located and perform histogram calculation and comparison with an image of a blank area of ​​the same size on the same UI; or search for pre-prepared control images (such as . Figure 2 If the control combination contains an image, dynamic template matching can also be used to search for the corresponding image in this area. As long as there is a recognition result, it means that there is a missed detection. Then, the control position can be inferred by using the principle of control alignment and width and height consistency;

[0133] When no estimation is needed, if the two selected components are evenly spaced vertically, the y coordinate value of the component to be operated can be obtained by directly increasing (the component to be operated is at the bottom) or decreasing (the component to be operated is at the top) the number of components in the middle based on the spacing between a) and b) and the number of components in the middle. The x coordinate value remains unchanged. For example, if the text box to be operated is below the reference text box and there are two text boxes of the same size and evenly spaced vertically between the two, the coordinates of the component to be operated are (x,y+3d); if Figure 2 In the example, the "Number of Successful Kits" field is not recognized, but the "Product Name" field is recognized correctly. Furthermore, the coordinates of the input point for the product name are (x1, y1). Since there are four evenly spaced controls above and below the two, assuming the vertical distance between the center points of the corresponding explanatory text in two adjacent text boxes is d, the coordinates of the corresponding input point for the "Number of Successful Kits" text box are (x1, y+5d). Similarly, when two selected components are evenly spaced horizontally, the y coordinate remains unchanged; simply increase or decrease the horizontal distance in the x coordinate. If an estimate is needed, add the estimated value to the final x or y coordinate based on the horizontal or vertical alignment of the two components.

[0134] Step 106: Obtain the actual coordinates of the element on the software UI based on the target coordinates and the region information;

[0135] In a specific implementation, the region information may include the starting point coordinates. After the target coordinates are acquired, the actual coordinates of the element on the software UI may be calculated based on the target coordinates and the starting point coordinates.

[0136] Step 107, performing a preset operation on the element according to the actual coordinates, and generating a UI display result;

[0137] Step 108: Verify the UI display result using the expected UI display result.

[0138] In an embodiment of the present invention, the operation performed on the element may include at least one of adding, deleting, modifying and querying.

[0139] After obtaining the actual coordinates of the element, you can perform preset operations on the element based on the actual coordinates to obtain the UI display result; then compare the UI display result with the expected UI display result to determine whether the results are consistent. If they are inconsistent, it indicates that there is a quality problem with the software.

[0140] The present invention obtains the UI image of the running software and performs preliminary processing such as grayscale conversion. It then identifies elements and obtains their target coordinates on the grayscale image to determine their actual coordinates on the software UI, thereby accurately locating the elements. Furthermore, by operating on the located elements, corresponding UI display results are obtained, and the UI display results are verified using the expected UI display results, thereby achieving automatic evaluation of the software UI. This eliminates the need to develop separate automatic evaluation code for different software forms, reducing the workload of code maintenance.

[0141] For ease of understanding, the embodiments of the present invention are described below with reference to specific examples:

[0142] See also Figure 3 , Figure 3 The present invention provides a flowchart of a software verification method.

[0143] Adding, deleting, modifying, and checking (browsing) are extremely common functions in many categories of software. Here, the automatic evaluation of the contact management function of a certain product is used as an example to illustrate the embodiment of the present invention, which may specifically include the following steps:

[0144] 1) According to the above scheme, the text string rectangular area corresponding to the target module is found through OCR recognition, and the coordinates of the center point of the rectangle are calculated. PyAutoGUI's click() clicks this point to open the function to be evaluated. There is no data at the beginning;

[0145] 2) Click to add a new contact record;

[0146] 3) In the pop-up form for adding contact records, use text recognition and distance calculation between components to obtain the coordinates of the text prompt next to each input control where data is to be entered, and then obtain the coordinates of the center node of the corresponding rectangle based on these coordinates;

[0147] 4) Taking advantage of the horizontal center alignment of the text and the input control, increase the x value of the coordinate in the previous step by an offset so that it exceeds the distance between the text prompt label and the input control. The y value remains unchanged, and the new (x, y) is used as the position to be operated;

[0148] 5) Call PyAutoGUI's click() to click the coordinates obtained in the previous step, and then call typewrite() to enter content into the text box. Here, you can record the values ​​of the main attributes added here, such as the contact name, based on the data displayed on the business entity list page;

[0149] 6) If the data is entered in the text box, then the above step 5) has completed the input of the current component; if it is a drop-down box, after the click() in step 5) is executed, wait for a certain time (how long to wait is determined by the time-consuming background function), then use the above solution to take a screenshot, identify the coordinates of the pop-up item, then click the item to be entered, and wait for about 0.5 seconds to complete the selection and input operation of the drop-down box; if it is a single-select or multi-select component, directly call the above image search function, search for the coordinates of the corresponding single-select or multi-select component and click it. If it cannot be found, use the distance calculation method between the above components to obtain the coordinates of the component to be operated to realize data input;

[0150] 7) After entering all components, use OCR to find the center coordinates of the button containing the "OK" or "Save" string and click it to save the data. Then wait for the background processing to complete. If the data is successfully added, the pop-up window that appeared in step 3) should have closed.

[0151] 8) If the input data does not meet the requirements, such as missing data, incorrect data format, or data exceeding the specified range, the UI should display corresponding prompts. By using OCR to check whether the corresponding prompt information exists in the specified area on the UI, you can verify whether the system's data verification function exists and whether the prompt content meets expectations. After successful addition, proceed to the next step;

[0152] 9) Identify the text in the content area and obtain the result of the above operation;

[0153] 10) Check the recognition results to see if the data recorded in step 5) appears on the UI. If it does, it indicates that the previous step was successful. If it does not, it indicates failure and the current software screenshot is saved as a local image.

[0154] 11) If an operation fails, you can call PyAutoGUI to enter the shortcut key to open the browser console, and then use OCR to extract the text of the error output that may appear in the console;

[0155] 12) If it is added successfully, the image search function is used to find the corresponding rectangular area on the UI for the single-select or multi-select component of the list row where the newly added record is located. Since the single-select or multi-select components are generally displayed in the first column of the list, the main attribute value of the corresponding record can be used as a reference. Take a screenshot to the left to include the corresponding CheckBox component and then perform an image search. The coordinates of the main attribute text of the record obtained above can be used to calculate the coordinates of the single-select or multi-select component on the UI. Then calculate the coordinates of its center point and click to select it. Then use OCR to find the delete button or use the image search function to find the rectangle of the icon for deleting the record on the same row and its center point coordinates. After finding it, click to delete the selected data. Here, the distance between controls is calculated as needed to handle inaccurate searches, and the key attributes of the business entity, such as the contact name, are also recorded.

[0156] 13) After deleting the data, wait for the background processing to complete, and then execute 9) to 10). If the key attribute data of the business entity recorded above cannot be found on the UI, it can be determined that the deletion is successful. If it fails, the current software screenshot is saved as a local image;

[0157] 14) Repeat the above steps 2) to 13). If you encounter problems such as a blank screen (no content is displayed in the content area, and the corresponding web recognition result set is always empty), the added data cannot be displayed (the newly added business data cannot be found after text recognition correction), the business record data cannot be completely deleted (the content of some fields is still displayed after a business data item is deleted), or the displayed data is inconsistent with the pre-recorded data (for example, more records are displayed on the UI after deletion), it indicates that there is a bug in the relevant function.

[0158] 15) Repeat steps 2) through 14, counting how many times the function works normally, how many times it encounters a bug, and what the specific issues are. This statistics can be used to determine whether the relevant functions are unstable and what the main issues are. Once this information is collected, it can be recorded and a report can be generated to notify relevant personnel.

[0159] Furthermore, due to the performance of text and image recognition and the accuracy of image search, automatic adaptation for different graphics cards and different screen sizes is very meaningful. Otherwise, problems may include the evaluation script running time increasing by dozens of times or even more, the layout of controls on the UI becoming chaotic and greatly different from expectations, and even the inability to find the required text or image on the UI. The automatic adaptation function is used to record the coordinates of a string, image, or control running on a device as a reference. This function can be applied to every UI of the software to be evaluated. The specific plan and steps for automatic adaptation include:

[0160] When opening a new UI and selecting an area to search for UI elements, check whether the configuration file that stores the reference object and the coordinate information of the reference object specified in the configuration file exist and are both greater than 0.

[0161] When the coordinate data of the reference object exists and is greater than 0, you can select a sub-region in the eight directions of the reference object on the UI, then search and operate UI elements in the selected region and verify the results. This helps reduce useless recognition operations and thus improves performance.

[0162] When the coordinate data of the reference object does not exist or the value is less than 0, the reference object is searched globally on the entire UI according to the above scheme. Depending on the software to be processed and the category of the reference object on the UI, the above scheme can be used to search for text, search for images, or use the YOLO v5 model to find the coordinate position of the reference object on the UI. As long as the reference object can be found correctly on the UI, it will be fine.

[0163] Verify the position of the found object to confirm whether it is actually found. Here, you can either draw the found object with a rectangle of a specific color (such as red) for debugging, or operate the found object on the UI and then search for elements on the UI to verify whether the UI effect after the operation meets the expectations. For example, if there is a text box on the reference object or next to it, you can directly enter any string into the text box, and then use the position of the reference object relative to the OCR to select a small area containing the text box, then identify this area and check from the results whether it contains the string just entered. If it does, it proves that the reference object found in the previous step is correct;

[0164] After finding the object, you can use ConfigParser to record the (x, y) coordinates of the reference object into the configuration file corresponding to step 1). You can use the UI code and reference object code as part of the data key. Here, you need to ensure that the code combination of each UI and reference object is unique, so that multiple UIs of a project and one UI can also be adapted according to multiple reference objects. If it cannot be found, an error prompt will be output. At the same time, check whether the width and height of the current screen are lower than the specified value, such as 1024*768. If so, a corresponding warning will be given to facilitate checking whether the graphics card driver or display configuration of the relevant device is normal.

[0165] When searching for images on the software UI through dynamic template matching, the width and height of the display screen when obtaining the template image file and the width and height of the current screen when performing the image search may be different. The largest value greater than 1 in the ratio of the width or height of the two screens is obtained, and a certain amount of redundancy is added as the upper limit of scaling the image according to the original aspect ratio of the image during the dynamic template matching process. By continuously scaling the image according to a series of scaling factors, the matching results help improve the success rate of finding pre-prepared template images on different devices and improve the stability and accuracy of the automatic evaluation process.

[0166] Furthermore, software errors may occur during UI operation, and comprehensive automatic collection of relevant error information may be very helpful in discovering and resolving software quality issues.

[0167] Error messages such as pop-up prompts on the UI can be directly captured and saved and associated with the automatic evaluation log in HTML format, so that the error screenshots can be automatically displayed when viewing the evaluation log.

[0168] The process of extracting the error message output by the browser console includes:

[0169] You can use PyAutoGUI's hotkey() to call the browser's shortcut key (such as Chrome's shortcut key ctrl+shift+i) to open the console;

[0170] If the console is displayed at the bottom of the browser, you can take a screenshot of the area in the lower left corner that is approximately 40% of the height and 30% of the width, and then use it for recognition. If the console is located elsewhere in the browser, adjust the screenshot area accordingly to ensure that it covers the console display area. Then, search for "Console" in the recognition results, find the corresponding rectangular area, and use the coordinates of the upper left corner of the screenshot to calculate the coordinates of the center point of the rectangle corresponding to "Console" on the entire UI. Clicking this coordinate will open the console output.

[0171] Then, the entire bottom area can be recognized by OCR to obtain all the output text in the bottom area;

[0172] Finally, we can use the characteristics of the text to filter out content that is definitely not error information, and retain the exceptions and error information triggered by front-end codes such as Vue and React.

[0173] Navigation information of software business functions, that is, which modules the software is composed of and what functional points are contained in each module, is helpful for automatically drawing the relationship diagram between the functional modules of the software. Through this relationship diagram, important quality information such as which parts of the system all software quality issues belong to and which modules have more problems can be intuitively displayed on the UI.

[0174] Since software module information is often displayed in several modular ways in the UI layout, such as the top of the software UI, the left column, the combination of the top and left columns, or the bottom navigation bar of a mobile app, the functional module relationship diagram of the software can be obtained through the following process:

[0175] Taking the software name as the root node, a tree diagram of at least three levels of system->module->function point can be established. If the module is organized through one or more levels of menus, then there are as many levels of modules as there are menu levels, and each module has one or more levels of function points.

[0176] According to the different organization of module content on the UI, if text tiling is used, all module names can be obtained by taking screenshots of these specific areas on the UI and then recognizing the text. If the module name text cannot be recognized, it can be improved by training a new model or font library, or by utilizing the features on the UI. For example, if the menu item consists of image + text, you can also search for images in the area where the menu item is located, and use the images as the module identifier after finding them. If the module name is organized by menu items, first use PyAutoGUI's moveTo() to control the mouse to move above the menu item text, wait for a while, and then capture the image of the area where the pop-up menu is located after the menu pops up. After processing it into a grayscale image, recognize, correct or find the text and position of each menu in the pop-up menu item. Here, each menu text is a module, and clicking the center point of the corresponding rectangle can open the corresponding function.

[0177] After clicking a module name and before clicking another module name, all click operations except for OK, Save, and Cancel are considered function points. The title of the UI corresponding to the function point, the button that opens the function point, or the text of the menu item can all be used as function point names. All function point names can be used as functions of the next layer of the module.

[0178] If a non-prompt pop-up window can be opened on the corresponding UI after a function point is opened, the title of the pop-up window can be used as the next-level function point of the function point;

[0179] The complete navigation tree information can be organized in JSON format and saved as a local file or saved to the server. It can also be used to draw a tree-like functional navigation diagram through Graphviz or some front-end libraries.

[0180] See also Figure 4 , Figure 4 This is a structural block diagram of a software verification device provided by an embodiment of the present invention.

[0181] An embodiment of the present invention provides a software verification device, comprising:

[0182] The element determination module 401 is used to determine the element to be operated on the software UI of the preset software;

[0183] The screen sub-area acquisition module 402 is used to acquire the screen sub-area containing the element on the software UI and acquire the area information of the screen sub-area;

[0184] The screenshot data acquisition module 403 is used to acquire the screenshot data of the screen sub-region according to the region information;

[0185] Grayscale image conversion module 404, used to convert the screenshot data into a grayscale image;

[0186] The target coordinate acquisition module 405 is used to acquire the target coordinates of the element on the grayscale image;

[0187] The actual coordinate acquisition module 406 is used to acquire the actual coordinates of the element on the software UI according to the target coordinates and the area information;

[0188] A UI display result generating module 407 is used to perform a preset operation on the element according to the actual coordinates to generate a UI display result;

[0189] The verification module 408 is configured to verify the UI display result using the expected UI display result.

[0190] In the embodiment of the present invention, the region information includes width and height; the screen sub-region acquisition module 402 includes:

[0191] The starting point coordinate selection submodule is used to determine the reference object on the software UI, select the starting point coordinates on the reference object, and save the starting point coordinates as a position offset;

[0192] The relative direction determination submodule is used to determine the relative direction of the element and the reference object;

[0193] The screen sub-area capture sub-module is used to capture the screen sub-area containing elements on the software UI based on the relative direction, taking the starting point coordinates as the corner point, and obtain the width and height of the screen sub-area.

[0194] In this embodiment of the present invention, the target coordinate acquisition module 405 includes:

[0195] The text recognition submodule is used to identify the text in the grayscale image through text recognition when the reference object is text, and obtain the text coordinates of each text in the grayscale image;

[0196] The target text acquisition submodule is used to obtain the target text corresponding to the element;

[0197] The first target coordinate determination submodule is used to match the target text in the text to obtain the target coordinates corresponding to the element.

[0198] In this embodiment of the present invention, the target coordinate acquisition module 405 includes:

[0199] The submodule for obtaining the image to be searched is used to obtain the image to be searched corresponding to the element when the type of the reference object is an image;

[0200] An image content edge acquisition submodule is used to acquire the image content edge of the image to be searched;

[0201] An image recognition submodule, configured to recognize an image on a grayscale image using image recognition;

[0202] The scaled image acquisition submodule is used to scale the image on the grayscale image using a plurality of preset scaling factors to obtain a scaled image;

[0203] A zoomed image content edge acquisition submodule is used to acquire the zoomed image content edge of the zoomed image;

[0204] a target image determination submodule, configured to match an image content edge in a scaled image content edge of the scaled image to determine a target image corresponding to the element;

[0205] The relative position relationship acquisition submodule is used to obtain the relative position relationship between the target image and the reference object;

[0206] The second target coordinate determination submodule is used to determine the target coordinates of the element on the grayscale image according to the relative position relationship and the starting point coordinates.

[0207] In this embodiment of the present invention, the target coordinate acquisition module 405 further includes:

[0208] A similarity calculation submodule is used to calculate the similarity between the target text and each text on the grayscale image and determine the maximum similarity when the target text cannot be matched in the text;

[0209] The first judgment submodule is used to judge whether the maximum similarity is greater than a preset similarity threshold;

[0210] The third target coordinate determination submodule is configured to determine the text coordinates of the text corresponding to the maximum similarity as the target coordinates of the element.

[0211] In this embodiment of the present invention, the target coordinate acquisition module 405 further includes:

[0212] A confidence and context information acquisition submodule, used to obtain the confidence and context information of each text when the maximum similarity is not greater than a preset similarity threshold;

[0213] A correction submodule, used for correcting the text according to the confidence and context information to obtain a corrected text;

[0214] A correction similarity calculation submodule is used to calculate the correction similarity between the similar text and the target text, and determine the maximum correction similarity;

[0215] The fourth target coordinate determination submodule is configured to determine the text coordinates of the corrected text corresponding to the maximum corrected similarity as the target coordinates of the element when the maximum corrected similarity is greater than a preset similarity threshold.

[0216] In the embodiment of the present invention, the preset operation includes at least one of adding, deleting, modifying and querying.

[0217] An embodiment of the present invention further provides an electronic device, the device including a processor and a memory:

[0218] The memory is used to store program codes and transmit the program codes to the processor;

[0219] The processor is configured to execute the software UI verification method according to the instructions in the program code.

[0220] An embodiment of the present invention further provides a computer-readable storage medium, which is used to store program code, and the program code is used to execute the software UI verification method of the embodiment of the present invention.

[0221] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0222] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0223] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, apparatus, or computer program products. Thus, embodiments of the present invention may take the form of a fully hardware embodiment, a fully software embodiment, or an embodiment combining software and hardware. Furthermore, embodiments of the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0224] The embodiments of the present invention are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the process in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0225] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0226] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0227] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they become aware of the basic creative concepts. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.

[0228] Finally, it should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that includes a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or terminal device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or terminal device that includes the element.

[0229] As described above, the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that the technical solutions described in the above embodiments can still be modified, or some of the technical features thereof can be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A software UI automatic verification method, characterized in that: Applied to a client; the method includes: Determine the elements to be operated on the software UI of the preset software; Acquire a screen sub-area containing the element on the software UI, and acquire area information of the screen sub-area; Acquire screenshot data of the screen sub-area according to the area information; Converting the screenshot data into a grayscale image; Obtaining the target coordinates of the element on the grayscale image; Acquire the actual coordinates of the element on the software UI according to the target coordinates and the area information; Perform a preset operation on the element according to the actual coordinates to generate a UI display result; Verifying the UI display result using the expected UI display result; The region information includes width, height, and starting point coordinates; and the step of obtaining a screen sub-region containing the element on the software UI and obtaining region information of the screen sub-region includes: Determine a reference object on the software UI, select a starting point coordinate on the reference object, and save the starting point coordinate as a position offset; determining the relative orientation of the element to the reference object; Taking the starting point coordinates as a corner point and based on the relative direction, a screen sub-region including the element is intercepted on the software UI, and the width and height of the screen sub-region are obtained.

2. The method according to claim 1, characterized in that The step of obtaining the target coordinates of the element on the grayscale image includes: When the reference object is text, identifying the text in the grayscale image through text recognition, and obtaining the text coordinates of each text in the grayscale image; Get the target text corresponding to the element; The target text is matched in the text to obtain the target coordinates corresponding to the element.

3. The method according to claim 1, characterized in that The step of obtaining the target coordinates of the element on the grayscale image includes: When the type of the reference object is an image, obtaining the image to be searched corresponding to the element; Obtaining the image content edge of the image to be searched; Using image recognition to identify the image on the grayscale image; Scaling the image on the grayscale image using a plurality of preset scaling factors to obtain a scaled image; Obtaining a zoomed image content edge of the zoomed image; matching the image content edge with the scaled image content edge of the scaled image to determine the target image corresponding to the element; Acquiring a relative positional relationship between the target image and the reference object; The target coordinates of the element on the grayscale image are determined according to the relative position relationship and the starting point coordinates.

4. The method according to claim 2, characterized in that Also includes: When the target text cannot be matched in the text, calculating the similarity between the target text and each text on the grayscale image, and determining the maximum similarity; Determining whether the maximum similarity is greater than a preset similarity threshold; If so, the text coordinates of the text corresponding to the maximum similarity are determined as the target coordinates of the element.

5. The method according to claim 4, characterized in that Also includes: When the maximum similarity is not greater than the preset similarity threshold, obtaining the confidence and context information of each of the texts; Correcting the text according to the confidence level and the context information to obtain a corrected text; Calculating the correction similarity between the correction text and the target text, and determining the maximum correction similarity; When the maximum corrected similarity is greater than the preset similarity threshold, the text coordinates of the corrected text corresponding to the maximum corrected similarity are determined as the target coordinates of the element.

6. The method according to claim 1, characterized in that The preset operation includes at least one of adding, deleting, modifying and querying.

7. A software UI automatic verification device, characterized in that: include: An element determination module is used to determine the element to be operated on the software UI of the preset software; A screen sub-area acquisition module, configured to acquire a screen sub-area containing the element on the software UI and acquire area information of the screen sub-area; A screenshot data acquisition module, configured to acquire screenshot data of the screen sub-region according to the region information; A grayscale image conversion module, used to convert the screenshot data into a grayscale image; A target coordinate acquisition module, used to acquire the target coordinates of the element on the grayscale image; An actual coordinate acquisition module, configured to acquire the actual coordinates of the element on the software UI according to the target coordinates and the area information; A UI display result generating module, configured to perform a preset operation on the element according to the actual coordinates to generate a UI display result; A verification module, configured to verify the UI display result using an expected UI display result; The screen sub-area acquisition module includes: The starting point coordinate selection submodule is used to determine the reference object on the software UI, select the starting point coordinates on the reference object, and save the starting point coordinates as a position offset; The relative direction determination submodule is used to determine the relative direction of the element and the reference object; The screen sub-area capture sub-module is used to capture the screen sub-area containing elements on the software UI based on the relative direction, taking the starting point coordinates as the corner point, and obtain the width and height of the screen sub-area.

8. An electronic device, characterized in that: The device includes a processor and a memory: The memory is used to store program code and transmit the program code to the processor; The processor is used to execute the software UI automatic verification method according to any one of claims 1 to 6 according to the instructions in the program code.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium is used to store program code, and the program code is used to execute the software UI automatic verification method according to any one of claims 1 to 6.

Images