Method and device for intelligently identifying and monitoring expected state of man-machine interaction software

Abstract

The invention discloses a human-computer interaction software expected state intelligent identification and monitoring method and device, and relates to the technical field of software test.According to the method, relevant parameters are defined in a monitoring event script, and a software interface image of controlled software is processed based on the relevant parameters; comprising the steps of judging whether a monitoring object exists in a software interface image or not, identifying the monitoring object and judging the software state, automatic and intelligent identification and monitoring of man-machine interaction software can be achieved, wrong watching and missing watching caused by human factors can be avoided, and higher efficiency and accuracy are achieved.

Description

Technical Field

[0001] The present application relates to the field of software testing technology, and in particular to a method and device for intelligently identifying and monitoring the expected state of human-computer interaction software based on machine vision. Background Art

[0002] Human-computer interaction software usually displays the running status of the software in the form of graphics, colors, text displays, etc. During the software testing or joint testing process, testers usually judge whether the software status is consistent with the expected status based on the observed specific images, colors and text displays, so as to judge whether the software has any faults. However, as the scale of software becomes larger and larger, and the fields involved become wider and wider, software testing has become more and more difficult. Traditional manual testing is not only time-consuming and labor-intensive, but the instability of testers will also affect the test results. If software vulnerabilities are not discovered and repaired due to factors such as testers' misreading or omissions, once some key programs fail during use, the software system will fail, causing a series of inestimable and significant losses. Summary of the invention

[0003] The purpose of this application is to provide a method and device for intelligently identifying and monitoring the expected state of human-computer interaction software, which can realize automatic and intelligent identification and monitoring of human-computer interaction software, avoid misreading and missing caused by human factors, and have higher efficiency and accuracy.

[0004] To achieve the above objectives, this application provides the following solutions:

[0005] In a first aspect, the present application provides a method for intelligently identifying and monitoring the expected state of human-computer interaction software, comprising:

[0006] Obtaining a monitoring event script and a software interface image of the controlled software at the current moment, wherein the monitoring event script defines monitoring event frequency information, monitoring object information, expected status and monitoring response information, and the monitoring object is a software interface element whose status needs to be monitored in the monitoring event;

[0007] Determine whether there is a monitored object in the software interface image at the current moment according to the monitored object information, and obtain a first determination result;

[0008] When the first judgment result is no, reacquire the software interface image of the controlled software, and use the reacquired software interface image as the new software interface image at the current moment, and return to the step of "determining whether there is a monitored object in the software interface image at the current moment according to the monitored object information to obtain the first judgment result";

[0009] When the first judgment result is yes, identifying the position information of the monitored object in the software interface image at the current moment, and determining the software state according to the visual features of the identified monitored object to determine whether the software state reaches the expected state, to obtain a second judgment result;

[0010] When the second judgment result is yes, executing a corresponding monitoring response according to the software state and the monitoring response information;

[0011] When the second judgment result is no, judging whether to end the monitoring event according to the monitoring event frequency information, and obtaining a third judgment result;

[0012] When the third judgment result is no, the next round of state identification is entered, and the process returns to the step of "obtaining the monitoring event script and the software interface image of the controlled software at the current moment".

[0013] Optionally, judging whether there is a monitored object in the software interface image at the current moment according to the monitored object information to obtain a first judgment result specifically includes:

[0014] Calculating the similarity between the software interface image at the current moment and the template image to obtain a first similarity, wherein the template image is an image of the page where the monitored object is located;

[0015] Determine whether the first similarity is greater than a similarity threshold to obtain a first judgment result, wherein when the first similarity is greater than the similarity threshold, the first judgment result is that a monitored object exists in the software interface image at the current moment; when the first similarity is less than or equal to the similarity threshold, the first judgment result is that no monitored object exists in the software interface image at the current moment.

[0016] Optionally, identifying the location information of the monitored object in the software interface image at the current moment specifically includes:

[0017] Determine the monitoring area in the software interface image at the current moment according to the monitoring object information;

[0018] Determine the monitored object semantic information according to the monitored area, wherein the monitored object semantic information includes the monitored object semantics and the monitored object semantic location information, and the monitored object semantics refers to the text describing the monitored object on the software interface image;

[0019] Determine the location information of the interface elements in the monitoring area;

[0020] According to the location information of the monitored object semantics and the location information of the interface elements, the location information of the interface elements that meets the target correspondence relationship is selected as the location information of the monitored object in the software interface image at the current moment, wherein the target correspondence relationship includes a nearby position relationship and a concentrated arrangement relationship, the nearby position relationship refers to a position relationship in which the monitored object semantics is located above, below, on the left or on the right of the monitored object, or a position relationship in which the monitored object semantics overlaps with the monitored object position, the concentrated arrangement relationship refers to an arrangement manner of a monitored object semantic group is the same as an arrangement manner of a monitored object group, the monitored object semantic group includes multiple monitored object semantics, and the monitored object group includes multiple monitored objects.

[0021] Optionally, determining the software state according to the identified visual features of the monitored object specifically includes:

[0022] Extracting visual features of the identified surveillance object;

[0023] The visual features are converted into software states according to mapping rules, wherein the mapping rules are mapping relationships between visual features of different data types and software states, and the data types of the visual features include string types, numerical types, and vector types.

[0024] Optionally, the visual features include natural language text, digital text, font type, font size, text color, icon type, rotation angle, image color, image size and position information.

[0025] Optionally, determining the semantic information of the monitored object according to the monitored area specifically includes:

[0026] Determine all the characters and corresponding character positions in the monitoring area using optical character recognition technology;

[0027] Calculating a second similarity between the text and the semantics of the monitored object;

[0028] The text with the second highest similarity is selected as the monitored object semantics, and the position of the text with the second highest similarity is used as the location information of the monitored object semantics.

[0029] Optionally, the interface elements include text interface elements and graphic interface elements; and determining the location information of the interface elements in the monitoring area specifically includes:

[0030] Determine all text interface elements and corresponding text interface element positions in the monitoring area using optical character recognition technology;

[0031] Template matching, edge detection or YOLO target detection is used to determine all graphic interface elements and the corresponding positions of graphic interface elements in the monitoring area.

[0032] In a second aspect, the present application provides a computer device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement any of the above-described methods for intelligently identifying and monitoring the expected state of human-computer interaction software.

[0033] In a third aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the above-described methods for intelligently identifying and monitoring the expected state of human-computer interaction software.

[0034] In a fourth aspect, the present application provides a computer program product, including a computer program, which, when executed by a processor, implements any of the above-mentioned methods for intelligently identifying and monitoring the expected state of human-computer interaction software.

[0035] According to the specific embodiments provided in this application, this application discloses the following technical effects:

[0036] The present application provides a method and device for intelligently identifying and monitoring the expected state of human-computer interaction software. By defining relevant parameters in a monitoring event script and processing the software interface image of the controlled software based on the relevant parameters, including determining whether a monitoring object exists in the software interface image, identifying the monitoring object and determining the software state, the method and device can realize automatic and intelligent identification and monitoring of the human-computer interaction software, avoid misreading and missing caused by human factors, and have higher efficiency and accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0038] Figure 1 This is an application environment diagram of a method for intelligently identifying and monitoring the expected state of human-computer interaction software in Example 1 of the present application;

[0039] Figure 2 A schematic diagram of the positional relationship between the monitoring subject semantics and the interface elements in Example 1 of the present application;

[0040] Figure 3A flowchart of a method for intelligently identifying and monitoring the expected state of human-computer interaction software provided in Example 1 of the present application;

[0041] Figure 4 A schematic diagram of the structure of a computer device provided in Example 2 of the present application. DETAILED DESCRIPTION

[0042] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0043] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0044] Example 1

[0045] The embodiment of the present application provides a method for intelligently identifying and monitoring the expected state of human-computer interaction software, which can be applied to Figure 1In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store data that the server 104 needs to process. The data storage system can be set up separately, integrated on the server 104, or placed on the cloud or other servers. The terminal 102 can send the monitoring event script and the software interface image of the controlled software at the current moment to the server 104. After the server 104 receives the monitoring event script and the software interface image of the controlled software at the current moment, it parses the relevant parameters defined in the monitoring event script and processes the software interface image, including: judging whether there is a monitoring object in the software interface image at the current moment according to the monitoring object information, and obtaining a first judgment result; when the first judgment result is no, re-acquiring the software interface image of the controlled software, and using the re-acquired software interface image as the new software interface image at the current moment, and returning to the step of "judging whether there is a monitoring object in the software interface image at the current moment according to the monitoring object information, and obtaining the first "Judgment result"; when the first judgment result is yes, identify the location information of the monitored object in the software interface image at the current moment, and determine the software state according to the visual features of the identified monitored object; judge whether the software state reaches the expected state to obtain a second judgment result; when the second judgment result is yes, execute the corresponding monitoring response according to the software state and the monitoring response information; when the second judgment result is no, judge whether to end the monitoring event according to the monitoring event frequency information to obtain a third judgment result; when the third judgment result is no, enter the next round of state identification and return to the step "obtain the monitoring event script and the software interface image of the controlled software at the current moment". The server 104 can feedback the obtained second judgment result and third judgment result to the terminal 102. In addition, in some embodiments, the method for intelligently identifying and monitoring the expected state of human-computer interaction software can also be implemented independently by the server 104 or the terminal 102. For example, the terminal 102 can directly use the method for intelligently identifying and monitoring the expected state of human-computer interaction software to process the monitoring event script and the software interface image of the controlled software at the current moment, or the server 104 can obtain the monitoring event script and the software interface image of the controlled software at the current moment from the data storage system and process them using the method for intelligently identifying and monitoring the expected state of human-computer interaction software.

[0046] The terminal 102 may be, but is not limited to, various desktop computers, laptop computers, smart phones, tablet computers, IoT devices, and portable wearable devices. The IoT devices may be smart speakers, smart TVs, smart air conditioners, smart vehicle-mounted devices, etc. The portable wearable devices may be smart watches, smart bracelets, head-mounted devices, etc. The server 104 may be implemented as an independent server or a server cluster consisting of multiple servers, or may be a cloud server.

[0047] Related software testing technologies usually use manual testing, which is time-consuming, labor-intensive and has low accuracy. With the development of machine vision and target detection technology, the automated and intelligent state recognition of monitored objects has become a new demand in software testing. Machine vision refers to the use of cameras or computers to replace human eyes to identify, track and measure targets, and further perform image processing, using computers to process images that are more suitable for human eye observation or transmission to instrument detection. As a major branch of machine vision, target detection plays an irreplaceable and important role in realizing the automation and intelligence of software testing.

[0048] There are many cases of using computer vision to monitor real scenes. Often, the location of the object to be monitored is obtained first, and then an algorithm is designed to extract its state. However, previous studies have focused more on the design of special algorithms for target detection and state analysis of specific monitoring objects in real scenes, which can only be applied to specific and clear monitoring objects. For human-computer interaction software interfaces, the interface elements are not only numerous but also complex in type, which makes it difficult for a specific recognition method to adapt to complex software interfaces. On the one hand, designing an algorithm for each monitoring object will make the development work very cumbersome; on the other hand, compared with a specific recognition task, monitoring the software interface also requires finding the specified monitoring object among many interface elements, and since the software interface often has multiple interface elements with similar recognition features, the interference of non-monitoring objects must be excluded during recognition. Therefore, how to classify and process software interface elements and perform state recognition and monitoring on software with multiple interface elements at the same time requires more in-depth research.

[0049] In this regard, in an exemplary embodiment, a method for intelligently identifying and monitoring the expected state of human-computer interaction software is provided. The method is executed by a computer device, and can be executed by a computer device such as a terminal or a server alone, or by a terminal and a server together. In the embodiment of the present application, the method is applied to Figure 1 The server 104 in the example is used as an example to illustrate the method, which includes the following steps 201 to 208. Among them:

[0050] Step 201, obtain the monitoring event script and the software interface image of the controlled software at the current moment, wherein the monitoring event script defines the monitoring event frequency information, monitoring object information, expected status and monitoring response information, and the monitoring object is the software interface element whose status needs to be monitored in the monitoring event.

[0051] Step 202: determine whether a monitored object exists in the software interface image at the current moment according to the monitored object information, and obtain a first determination result.

[0052] Step 203, when the first judgment result is no, re-acquire the software interface image of the controlled software, and use the re-acquired software interface image as the new software interface image at the current moment, and return to step 202.

[0053] Step 204, when the first judgment result is yes, identify the position information of the monitored object in the software interface image at the current moment, and determine the software status according to the visual features of the identified monitored object.

[0054] Step 205, determining whether the software state has reached the expected state, and obtaining a second determination result;

[0055] Step 206: When the second judgment result is yes, a corresponding monitoring response is executed according to the software status and the monitoring response information.

[0056] Step 207: When the second judgment result is no, determine whether to end the monitoring event according to the monitoring event frequency information to obtain a third judgment result.

[0057] Step 208 , when the third judgment result is no, enter the next round of state identification and return to step 201 .

[0058] By adopting the above steps 201 to 208, the human-computer interaction software interface is non-invasively and intelligently identified and monitored in the expected state by means of machine vision, which can avoid misreading and missing caused by human factors and has higher efficiency and accuracy.

[0059] This embodiment includes the design of the process of intelligent identification and monitoring of expected states and the research of the method of intelligent identification and monitoring of expected states. In terms of process design, the functions of each part of the program and the monitoring process are clarified for the monitoring software. In terms of method research, an implementation method is designed for the monitoring software to perform intelligent identification and monitoring of expected states, including judging the software page type (i.e., judging whether there is a monitoring object in the software interface image at the current moment in step 202, the software page type includes the presence of a monitoring object in the software interface image, and the absence of a monitoring object in the software interface image), dividing the monitoring area, obtaining the location information of the monitoring object (i.e., identifying the location information of the monitoring object in step 204), extracting the software state, and comparing with the expected state. The specific process of this is further explained below.

[0060] Step 1: Definition of monitoring status of software interface elements

[0061] Before designing the intelligent recognition and monitoring process of the expected state of human-computer interaction software based on machine vision, it is necessary to define the monitoring state of the software interface elements. This section defines the monitoring state of the software interface elements from four aspects: monitoring event frequency information, monitoring object information, expected state, and monitoring response information, and describes it in formatted text language so that the monitoring software can read and parse it. In this way, the modeling of monitoring events is realized. The specific research steps are as follows:

[0062] Step 1.1: Determine the monitoring event frequency information. The monitoring event frequency information includes the state recognition frequency and the number of monitoring event executions. The state recognition frequency determines the time interval for the next state recognition when the controlled element does not reach the expected state. If the time interval is not specified, the next recognition will be performed immediately, and after the specified time interval, the next recognition will start after the specified time interval. The number of monitoring event executions determines how many times the controlled element will stop monitoring after reaching the expected state. It can be divided into two types: single-shot and continuous. Single-shot events stop after the controlled element reaches the expected state once, and continuous events stop after the controlled element reaches the expected state several times or never stop.

[0063] Step 1.2: Determine the monitoring object information. The monitoring object information includes the monitoring object, the monitoring area where the monitoring object is located, and the monitoring object semantic information. The monitoring object is the software interface element whose status needs to be monitored in the monitoring event. The determination of the monitoring object is divided into the determination of the monitoring area to which the monitoring object belongs and the determination of the monitoring object semantic information. The detailed steps are as follows:

[0064] Step 1.2.1: Determine the monitoring area to which the monitoring object belongs. The monitoring area is an area where the interface elements are concentrated in the software interface design and separated from the rest of the software interface by boundary lines and outlines. First, divide the software into multiple monitoring areas. By specifying the name of the monitoring area, the monitoring area where the monitoring object is located can be determined, thereby determining the monitoring range of the software.

[0065] Step 1.2.2: Determination of the semantic information of the monitoring object. The semantic information of the monitoring object includes the monitoring object semantics and the target correspondence. The monitoring object semantics is the text that describes the interface elements on the software interface. There is a correspondence between the semantics and the interface elements it describes (i.e., the target correspondence). By specifying the semantics and the correspondence, the monitoring object can be specified within the monitoring area. When the semantics is located near the interface element it describes, it forms four positional relationships with the controlled element: up, down, left, and right, or overlaps with the controlled element (nearby positional relationship). When multiple semantics are concentrated in a certain area according to a specific arrangement, the interface elements described by each of them are also arranged in another area according to the same arrangement. The positional relationship between the monitoring object semantics and the monitoring object is as follows: Figure 2 shown.

[0066] Step 1.3: Determine the expected state. The expected state is the software state that is expected to appear in the monitoring event. During the running of the software, the interface elements will undergo visible state changes as the internal data of the software changes, that is, the visual characteristics of the interface elements have changed. In actual software use, people tend to map the state of interface elements to artificially assigned meanings, which are the software states. For a specific interface element, the software state it expresses changes continuously within an artificially specified range. This range is expressed as the full set, and the expected state is a subset of , which is expressed as . The expected state is the threshold of the monitoring event. When the software state is the expected state, it means that the monitoring event is established.

[0067] Step 1.4: Determine the monitoring response information. The monitoring response is a series of events initiated by the monitoring software when the software reaches the expected state. The monitoring response determines the output when the software reaches the expected state, so that the user can perceive the establishment of the monitoring event. According to the different response contents, the monitoring response is divided into the following three categories, namely sending messages, pop-up warnings, and executing actions:

[0068] (1) Sending messages, that is, outputting the specified content in text form when the expected state is reached. It can be used for monitoring reports on the command line interface. There is no desktop requirement for the monitoring terminal, which is relatively simple.

[0069] (2) Pop-up warning, that is, when the expected state is reached, a dialog box is generated to display the specified content. At this time, the monitoring terminal will pop up a dialog box on the desktop to report the monitoring to the user, which is more eye-catching than the message.

[0070] (3) Executing an action, that is, operating the software or hardware that the monitoring terminal can control when the expected state is reached. This requires the monitoring terminal to have the function of responding to operations and to exchange data with the monitoring software, and the monitoring software initiates the permission for the execution of the action.

[0071] Step 1.5: Describe using formatted text language. For each element of the monitoring status proposed in steps 1.1 to 1.4, it is necessary to input instructions to the monitoring software from the outside to pass the monitored parameters to the software and drive the software to run. The writing method of the monitoring instructions is as follows:

[0072] Monitoring time frequency. Use the parameter "stop" to specify the number of times the software needs to reach the expected state, and the value is an integer; use the parameter "time" to specify the interval time for each state recognition, and the value is a numeric value.

[0073] Semantics of monitoring objects. Use the parameter "area_name" to specify the name of the monitoring area where the monitoring object is located, and the value is a string; use the parameter "name" to specify the semantics of the monitoring object, and the value is a string.

[0074] Expected status. Use the parameter "status" to specify all possible forms of software status, and the value is a string list; use the parameter "map" to specify the rules for mapping the monitoring object status to the software status, and the value is a key-value pair, where the key is the software status and the value is the monitoring object status; use the parameter "except" to specify the expected status, and the value is a string list.

[0075] Monitor response. Use the parameter "response" to specify the monitor response, and the value is a key-value pair. The key "type" specifies the type of the monitor response, and the value is a string; the key "format" specifies the monitor response script, and the value is the script content.

[0076] Step 2: Design of intelligent recognition and monitoring process of expected state of human-computer interaction software based on machine vision

[0077] Based on the definition of monitoring status in step 1, the intelligent identification and monitoring process design of the expected status of human-computer interaction software is carried out, and the prototype architecture of the monitoring software is constructed. The software architecture is as follows: Figure 3As shown. The monitoring software consists of three parts: the main program, the monitoring program and the execution program. The software takes the monitoring event script and the controlled software interface provided by the user as input, and finally outputs in the form of monitoring response, indicating that the controlled software has reached the expected state. The main program part is the core of the monitoring software, responsible for parsing the monitoring event script input by the user, obtaining the controlled software interface and preprocessing it, passing parameters and communicating with the monitoring program and the execution program. The monitoring program part takes the monitoring object, expected state and controlled software interface specified by the monitoring event as input, searches for the target controlled element in the interface, and performs state identification and expected state comparison. The execution program part takes the monitoring response script in the monitoring event script and the current software state as input, and executes the corresponding monitoring response after parsing the response script.

[0078] The detailed process is as follows:

[0079] (1) The main program parses the monitoring event script, obtains parameters such as monitoring event frequency information, monitoring object information, expected status, and monitoring response information, and obtains the software interface image of the controlled software through the acquisition device. Since the controlled software often displays multiple software interfaces at the same time, it is necessary to split the complete screenshot into several single-page screenshots and transmit them to the monitoring program one by one.

[0080] (2) The monitoring program first determines whether the monitored object is located on the current page (i.e., determines the software page type). If the result is true, the current page is divided into a monitoring area, the location information of the monitored object is obtained in the monitoring area, the software status of the monitored object is extracted, and the software status is compared with the expected status specified by the monitoring event, and the comparison result and the current software status are transmitted back to the main program. If the result is false, the controlled software is switched to the page where the monitored object is located, and the interface image is re-obtained for status recognition.

[0081] (3) When the software reaches the expected state, the main program passes the monitoring response parameters and software state to the execution program, and executes the corresponding monitoring response. When the software does not reach the expected state, the software interface is re-acquired to enter the next round of state identification and comparison with the expected state.

[0082] (4) After the monitoring response is completed, the execution completion information is transmitted back to the main program. If the monitoring event is a one-time event, the main program ends the current monitoring event. If the monitoring event is a continuous event, the main program decides to start the next round of state identification or end the monitoring event based on the remaining number of times the software state reaches the expected state.

[0083] Step 3: Research on intelligent recognition and monitoring methods of expected states of human-computer interaction software based on machine vision

[0084] Based on the design of the expected state intelligent identification and monitoring process in step 2, the technical details of each part of the monitoring program are designed to realize the intelligent identification and monitoring of the expected state of the human-computer interaction software. The detailed steps are as follows:

[0085] Step 3.1: Determine the software page type. Since the layout of interface elements of each page is quite different, template matching can be used to calculate the similarity between the current page and the page where the monitoring area is located, and divide the input page image into two categories: "high similarity" and "low similarity". When the similarity is high, the current page is the page where the monitoring area is located. By sliding the template from left to right and from top to bottom on the original image one pixel at a time, a measurement is performed at each position to calculate the matching metric value between the image block and the area where it is located, and it is stored in the image matrix. Each element in the image matrix contains a matching metric value. The correlation coefficient is calculated using standard correlation matching (CV_TM_CCOEFF_NORMED), and the calculation method is as follows:

[0086]

[0087] T '(x',y')= T (x',y')- 1 / (w·h)·∑ x”,y” T (x”,y”);

[0088] I '(x+x',y+y')= I (x+x',y+y')- 1 / (w·h)·∑ x”,y” I(x+x”,y+y”);

[0089] Among them, I is the original image, T is the template image, R is the image matrix, R(x,y) is the matching metric value with the original image coordinate (x,y) as the center and the template image size as the range, w and h are the length and width of the original image. By selecting the maximum value of R(x,y), the similarity between the current page and the page where the monitoring area is located can be obtained. When this similarity is greater than the similarity between the current page and other pages, the current page is considered to be the page where the monitoring area is located.

[0090] Step 3.2: Divide the monitoring area. First, construct a formatted text configuration file externally to describe each monitoring area. The first-level key is the area name, and the value is a dictionary. The key "type" in the dictionary represents the area category, and the value is a string, which is used to classify the monitoring area in the following steps. The key "position" represents the area coordinate range, and the value is the horizontal and vertical coordinates of the upper left corner of the monitoring area and the horizontal and vertical coordinates of the lower right corner of the monitoring area.

[0091] Step 3.3: Obtain the location information of the monitored object. The acquisition of the location information of the monitored object is divided into three stages: acquisition of the semantics of the monitored object, acquisition of the location information of the interface elements, and semantic matching of the monitored object. The detailed steps are as follows:

[0092] Step 3.1.1: Identify the semantic information of the monitored object. The identification of the semantic information of the monitored object can be achieved with the help of optical character recognition (OCR) technology. Take the image of the monitored area as input, perform OCR recognition on it, and obtain all the text and its coordinate positions in the area. Then, for each recognition result, take the monitored object semantics as input, use fuzzy matching to determine the matching degree of all the recognition result texts with the monitored object semantics, and select the recognition result coordinate position with the highest matching degree as the coordinate position of the semantics in the monitored area. The matching degree of fuzzy matching is calculated as follows:

[0093]

[0094] Where X and Y are two string sequences, f[i][j] is the length of the longest common subsequence (LCS) before the i-th position of X and the j-th position of Y, same(a,b) means that when the a-th position of X and the b-th position of Y are exactly the same, it is 1, otherwise it is 0, fuzz.ratio(X ,Y ) is the matching degree between X and Y.

[0095] Step 3.1.2: Obtain the position information of interface elements. Since there are many interface elements in the monitoring area and their types are complex, it is difficult to identify the position information of the monitored object at one time. Therefore, various machine vision technologies and algorithms are first used to obtain the position information of a series of interface elements including the monitored object. The acquisition of interface element position information is divided into text interface elements and graphic interface elements. The text interface elements are interface elements expressed in text, and the graphic interface elements are interface elements expressed in graphics.

[0096] For text-based interface elements, when OCR recognition is used in step 3.1.1, the position information of all text-based interface elements is obtained at the same time, which can be directly used as the output of interface element position information acquisition.

[0097] For graphic interface elements, the following three machine vision algorithms can be used to obtain position information.

[0098] (1) Template matching. When the outline of the controlled element remains basically unchanged and the image recognition feature changes caused by the change of visual features are small, its correlation coefficient with the template can still be at a high level. Same as step 3.2, use the image of the monitored area as the original image and the image of the monitored object as the template, perform template matching on the monitored area, and select the result with the higher correlation coefficient as the output for obtaining the position information of the interface element.

[0099] (2) Edge detection. When the contour and visual features of the controlled element change greatly, template matching may fail due to a small correlation coefficient. However, if the contour boundary of the controlled element is clear, edge detection can be used to identify the position information. Using the image of the monitored area as input, the edge detection algorithm of OpenCV is used to adjust the threshold parameters to obtain the contour information of the interface element as the output of the interface element position information acquisition.

[0100] (3) YOLO target detection. When the position information of the controlled element cannot be obtained using traditional digital image processing algorithms, a controlled element recognition model of the monitoring software can be created, and the position of the controlled element can be obtained using YOLO target detection. The image of the monitoring area and the recognition model are used as input, and the YOLO target detection result is used as the output for obtaining the position information of the interface element.

[0101] Step 3.1.3: Matching of semantic information of monitored objects. According to the correspondence between the semantics of the monitored object and the corresponding interface element (i.e., the monitored object) (i.e., the target correspondence), the location information of the semantics of the monitored object and the location information of the interface element are matched according to the target correspondence, and the matching method of the location information of the monitored object is found from a series of interface element location information. In step 1.2.2, the correspondence between semantics and interface elements is divided into two categories: located near the interface elements and concentrated arrangement. The semantic location information of step 3.1.1, the interface element location information of step 3.1.2, and the location relationship specified in step 1.2.2 are used as inputs to perform semantic matching for the above two correspondence methods.

[0102] When centrally arranged, for a certain semantics, the index of the corresponding interface element in the list storing the recognition results of step 3.1.2 is fixed, so the list is directly searched according to the specified index as the output of the monitored object location information.

[0103] When located near an interface element, the semantic location information is represented by a bounding box as [X1, Y1, X2, Y2] using the coordinates of the upper left corner and the lower right corner. The center point is ( X,Y ); for the i-th controlled element, it is expressed as [x i1 ,y i1 ,x i2 ,y i2 ] and (x i ,y i ). Table 1 shows the discrimination calculation methods of five kinds of position relationships.

[0104] Table 1 Five methods for determining positional relationships

[0105] Positional Relationship Judgment method superior <![CDATA[y i ≥Y∩X1≤x i ≤X2]]> Down <![CDATA[y i ≤ Y∩X1 ≤ x i ≤ X2]]> Left <![CDATA[x i ≥X∩Y1≤y i ≤Y2]]> right <![CDATA[x i ≤X∩Y1≤y i ≤Y2]]> coincide <![CDATA[X1≤x i ≤X2∩Y1≤y i ≤Y2]]>

[0106] After the above processing, several interface elements can be obtained. In order to exclude interface elements that meet the position relationship but are not semantically described, the interface element with the shortest center point distance is selected as the matching result as the output of the monitored object location information. The center point distance is calculated as follows:

[0107]

[0108] Step 3.4: Software status extraction. The extraction of software status is divided into two stages: interface element visual feature extraction and software status mapping. The detailed steps are as follows:

[0109] Step 3.4.1: Visual feature extraction of monitored objects. The visual feature extraction of monitored objects is to take the image of the monitored object as input and output the visual features of the monitored object in the form of computer storage data. Table 2 shows the 10 types of visual features covered by the technical solution of this embodiment, and the visual feature extraction methods are designed for them respectively.

[0110] Table 2 Types of visual features

[0111]

[0112] (1) Natural language text. Take the OCR recognition result matched in step 3.1.3 as input and extract its text content. Since there are errors in the text content recognition results and this type of visual features are often displayed using a few fixed types of text, fuzzy matching is used to calculate the similarity between the recognition results and all the texts that may be displayed by the monitored object. The result with the highest similarity is selected and output as a string as the visual feature extraction result.

[0113] (2) Digital text. Take the OCR recognition result matched in step 3.1.3 as input, extract its text content, and then use regular expressions to extract the numerical part of the text content, and output it as a visual feature extraction result in the form of a numerical value.

[0114] (3) Font type. Take the OCR recognition result matched in step 3.1.3 as input, extract its text location information, use the font recognition model to obtain the font type of the text in the area indicated by the location information, and output it as a visual feature extraction result in the form of a string.

[0115] (4) Font size. Take the OCR recognition result matched in step 3.1.3 as input, extract its text position information, and calculate the height of a single line of text. The height is in pixels. The font size is calculated using the number of pixels per inch (DPI) of the machine where the controlled software is located in the following way and output as a visual feature extraction result in the form of a numerical value.

[0116] pt = px·DPI / 72;

[0117] Among them, pt is the font size and px is pixels.

[0118] (5) Text color. Take the OCR recognition result matched in step 3.1.3 as input, extract the text position information, and sum and average all pixel values ​​in the bounding box while keeping the text content and background color unchanged, to indirectly represent the text color, and output it as a visual feature extraction result in the form of a vector.

[0119] (6) Icon type. To identify the icon type, you need to use template matching or YOLO target detection with classification function in step 3.1.2 to obtain both the location information and the icon category. When using template matching for recognition, the name of the template is output as a string as the visual feature extraction result; when using YOLO target detection for recognition, the classification label of the target in the recognition result is output as a string as the visual feature extraction result.

[0120] (7) Rotation angle. To identify the rotation angle, you need to use template matching or YOLO target detection with angle recognition function in step 3.1.2. When using template matching, the matching degree is calculated by continuously rotating the template, and the template rotation angle corresponding to the highest matching degree is selected as the recognition result, which is output as a visual feature extraction result in the form of a numerical value; when using YOLO target detection, the angle recognition result in the recognition result is selected and output as a visual feature extraction result in the form of a numerical value.

[0121] (8) Image color. The bounding box of the monitored object is used as input, and a feature point, such as the pixel value of the center point of the image, is selected as the image color recognition result, which is output in the form of a vector as the visual feature extraction result.

[0122] (9) Image size. The bounding box of the monitored object is used as input. When obtaining the length of the monitored object, the vertical coordinate difference of the bounding box is selected and output as a numerical value as the visual feature extraction result. When obtaining the width of the monitored object, the horizontal coordinate difference of the bounding box is selected and output as a numerical value as the visual feature extraction result. When obtaining the area of ​​the monitored object, edge detection is used to identify the outline of the monitored object, and the area wrapped by the outline is calculated and output as a numerical value as the visual feature extraction result.

[0123] (10) Position information. The bounding boxes of the monitored object and the reference element are used as input. When the relative positional relationship between the monitored object and a reference element is obtained, the positional relationship between the two is determined by the calculation method in Table 1, and the result of visual feature extraction is output in the form of a string. When the arrangement order of the monitored object in a series of reference elements is obtained, the monitored object and each reference element are located in a certain direction, and the arrangement order of the monitored object in this direction is recorded and output as a visual feature extraction result in the form of a string.

[0124] Step 3.4.2: Software state mapping. The software state mapping is to convert the visual features of the monitored object into the software state at the time of obtaining the software interface information according to the mapping rules. The visual features output in step 3.4.1 include three data types: string, value and vector. The visual feature data is used as input, and different mapping rules are used for software state mapping according to different data types.

[0125] (1) String. Since the visual features of string types are limited and clear, each possible string is mapped to a type of software state. When a string is input, it is directly converted to the corresponding software state.

[0126] (2) Numerical values. Since the visual characteristics of numerical values ​​have changing boundaries, but the changes are relatively continuous, the numerical range is divided into multiple intervals with certain special values ​​as boundaries, and each interval is assigned a software state. When a numerical value is input, it is first determined which interval it is in, and then converted into the software state corresponding to the interval.

[0127] (3) Vector. When recording pixel values ​​in the form of vectors, although the colors only change in a few fixed types, in actual color acquisition, due to problems such as acquisition point offset and color difference in image acquisition, it is difficult to ensure that the acquisition results of the same color remain completely consistent. Therefore, pixel values ​​are pre-collected for each color, and then in actual state recognition, the Euclidean distance is compared with the pre-collected pixel values ​​based on the actual collected pixel values. The Euclidean distance is calculated as follows:

[0128]

[0129] Among them, R, G, B are the three elements of the pixel value, and distance is the Euclidean distance. The pixel value with the shortest Euclidean distance is selected as the matching result, and the software state represented by the color corresponding to the pixel value is output.

[0130] Step 3.5: Compare with the expected state. Take the current software state obtained in step 3.4.2 and the expected state determined in step 1.3 as input. When the expected state has only one state, if the current software state is the same as the expected state, it is determined that the software has reached the expected state. When the expected state contains multiple different software states, if the current software state is an element in the expected state, it is determined that the software has reached the expected state.

[0131] The method for intelligent identification and monitoring of the expected state of human-computer interaction software based on machine vision provided in this embodiment can realize the automatic and intelligent identification and monitoring of human-computer interaction software. Compared with manual monitoring, the use of machine vision can monitor multiple objects at the same time, automatically complete the judgment of whether the software has reached the expected state, avoid misreading and missing caused by human factors, and have higher efficiency and accuracy. It realizes unmanned operation of some monitoring posts, which has lower monitoring costs than manual operation. Using machine vision to obtain the software interface will not cause changes to the original controlled software system, will not invade the terminal where the controlled software is located, and can meet the stability and security requirements of the controlled software.

[0132] The present application also provides an application scenario, which applies the above-mentioned method for intelligently identifying and monitoring the expected state of human-computer interaction software. Specifically: the method for intelligently identifying and monitoring the expected state of human-computer interaction software provided in this embodiment can be applied in scenarios for user experience optimization and system maintenance. The scenarios of user experience optimization and system maintenance include user behavior analysis, system status monitoring, and early warning processing. The method for intelligently identifying and monitoring the expected state of human-computer interaction software provided in this embodiment belongs to the intelligent identification link in system status monitoring. The method for intelligently identifying and monitoring the expected state of human-computer interaction software provided in this embodiment improves the response accuracy of human-computer interaction software and the timeliness of system maintenance through intelligent identification and monitoring, providing users with a smoother user experience and more reliable system services.

[0133] Example 2

[0134] This embodiment provides a computer device, which may be a server or a terminal. Its internal structure diagram may be as follows: Figure 4 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store any data in the method for intelligently identifying and monitoring the expected state of human-computer interaction software. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a method for intelligently identifying and monitoring the expected state of human-computer interaction software provided in Example 1 is implemented.

[0135] Those skilled in the art will understand that Figure 4 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0136] Example 3

[0137] This embodiment provides a computer device, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, a method for intelligently identifying and monitoring the expected state of human-computer interaction software provided in the above-mentioned embodiment 1 is implemented.

[0138] Example 4

[0139] This embodiment provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the method for intelligently identifying and monitoring the expected state of human-computer interaction software provided in the above-mentioned embodiment 1 is implemented.

[0140] Example 5

[0141] This embodiment provides a computer program product, including a computer program. When the computer program is executed by a processor, the method for intelligently identifying and monitoring the expected state of human-computer interaction software provided in the above-mentioned embodiment 1 is implemented.

[0142] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0143] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM may be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0144] The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. The non-relational database may include a distributed database based on blockchain, etc., but is not limited thereto. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but is not limited thereto.

[0145] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0146] This article uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method and core ideas of this application. At the same time, for those skilled in the art, according to the ideas of this application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. A method for intelligently identifying and monitoring the expected state of human-computer interaction software, characterized in that: The method for intelligently identifying and monitoring the expected state of human-computer interaction software includes: Acquire a monitoring event script and a software interface image of the controlled software at the current moment, wherein the monitoring event script includes monitoring event frequency information, monitoring object information, expected status and monitoring response information, and the monitoring object is a software interface element whose status needs to be monitored in the monitoring event; Determine whether there is a monitored object in the software interface image at the current moment according to the monitored object information, and obtain a first determination result; When the first judgment result is no, reacquire the software interface image of the controlled software, and use the reacquired software interface image as the new software interface image at the current moment, and return to the step of "determining whether there is a monitored object in the software interface image at the current moment according to the monitored object information to obtain the first judgment result"; When the first judgment result is yes, identifying the location information of the monitored object in the software interface image at the current moment, and determining the software state according to the visual features of the identified monitored object; Determine whether the software state reaches the expected state, and obtain a second determination result; When the second judgment result is yes, executing a corresponding monitoring response according to the software state and the monitoring response information; When the second judgment result is no, judging whether to end the monitoring event according to the monitoring event frequency information, and obtaining a third judgment result; When the third judgment result is no, the next round of state identification is entered, and the process returns to the step of "obtaining the monitoring event script and the software interface image of the controlled software at the current moment".

2. The method for intelligently identifying and monitoring the expected state of human-computer interaction software according to claim 1, characterized in that: The method further comprises: determining whether there is a monitored object in the software interface image at the current moment according to the monitored object information, and obtaining a first judgment result, specifically including: Calculating the similarity between the software interface image at the current moment and the template image to obtain a first similarity, wherein the template image is an image of the page where the monitored object is located; Determine whether the first similarity is greater than a similarity threshold to obtain a first judgment result, wherein when the first similarity is greater than the similarity threshold, the first judgment result is that a monitored object exists in the software interface image at the current moment; when the first similarity is less than or equal to the similarity threshold, the first judgment result is that no monitored object exists in the software interface image at the current moment.

3. The method for intelligently identifying and monitoring the expected state of human-computer interaction software according to claim 1, characterized in that: Identifying the location information of the monitored object in the software interface image at the current moment specifically includes: Determine the monitoring area in the software interface image at the current moment according to the monitoring object information; Determine the monitored object semantic information according to the monitored area, wherein the monitored object semantic information includes the monitored object semantics and the monitored object semantic location information, and the monitored object semantics refers to the text describing the monitored object on the software interface image; Determine the location information of the interface elements in the monitoring area; According to the location information of the monitored object semantics and the location information of the interface elements, the location information of the interface elements that meets the target correspondence relationship is selected as the location information of the monitored object in the software interface image at the current moment, wherein the target correspondence relationship includes a nearby position relationship and a concentrated arrangement relationship, the nearby position relationship refers to a position relationship in which the monitored object semantics is located above, below, on the left or on the right of the monitored object, or a position relationship in which the monitored object semantics overlaps with the monitored object position, the concentrated arrangement relationship refers to an arrangement manner of a monitored object semantic group is the same as an arrangement manner of a monitored object group, the monitored object semantic group includes multiple monitored object semantics, and the monitored object group includes multiple monitored objects.

4. The method for intelligently identifying and monitoring the expected state of human-computer interaction software according to claim 1, characterized in that: Determine the software status based on the visual features of the identified monitoring object, including: Extracting visual features of the identified surveillance object; The visual features are converted into software states according to mapping rules, wherein the mapping rules are mapping relationships between visual features of different data types and software states, and the data types of the visual features include string types, numerical types, and vector types.

5. The method for intelligently identifying and monitoring the expected state of human-computer interaction software according to claim 4, characterized in that: The visual features include natural language text, digital text, font type, font size, text color, icon type, rotation angle, image color, image size and position information.

6. The method for intelligently identifying and monitoring the expected state of human-computer interaction software according to claim 3, characterized in that: Determining the semantic information of the monitored object according to the monitored area specifically includes: Determine all the characters and corresponding character positions in the monitoring area using optical character recognition technology; Calculating a second similarity between the text and the semantics of the monitored object; The text with the second highest similarity is selected as the monitored object semantics, and the position of the text with the second highest similarity is used as the location information of the monitored object semantics.

7. The method for intelligently identifying and monitoring the expected state of human-computer interaction software according to claim 3, characterized in that: The interface elements include text interface elements and graphic interface elements; Determining the location information of the interface elements in the monitoring area specifically includes: Determine all text interface elements and corresponding text interface element positions in the monitoring area using optical character recognition technology; Template matching, edge detection or YOLO target detection is used to determine all graphic interface elements and the corresponding positions of graphic interface elements in the monitoring area.

8. A computer device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method for intelligently identifying and monitoring the expected state of human-computer interaction software according to any one of claims 1 to 7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for intelligently identifying and monitoring the expected state of human-computer interaction software described in any one of claims 1 to 7 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method for intelligently identifying and monitoring the expected state of human-computer interaction software described in any one of claims 1 to 7 is implemented.

Images