Abnormality detection method and device, terminal equipment and storage medium

Abstract

The invention relates to an anomaly detection method and device, terminal equipment and a storage medium. A target control in a first display interface is determined; the probability that the user input operation acts on the target control is maximum; displaying a second display interface associated with the target control; detecting whether the second display interface has display abnormity or not through multiple detection modes; and reporting abnormal information under the condition that the display of the second display interface is abnormal. The target control is the control which is most likely to be operated by the user input operation, so that the real use scene of the user is simulated, and the target control determined by the scheme is more targeted. Whether the display of the second display interface is abnormal or not is detected through multiple detection modes, and the efficiency and accuracy of detecting the display abnormality of the second display interface are improved. And reporting abnormal information under the condition that the display of the second display interface is abnormal.

Description

Technical Field

[0001] The present disclosure relates to the field of information processing technology, and in particular to an anomaly detection method, apparatus, terminal device and storage medium. Background Art

[0002] With the development of smart devices, smart devices are used in more and more fields and scenarios. For example, mobile smart terminals account for a large proportion of smart devices. These smart devices usually have a display screen and are also installed with applications. Different applications have different functions; the display screen can display various interfaces of the smart device. In some cases, during the operation of the application, some interfaces may display abnormally due to various problems, affecting the display effect and thus affecting the user experience. Summary of the invention

[0003] The present invention provides an anomaly detection method, an apparatus, a terminal device and a storage medium.

[0004] According to a first aspect of an embodiment of the present disclosure, there is provided an abnormality detection method, comprising: determining a target control in a first display interface; wherein the probability that a user input operation acts on the target control is greatest; displaying a second display interface associated with the target control; detecting whether a display abnormality occurs in the second display interface through a plurality of detection methods; and reporting abnormality information when the display abnormality occurs in the second display interface.

[0005] In one embodiment, determining the target control in the first display interface includes: obtaining the hierarchical relationship between each interface in the application to which the first display interface belongs; and determining the target control according to the feature information of each preset control in the first display interface and the hierarchical relationship through a target reinforcement learning model.

[0006] In one embodiment, the method further comprises:

[0007] Acquire multiple training sample sets, wherein the training sample sets include: a first display interface sample, a reference control sample, a target control sample, display information of a second display interface sample associated with the target control sample, historical data of each control in each first display interface sample as the target control, and a hierarchical relationship between each interface of the first display interface sample in the application; use the training sample set to update the parameters of the value function of the value network in the initial reinforcement learning model to obtain the target reinforcement learning model; wherein the value function is used to update the parameters of the decision function of the decision network in the initial reinforcement learning network model, and the decision network is used to determine the target control.

[0008] In one embodiment, the display information at least includes: a display duration of the second display interface sample.

[0009] In one embodiment, the method further comprises: extracting characteristic information of the preset control, wherein the characteristic information comprises at least: location information, icon information, text information and / or unread message mark information.

[0010] In one embodiment, the first display interface is an interface in an application or a system interface; the method further comprises: when the first display interface is an interface in an application, detecting the stability of the application by means of program instrumentation.

[0011] In one embodiment, the method further includes: detecting whether a display abnormality occurs on the first display interface.

[0012] In one embodiment, the display abnormality includes at least: text or picture overlap; image missing; overexposure; patches; white screen; black screen; flowery screen; and / or abnormal control position.

[0013] According to a second aspect of an embodiment of the present disclosure, there is provided an abnormality detection device, comprising: a determination module, for determining a target control in a first display interface; wherein the probability that a user input operation acts on the target control is the highest; a display module, for displaying a second display interface associated with the target control; a detection module, for detecting whether a display abnormality occurs in the second display interface through a plurality of detection methods; and a reporting module, for reporting abnormality information when the display abnormality occurs in the second display interface.

[0014] According to a third aspect of the embodiments of the present disclosure, a terminal device is provided, including:

[0015] A processor and a memory for storing executable instructions that can be run on the processor, wherein: when the processor is used to run the executable instructions, the executable instructions execute the method described in any one of the above embodiments.

[0016] According to a fourth aspect of the embodiments of the present disclosure, a non-temporary computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, the method described in any of the above embodiments is implemented.

[0017] The technical solution provided by the embodiments of the present disclosure may have the following beneficial effects:

[0018] The scheme of the disclosed embodiment can determine the target control in the first display interface in the first display interface. The probability that the user input operation acts on the target control is the highest, that is, the target control is the control that the user input operation is most likely to operate, thereby simulating the user's real usage scenario. The target control determined by the scheme is more targeted, and the operation of the control with the highest probability of user use is improved, so as to facilitate the display of the second display interface associated with the target control and the subsequent detection of the display of the second display interface. The second display interface is detected by multiple detection methods to see if the display is abnormal, which improves the efficiency and accuracy of detecting the display abnormality of the second display interface. In the case of display abnormality in the second display interface, the abnormal information is reported.

[0019] The first display interface can be the current display interface, and then the second display interface is determined, and the display of the second display interface is detected. After the detection of the second display interface is completed, the second display interface is used as the first display interface, and the target control in the second display interface is further determined. In this way, the display of continuous interfaces can be detected, and these interfaces are the interfaces that are most likely to be operated during the simulated user use process, thereby realizing automatic detection without manual operation, having strong versatility, and improving the detection effect.

[0020] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0022] Figure 1 is a schematic diagram showing an abnormality detection method according to an exemplary embodiment;

[0023] Figure 2 is a schematic diagram showing a method of determining a target control according to an exemplary embodiment;

[0024] Figure 3 is a schematic diagram of obtaining a target reinforcement learning model according to an exemplary embodiment;

[0025] Figure 4 is a schematic diagram showing another method of determining a target control according to an exemplary embodiment;

[0026] Figure 5 is a schematic diagram showing a method of detecting anomalies according to an exemplary embodiment;

[0027] Figure 6is a schematic diagram showing another method of detecting anomalies according to an exemplary embodiment;

[0028] Figure 7 is another detection schematic diagram shown according to an exemplary embodiment;

[0029] Figure 8 is a partial schematic diagram of a target detection model according to an exemplary embodiment;

[0030] Fig. 9 is a partial schematic diagram of another target detection model according to an exemplary embodiment;

[0031] Fig.10 is a schematic diagram of an abnormality detection device according to an exemplary embodiment;

[0032] Fig.11 is a schematic diagram of another testing process according to an exemplary embodiment;

[0033] Fig.12 is a schematic diagram of a testing system according to an exemplary embodiment;

[0034] Fig.13 The present invention is a block diagram of a terminal device according to an exemplary embodiment. DETAILED DESCRIPTION

[0035] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices consistent with some aspects of the present disclosure as detailed in the appended claims.

[0036] refer to Figure 1 , is a schematic diagram of an anomaly detection method, the method comprising:

[0037] S100: Determine a target control in the first display interface; wherein the probability that the user input operation acts on the target control is the highest.

[0038] S200: Displaying a second display interface associated with the target control.

[0039] S300: Detecting whether the second display interface has display abnormality by using multiple detection methods.

[0040] S400: When a display abnormality occurs on the second display interface, report abnormality information.

[0041] The method can be applied to terminal devices, which may include mobile terminals and fixed terminals. Mobile terminals may be mobile phones, tablet computers, wearable devices, vehicle-mounted central control devices, smart homes, and other devices with display screens and installed applications.

[0042] For S100, an operating system and various applications are usually installed in the terminal device. The operating system has a desktop interface, and the desktop has various installed applications. Each application can be displayed through the desktop. Each application has a different display interface, and the interfaces in different applications may be different. The interface in the application usually has multiple controls, and these controls are used to realize the jump and switching of the interface in the application, thereby realizing different functions. Different controls have different functions, and according to different controls, you can switch to the associated second display interface. In this embodiment, each control in the display interface is recorded as a preset control, that is, there are multiple preset controls in the first display interface.

[0043] Exemplarily, each preset control is associated with a second display interface.

[0044] Exemplarily, the first display interface includes a plurality of preset controls, and a target control is determined from the plurality of preset controls.

[0045] Exemplarily, any control with an operable function may be used as a preset control, and the operable function may also be a control that can respond to an input operation acting on the control.

[0046] For example, the type of application is not limited, and may be any type of application, for example, instant messaging, game, video playback, map, or other types of applications.

[0047] Any display interface can be used as the first display interface. Exemplarily, the first display interface is a preset interface, which can be an interface in a preset application, a desktop, or other interfaces.

[0048] Exemplarily, the first display interface is the current display interface of the display screen of the terminal device.

[0049] Exemplarily, the style, size, color, position, etc. of each control in the first display interface are not limited and can be determined according to the display requirements of the first display interface.

[0050] After determining the first display interface, since the first display interface includes multiple controls, a target control is determined from the controls of the first display interface. The probability that the user input operation acts on the target control is the highest, that is, the target control is the control that the user input operation is most likely to operate.

[0051] The method of determining the target control is not limited, and any method that can determine the target control from multiple controls in the first display interface is within the protection scope of this embodiment. For example, it can be determined by a reinforcement learning network model, or by a preset algorithm.

[0052] In one embodiment, it may also include:

[0053] Detect or identify each control in the first display interface to obtain a preset control, so as to facilitate determining a target control from each preset control. Each control in the first display interface can be determined by a detection algorithm or a recognition algorithm or a trained detection model.

[0054] Exemplarily, each control has a control identifier, and the control identifier is used to indicate that the area is a control. When detecting a preset control, it can be determined whether it is a preset control by detecting or identifying the control identifier, and the control identifiers of different preset controls may be different. The control identifier may include a text identifier and / or a graphic identifier, etc.

[0055] For S200: each control in the first display interface is associated with a corresponding interface, and according to the operation performed on the control, the display interface associated with the control can be switched.

[0056] For example, the current display interface of application A includes preset controls 1, preset controls 2, and preset controls 3 with different functions and associated with different interfaces. Preset control 1 is associated with display interface 1, preset control 2 is associated with display interface 2, and preset control 3 is associated with display interface 3. By operating on preset control 1, you can switch from the current interface to display interface 1, and the interface displayed on the display screen is display interface 1. By operating on preset control 2, you can switch from the current interface to display interface 2, and the interface displayed on the display screen is display interface 2. By operating on preset control 3, you can switch from the current interface to display interface 3, and the interface displayed on the display screen is display interface 3. The current interface here can be the first display interface. The operations acting on each preset control include click operations or other operations that can switch from the current interface to the interface associated with the preset control.

[0057] For another example, the current display interface is the desktop of the terminal device, that is, the first display interface is the desktop, and the desktop includes various installed applications. In this case, the icons of various applications in the current display interface are preset controls. For example, the desktop includes application A, application B, and application C, then the icon of application A is used as preset control 1, the icon of application B is used as preset control 2, and the icon of application C is used as preset control 3. By operating on the icon of application A (such as clicking), you can switch from the desktop to the interface in application A. By operating on the icon of application B (such as clicking), you can switch from the desktop to the interface in application B. By operating on the icon of application C (such as clicking), you can switch from the desktop to the interface in application C.

[0058] After determining the target control, the second display interface associated with the target control is displayed. The content of the second display interface can be determined according to the target control, and each target control is associated with its own second display interface. Exemplarily, different target controls are associated with different second display interfaces.

[0059] Exemplarily, after the target control is determined, the second display interface may be displayed according to the operation performed on the target control.

[0060] In one embodiment, the display form of the second display interface can be full screen display, or can be displayed in the form of a floating window based on the first display interface.

[0061] For S300: After displaying the second display interface, detect whether the second display interface has a display abnormality through a variety of different detection methods. There can be a variety of ways to detect whether the second display interface has a display abnormality, as long as the method can achieve display abnormality detection. For example, detection can be performed through a detection algorithm, and detection can also be performed through a trained detection model. Each detection method can detect display abnormalities on the second display interface through a detection network. The network structures of different detection networks can be different, and the processing process of the second display interface is different, and the detection results may also be different. These detection networks can be trained networks.

[0062] Exemplarily, the display abnormality includes at least: text or picture overlap, image missing, overexposure, patches, white screen, black screen, distorted screen and / or abnormal control position, etc. Of course, there may be other display abnormalities.

[0063] In this embodiment, the second display interface is detected by a plurality of different detection methods, and different detection methods have different focuses on the second display interface and different detection accuracy. In this way, the results of the detection by the plurality of different detection methods can be combined to determine whether the second display interface has display abnormality, thereby improving the efficiency and accuracy of the detection.

[0064] S400: When a display abnormality occurs on the second display interface, abnormal information is reported, which may be reported to the cloud or server.

[0065] Exemplarily, the abnormal information may be first indication information indicating that the second display interface is displaying an abnormal interface, and the first indication information may be information in any form, such as characters.

[0066] Exemplarily, when the second display interface is a normal display interface, the normal display information is reported. The normal display information may be second indication information indicating that the second display interface is a normal display interface, and the second indication information may be information in any form, such as characters. The first indication information is different from the second indication information.

[0067] Exemplarily, the abnormal information may include the presence of a display abnormality on the second display interface, location information of the display abnormality, the abnormality type, a screenshot of the abnormal area, and / or a screenshot of the second display interface, etc.

[0068] The solution of the embodiment of the present disclosure can determine the target control in the first display interface in the first display interface, and the probability that the user input operation acts on the target control is the highest, that is, the target control is the control that the user input operation is most likely to operate, thereby simulating the user's real usage scenario. The target control determined by the solution is more targeted, and the operation of the control with the highest probability of user use is improved, so as to facilitate the display of the second display interface associated with the target control and the subsequent detection of the display of the second display interface.

[0069] By using multiple detection methods to detect whether the second display interface displays abnormalities, the efficiency and accuracy of detecting display abnormalities on the second display interface are improved. When display abnormalities occur on the second display interface, abnormal information is reported.

[0070] The first display interface can be the current display interface, and then the second display interface is determined, and the display of the second display interface is detected. After the detection of the second display interface is completed, the second display interface is used as the first display interface to continue to determine the target control in the second display interface. In this way, the display of continuous interfaces can be detected, and these interfaces are the interfaces that are most likely to be operated during the simulated user use process. Then, the second display interface is detected for display anomalies by combining a variety of different detection methods, thereby realizing the automation of the process from determining the target control to detecting whether the second display interface displays abnormalities and then reporting abnormal information. The whole process does not require manual operation, which improves the detection accuracy and reduces the inaccuracy caused by a single test method for the second display interface test. Any application can be detected, with strong versatility.

[0071] Exemplarily, the method further includes: acquiring a screenshot of the first display interface, and determining a target control according to the screenshot of the first display interface.

[0072] Exemplarily, a screenshot of the second display interface is obtained, and the second display interface is detected for display abnormality according to the screenshot of the second display interface. Exemplarily, the second display interface is in the form of an image.

[0073] In one embodiment, reference Figure 2 , is a schematic diagram of determining a target control, the method comprising:

[0074] S101, obtaining the hierarchical relationship between various interfaces in an application to which a first display interface belongs.

[0075] S102, determining a target control according to feature information and hierarchical relationships of each preset control in the first display interface through a target reinforcement learning model.

[0076] For S101, it can be obtained through a corresponding application page hierarchical structure relationship acquisition tool, and can also be obtained from a terminal system. The terminal system can store page relationship information of various display interfaces of the application, such as a page relationship tree.

[0077] Exemplarily, the hierarchical relationship includes the association relationship between the first display interface and the upper and lower interfaces of the first display interface. The first display interface may have multiple upper and lower display interfaces, and may also have multiple lower and lower display interfaces. A preset control in the first display interface may be associated with a lower and lower display interface of the first display interface.

[0078] Exemplarily, when a display interface has a preset control and the interface associated with the preset control is a first display interface, the display interface is a parent display interface of the first display interface.

[0079] In one embodiment, the hierarchical relationship between the interfaces in the application to which the first display interface belongs may also be determined according to the relationship between the second display interfaces associated with the preset controls in the display interfaces in the application to which the first display interface belongs.

[0080] For S102, after obtaining the characteristic information of each preset control in the first display interface and the hierarchical relationship between each interface in the application to which the first display interface belongs, a target reinforcement learning model can be used to determine a target control from the preset controls based on the characteristic information of the preset controls and the hierarchical relationship between each interface in the application to which the first display interface belongs, that is, the target control is the control that is most likely to be operated by the user input operation.

[0081] The target reinforcement learning model here is a trained model, which includes a value network and a decision network. The decision network is used to determine the target control from the preset controls according to the feature information and hierarchical information of each preset control in the first display interface. The decision network has a decision function, and the decision function has a related first parameter, which can be multiple.

[0082] The value network is used to evaluate the target control determined by the decision network and output the value. The output value can also be called a reward. The output value or reward is used to evaluate the quality of the target control determined by the decision network. The value or reward price can be positive or negative. The evaluation or reward output by the value network is used by the decision network to determine the target control for the next time, so that the accuracy of the target control determined next time as the control that is most likely to be affected by the user operation can be improved, making the test more targeted and reducing the determination of other controls that are less used in daily life, thereby improving the test efficiency.

[0083] Exemplarily, the method of determining the target control is not limited, and any method that can determine the target control from multiple controls in the first display interface is within the protection scope of this embodiment, for example, determination by other non-reinforcement learning preset algorithms, etc.

[0084] Since there is an association relationship between each control in the first display interface and the second display interface, and there can be multiple upper-level display interfaces and lower-level display interfaces of the first display interface, the lower-level display interface of the first display interface can be determined through the hierarchical relationship. The lower-level display interface of the first display interface may include the lower-level display interface of the first display interface, and may also include subsequent multi-level display interfaces, and the contents displayed in different lower-level display interfaces may be different.

[0085] The next level display interface associated with the preset control may not have content that the user is interested in, but the lower level display interface after the next level of the preset control may have content that the user is interested in or needs. In this case, the preset control may also serve as the target control.

[0086] Therefore, the target control determined by the trained target reinforcement learning model based on the hierarchical relationship and the characteristic information of each control in the first display interface is more likely to be the control required for the user operation.

[0087] Exemplarily, the target control can also be determined through a target reinforcement learning model based on the N consecutive upper-level interfaces before the first display interface, the characteristic information of each preset control in the first display interface, and the hierarchical relationship. The N consecutive upper-level interfaces before the first display interface are used to determine the display path of the first display interface. The display path can be used as a reference trend for determining the target control, so the target control can also be determined based on the display path.

[0088] Exemplarily, the target control is determined through a target reinforcement learning model in combination with feature information of a preset control, a hierarchical relationship, and a display path for displaying the first display interface.

[0089] Since the first display interface may include multiple preset controls, all of the preset controls in the first display interface may serve as target controls, so there may be multiple next-level display interfaces of the first display interface. By combining the characteristic information, hierarchical relationships, and display paths of the preset controls and displaying the first display interface, the probability of determining the target control as the control required for the user's operation can also be increased.

[0090] The scheme of the disclosed embodiment can determine the target control through the trained target reinforcement learning model according to the hierarchical relationship of each interface in the application to which the first display interface belongs and the characteristic information of each preset control in the first display interface. The probability that the user input operation acts on the target control is the highest, that is, the target control is the control that the user input operation is most likely to operate. In this way, the user's real usage scenario can be simulated, and the detection of the scheme is more targeted, which improves the test operation of the control with the highest probability of user use, so that the test is focused and the test of other controls is reduced, so as to facilitate the display of the second display interface associated with the target control and the subsequent detection of the display of the second display interface.

[0091] The first display interface can be the current display interface, and then the second display interface is displayed through the target control to complete the test of the first display interface. After the first display interface is completed, the second display interface is used as the first display interface to continue to determine the target control in the second display interface. In this way, continuous display interfaces can be tested, and these interfaces are the most relevant interfaces in the simulated user use process, thereby realizing automated testing without manual operation, with strong versatility and improved test efficiency.

[0092] Exemplarily, the method further includes: obtaining a screenshot of the first display interface, and determining a preset control and a target control based on the screenshot of the display interface.

[0093] In one embodiment, it may also include:

[0094] Extract characteristic information of each preset control, where the characteristic information may at least include: location information, icon information, text information and / or unread message mark information.

[0095] Each control has its own characteristics, and the characteristic information between different controls is also different. The preset controls have their own position information. Different preset controls have different positions in the first display interface, and each preset control can be distinguished by the position information. The preset controls have their own icon information, and different control icons are different. The preset control can also have text information, and the text information is used to identify the name of the preset control. The preset control can also have message identification information, such as unread message mark information. In instant messaging applications, if the first display interface has message sending and receiving and message prompt functions, it will have message identification information after receiving the message. When there is message identification information on the preset control, it means that the user may read the corresponding message, so the message identification information increases the probability of the preset control being the target control.

[0096] The above information can be used as the characteristic information of the preset control, and can also be combined as the characteristic information of the preset control. The more information the characteristic information includes, the easier it is to determine the target control.

[0097] Exemplarily, the feature information may also include type information of a preset control, and the preset control may have type information.

[0098] There is no limitation on the method of extracting feature information of each preset control. It can be extracted through a feature extraction algorithm or a trained feature extraction model. The specific feature extraction algorithm can be determined according to actual needs. The methods that can extract feature information of each control are all within the scope of this embodiment.

[0099] In one embodiment, after determining the preset controls in the first display interface, the quantity information of the preset controls can be determined. In S100, the target control can also be determined based on the quantity information of the preset controls in the first display interface. The greater the quantity, the greater the difficulty in determining the target control.

[0100] In one embodiment, reference Figure 3 , is a schematic diagram of a target reinforcement learning model, including:

[0101] S1, obtaining multiple training sample sets. Each training sample set includes: a first display interface sample, a reference control sample, a target control sample, display information of a second display interface sample associated with the target control sample, historical data of each control in each first display interface sample as a target control, and a hierarchical relationship between each interface of the first display interface sample in an application.

[0102] S2, using the training sample set to update the parameters of the value function of the value network in the initial reinforcement learning model to obtain the target reinforcement learning model. The value function is used to update the parameters of the decision function of the decision network in the initial reinforcement learning network model to obtain the target reinforcement learning model; the decision network is used to determine the target control.

[0103] The target reinforcement learning model in this embodiment is a trained reinforcement learning model. The training process can be performed through S1 and S2, and at least the value network in the initial reinforcement learning model is updated to obtain a trained target reinforcement learning model. The parameters of the value function of the value network in the initial reinforcement learning model are updated through the training of the above-mentioned multiple training sample sets.

[0104] The value network in the initial reinforcement learning model is updated by obtaining multiple training sample sets. The larger the number of training sample sets, the higher the probability that the target control determined by the trained target reinforcement learning model is the control acted upon by the user operation.

[0105] Each training sample includes: a first display interface sample, a reference control sample, a target control sample, display information of a second display interface sample associated with the target control sample, historical data of each control in each first display interface sample as a target control, and the hierarchical relationship between each interface of the first display interface sample in the application.

[0106] The first display interface sample may be a sample interface of each display interface in the application, each first display interface sample corresponds to a reference control sample, and the reference control sample is a control operated by a user. The target control sample is a control determined by a reinforcement learning model.

[0107] The display information includes at least: the display duration and display content of the second display interface sample. The longer the display duration, the higher the user's interest in the second display interface, and the greater the probability that the target control associated with the second display interface is the control operated by the user. The higher the correlation and consistency between the display content of the second display interface sample and the display content of the display interface associated with the reference control sample, the higher the probability that the target control sample is the reference control sample.

[0108] The historical data may include the number of times each control in the first display interface sample is used as a target control. The greater the number of times, the greater the probability that the corresponding control is used as a target control.

[0109] Since the information in the above training sample set is all related to the target control samples determined by the decision network in the initial reinforcement learning model, the information in the above training sample set will affect the value (reward) output by the value network. Therefore, the value function of the value network in the initial reinforcement learning model is updated through the above training samples, and the parameters of the value function are adjusted so that the value or reward output by the value network according to the decision of the decision network is more accurate.

[0110] The training sample set is used to update the parameters of the value function of the value network in the initial reinforcement learning model. The value function is used to update the parameters of the decision function of the decision network in the initial reinforcement learning network model, and the decision network is used to determine the target control.

[0111] The initial reinforcement learning model may include a value network and a policy network, wherein the value network has a value function and the policy network has a policy function. The policy network is used to determine the target control, and the value network is used to evaluate the target control determined by the policy network and give the value of the decision. The value given by the updated value network is used to update the decision function of the decision network to facilitate making a decision next time and determine the target control, thereby improving the accuracy of determining the target control.

[0112] In one embodiment, the training sample set may also include: feature information of each control in the first display interface sample, so that the decision network can determine the target control sample according to the feature information of each control in the first display interface sample during the training process.

[0113] In one embodiment, it is also possible to determine the association between the target control sample and the display information of the second display interface sample associated with the target control sample, the reference control sample, the display information of the display interface associated with the reference control sample, the historical data of each control in each first display interface sample as a target control, and the hierarchical relationship between each interface of the first display interface sample in the application, and then update the parameters of the value function in the value network based on the association.

[0114] In one embodiment, the training sample set may also include: the first N consecutive upper-level interfaces of the first display interface sample, the characteristic information of each control in the first display interface sample, and the hierarchical relationship of each display interface in the application to which the first display interface sample belongs. The first N consecutive upper-level interfaces of the first display interface sample are used to determine the display path sample of the first display interface sample, and the display path sample can be used as a trend sample for determining the target control sample.

[0115] Since the first display interface sample may include multiple controls, all controls in the first display interface sample may be used as target control samples, so there may be multiple next-level display interfaces of the first display interface sample. By combining the characteristic information of the controls in the first display interface sample, the hierarchical relationship of each display interface in the application to which the first display interface sample belongs, and the display path sample of the first display interface sample, the probability that the determined target control is the control required for the user operation can also be increased. This facilitates the update of the value network, making the value output by the value network more accurate, and further increasing the probability that the target control determined by the decision network is the control acted upon by the user operation.

[0116] In one embodiment, the target reinforcement learning model can also be updated by updating the value network in the target reinforcement learning model based on the target control determined from the first display interface, the hierarchical relationship between the various interfaces in the application to which the first display interface belongs, and the characteristic information of each preset control in the first display interface.

[0117] In one embodiment, the reinforcement learning model may be an Actor-Critic model.

[0118] refer to Figure 4 , which is a schematic diagram of another method for determining the target control.

[0119] The first display interface is Figure 4 The original image in the first display interface includes controls such as "WeChat", "Address Book", "Discover", "Me", "Services", "Favorites", "Friends Circle", "Emoji", "Settings" and "Status", and these controls can be identified from the first display interface through the recognition model.

[0120] The target reinforcement learning model can be a model completed through S1 and S2 training, including the Actor-Critic network model. The target reinforcement learning model includes a value network and a policy network. The value network is used to optimize the decision of the policy network. The policy network is updated according to the value output of the value network to facilitate the policy network to make the next decision and determine the target control. The target control is as follows Figure 4 In the picture corresponding to the intelligent simulated click behavior, the "WeChat" control, the second display interface is the interface associated with the "WeChat" control.

[0121] The target reinforcement learning network may also include an action sampler, which collects the actions output by the strategy network, and may mark the target control clicked after the action is executed as 1, and mark other controls that are not clicked as 0.

[0122] refer to Figure 4 ,The process of determining the target control is mainly divided into three steps: data preprocessing 1, representation learning 2-3 and reinforcement learning 4.

[0123] S100 includes the following steps 1-4:

[0124] 1. Obtain the first display interface, including obtaining and saving a screenshot of the application page.

[0125] After obtaining the first display interface, it may also include extracting feature information of preset controls in the first display interface, including identifying page information through algorithms such as target detection, such as the number of controls, control coordinates, control clickability, visibility and other attributes as page layout attributes. The user's historical operation behavior is obtained by using system interfaces, logs, operation dot and other operations, such as the path of the user's historical operation, page dwell time, usage permissions and other attributes as user data.

[0126] 2. Process the page layout data and user data, and use feature engineering methods to encode the data into 0 and 1.

[0127] 3. After the state encoding is completed, since the user data and page layout data are highly correlated with the user's behavioral operations, the neural network MLP is used to model the complex relationship between the data, establish the relationship between the user data and the page layout data, and extract the representation characteristics.

[0128] 4. After converting the page layout data and user data into standard features, the feature factors are input into the reinforcement learning network. The reinforcement learning network adopts the Actor-Critic network structure that combines strategy and value.

[0129] Based on the reinforcement learning process constructed above, the target recognition algorithm is used to obtain page information. After obtaining page layout information such as label and coordinate information, it is combined with user historical behavior data and input into the feedforward neural network to learn the relationship features. Then the feature factors are input into reinforcement learning for decision-making, and the habitual path that conforms to user operations is obtained. The click operation is performed using an automated script. When entering the next page, the algorithm will continue to be used for identification, and the relationship between the current page control attributes and user data will be established. The next response path is obtained through the reinforcement learning algorithm. This cycle continues to imitate user behavior until the operation test of all paths is completed.

[0130] In the automated detection phase, the combination of reinforcement learning algorithm and target detection algorithm is mainly used. First, the target detection algorithm is used to extract the page information of the application, such as the number of controls in the page, the page path, the control coordinates, etc., and then these page attribute information is input into the multi-layer perceptron network for representation learning. Then, the learned information matrix is ​​input into the reinforcement learning algorithm to decide the path of the next frame operation, and realize intelligent simulation of click behavior. This solves the problem mentioned in the related technology that the existing test tools cannot solve the problem of user login and the unstable test coverage.

[0131] In one embodiment, reference Figure 5 , is a schematic diagram of detecting anomalies, Figure 6 , is another schematic diagram of detecting anomalies, combined with Figure 5 and Figure 6 , S300, detecting whether a display abnormality occurs on the second display interface, including:

[0132] S301, detect the second display interface by a first detection method to obtain a first detection result. Figure 4 The multi-layer network filtering in the , can detect whether the second display interface has display abnormality through the multi-layer convolutional network (Multi-Layer-CNN). The second display interface is the picture sample, corresponding to Figure 6 Sample collection annotation in .

[0133] S302, when the first detection result is that the display is normal, detect the second display interface by a second detection method to obtain a second detection result.

[0134] S303, determining whether the second display interface has display abnormality according to the second detection result, which is a process of performing secondary detection on the second display interface. The second detection result indicates whether the second display interface has display abnormality.

[0135] S304, when the first detection result is a display abnormality, the second display interface is detected by a third detection method to obtain a third detection result.

[0136] S305: Determine, based on the second detection result and the third detection result, a target detection result for indicating whether a display abnormality occurs on the second display interface.

[0137] Among them, the first detection method, the second detection method and the third detection method are all different.

[0138] In this embodiment, when detecting whether the second display interface displays abnormality, the second display interface is detected by a plurality of different detection methods, thereby improving detection accuracy and reducing missed detection situations.

[0139] The first detection method, the second detection method and the third detection method may be detection methods based on different target detection algorithms, and different target detection algorithms have different detection accuracy.

[0140] Exemplarily, the first detection method, the second detection method and the third detection method respectively correspond to their own trained network models, which are trained based on abnormal samples and normal samples. The second display interface can be used as input of these network models, and whether the second display interface displays an abnormality can be determined based on the output of these network models.

[0141] For example, the first detection method can be a method of detecting the second display interface based on a trained multi-layer convolutional network model (Multi-Layer-CNN). The second detection method can be a detection method based on a YOLO model, and the third detection method can be a target detection method based on a Faster-RCNN model. These network models are all trained models, which can be based on supervised learning training or unsupervised learning training. After the second display interface is input into these models, the output results of these models are the results of whether the second display receives an abnormal display. The process of combining multiple detection methods to determine whether the second display interface displays an abnormality belongs to the process of integrating multiple models.

[0142] Of course, the first detection method, the second detection method and the third detection method may also be other detection methods. In short, the three detection methods are detection methods based on different target detection algorithms or models.

[0143] The first detection method is used to perform a first detection on the second display interface to obtain a first detection result. If the first detection result is that the display is normal, the second display interface is detected by the second detection method for a second detection to obtain a second detection result, and then it is determined whether the second display interface displays abnormally according to the second detection result of the second detection. That is, the second display interface is detected for a second time to obtain a second detection result, and it is determined whether the second display interface displays abnormally according to the second detection result.

[0144] Exemplarily, the first detection result may be 1 or 0, where 1 and 0 represent different results, for example, 1 represents normal display, and 0 represents abnormal display.

[0145] When the second detection result is that the display is normal, it is determined that the second display interface is displayed normally and there is no abnormality. The first display interface is detected by two different detection methods, the first detection method and the second detection method. When the detection results of both methods are normal, it means that the second display interface is displayed normally, which improves the accuracy of the detection and can more accurately determine whether there is a display abnormality in the second display interface.

[0146] If the second detection result is abnormal, it means that the second display interface may have a display abnormality, and the third detection result of the third detection method is combined to determine whether the second display interface displays abnormally. For example, it is determined whether the second display interface is displayed normally based on the second detection result obtained by the second detection method and the third detection result obtained by the third detection method.

[0147] Exemplarily, the second detection result may be 1 or 0, where 1 and 0 represent different results, for example, 1 represents normal display, and 0 represents abnormal display.

[0148] In the case where the first detection result is that the display is abnormal, the second display interface is also detected by the second detection method to obtain the second detection result. The second display interface is also detected by the third detection method to obtain the third detection result, and then the second detection result and the third detection result are combined to determine whether the second display interface has a display abnormality. In this way, it is possible to determine whether the second display interface has a display abnormality based on the first detection result and the detection results of the two detection methods. Through the detection results of multiple detection methods, the accuracy of the detection is improved.

[0149] Exemplarily, the model used in the second detection method includes Figure 8 and Fig. 9 The corresponding structure.

[0150] Exemplarily, the second test result and the third test result have respective weights, and the weights can be determined according to actual needs. For example, the weight of the second test result is greater than the weight of the third test result, the weight of the second test result is x=0.6, and the weight of the third test result is y=0.4. Figure 6 shown.

[0151] Exemplarily, the first detection result, the second detection result and the third detection result are numerical values, the target detection result is determined based on the second detection result and the third detection result and their corresponding weights, and the target detection result is also in numerical form. Whether the second display interface displays an abnormality is determined based on the target detection result and a preset threshold.

[0152] Exemplarily, the first detection result, the second detection result, the third detection result and the target detection result may all be values ​​between 0 and 1.

[0153] When the target detection result is greater than a preset threshold, it is determined that the second display interface is displayed abnormally. When the target detection result is less than the preset threshold, it is determined that the second display interface is displayed normally.

[0154] In one embodiment, the first detection method is performed by a first detection model, the second detection method is performed by a second detection model, and the third detection method is performed by a third detection model. The second detection model includes a target detection model. The structure of the target detection model may include a subsequent Figure 8 and Fig. 9 The structure of the corresponding embodiment: The three detection models can all be neural network models, or deep learning network models.

[0155] The first detection model, the second detection model, and the third detection model are all obtained through training with training samples, and the training samples may include positive samples and negative samples, where the positive samples are samples that appear normal, and the negative samples are samples that appear abnormal. The training methods may include supervised learning, unsupervised learning, and semi-supervised learning. When the above models are trained through supervised learning and semi-supervised learning, the training samples may also include a first label and a second label, where the first label is used to indicate that the positive sample appears normal, and the second label is used to indicate that the negative sample appears abnormal.

[0156] Exemplarily, negative samples may include: samples with text overlap, icon overlap, image missing, overexposure, patches, white screen, black screen, distorted screen and / or abnormal control position.

[0157] In one embodiment, reference Figure 7 , is another detection schematic diagram, detecting the second display interface by a second detection method to obtain a second detection result, which at least includes:

[0158] S3021, extracting a first feature map of the second display interface.

[0159] S3022, extracting a second feature map and a third feature map from the first feature map through different feature channels; wherein the second feature map and the third feature map have different feature levels.

[0160] S3023, extract features from the second feature map using convolution kernels of different scales to obtain a fourth feature map.

[0161] S3024, extract features from the third feature map using convolution kernels of different scales to obtain a fifth feature map.

[0162] S3025: Determine a second detection result according to the fourth feature map and the fifth feature map.

[0163] For S3021, in this embodiment, when detecting whether the second display interface displays abnormally, the second display interface can be first acquired, and after acquiring the second display interface, the features of the second display interface are extracted to obtain a first feature map. The method of extracting the first feature map is not limited, and it can be extracted by a corresponding feature extraction algorithm, such as a feature extraction algorithm based on convolution.

[0164] For S3022, after obtaining the first feature map, extract the second feature map and the third feature map from the first feature map through different feature channels, and the feature levels of the second feature map and the third feature map are different. Extract features from the first feature map through different feature channels to obtain the second feature map and the third feature map, respectively. The second feature map is a high-level feature, and the third feature map is a low-level feature; or, the third feature map is a high-level feature, and the third feature map is a high-level feature. High-level features include high-level semantic features, and low-level features include low-level detail features.

[0165] In this way, features at different feature levels can be extracted from the first feature map. While extracting high-level features, more low-level detail features can be retained, thereby improving the receptive field and feature extraction capabilities, thereby improving the feature representation capability, reducing gradient vanishing and information loss, and combining features at different feature levels to facilitate improving the accuracy and efficiency of target detection.

[0166] For S3023, after obtaining the second feature map, feature extraction is performed on the second feature map through convolution kernels of different scales to obtain a fourth feature map.

[0167] The second feature map can be subjected to multi-branch convolution, and the size of the convolution kernel of each branch is different, so the corresponding receptive field is different, and the features extracted by each convolution branch are also different. Then, the features obtained by each convolution branch are fused to obtain the fourth feature map. The fusion method can include dimensional changes, such as dimensionality increase and dimensionality reduction, and then normalization and other operations.

[0168] For S3024, after obtaining the third feature map, feature extraction is performed on the third feature map through convolution kernels of different scales to obtain a fifth feature map.

[0169] The third feature map can be subjected to multi-branch convolution, and the size of the convolution kernel of each branch is different, so the corresponding receptive field is different, and the features extracted by each convolution branch are also different. Then, the features obtained by each convolution branch are fused to obtain the fifth feature map. The fusion method can include dimensional changes, such as dimensionality increase and dimensionality reduction, and then normalization and other operations.

[0170] The fourth feature map is obtained by selectively applying convolution kernels of different scales to extract features from the second feature map, and the fifth feature map is obtained by applying convolution kernels of different scales to extract features from the third feature map. Due to the different sizes of the convolution kernels, the scales of the extracted features are different, so that multi-scale features are obtained at different levels, the feature representation capability is improved, and thus the accuracy of target detection is improved.

[0171] For S3025, after obtaining the fourth characteristic map and the fifth characteristic map, a second detection result is obtained according to the fourth characteristic map and the fifth characteristic map, so as to determine whether a display abnormality occurs on the second display interface.

[0172] The method in this embodiment can perform multiple different feature extractions on the feature map of the second display interface to obtain feature maps at different levels and scales, and then determine whether the second display interface displays an abnormality, thereby improving the detection accuracy and reducing the false detection rate.

[0173] In one embodiment, S303, detecting the second display interface by a second detection method to obtain a second detection result includes: detecting the second display interface by a trained target detection model to obtain the second detection result.

[0174] refer to Figure 8 , is a partial schematic diagram of a target detection model, the target detection model at least includes:

[0175] The backbone network L includes at least a first branch L1 and a second branch L2, wherein the first branch L1 is used to extract the second feature map from the first feature map, and the second branch L2 is used to extract the third feature map from the first feature map. The backbone network L can implement the function of S3032, for example, the first branch L1 can extract high-level features of the first feature map, and the second branch L2 can extract low-level features of the first feature map.

[0176] Exemplarily, the target detection model includes at least: a first backbone network, the first branch L1 in the first backbone network may include multiple parts, such as a convolution unit (Conv), a residual unit and a fusion unit (CBL), etc., and the second branch L2 may include a convolution unit (Conv).

[0177] Exemplary, reference Fig. 9 , is a structural diagram of another target detection model, the target detection model at least includes: a second backbone network, the first branch L1 in the second backbone network may include a convolution unit (Conv) and multiple fusion units (CBL), and the second branch L2 may include a convolution unit (Conv).

[0178] Exemplarily, the target detection model includes at least: a first backbone network and a second backbone network. The connection relationship between the first backbone network and the second backbone network can be determined according to actual needs, such as the output end of the first backbone network is directly or indirectly connected to the input end of the second backbone network.

[0179] Exemplarily, the target detection model may include at least one backbone network L.

[0180] Figure 8 and Fig. 9The CSP shown in the figure is the backbone network, which is used to represent the cross-stage partial connection structure (Cross-Stage-Partial, CSP).

[0181] The selective kernel convolution network K is connected to the output end of the backbone network L, and is used to perform multi-branch convolution on the third feature and the second feature respectively to obtain the fourth feature map and the fifth feature map. The sizes of the convolution kernels of different branches are different.

[0182] The selective kernel convolution network K can implement the processes of S3023 and S3024 to obtain the fourth feature map and the fifth feature map.

[0183] Exemplarily, the selective kernel convolutional network K can be a Selective Kernel Network, i.e., a SKNet convolutional network.

[0184] Exemplarily, the selective kernel convolution network K is connected to the output end of the first branch L1 and the output end of the second branch L2 respectively.

[0185] The detection network W is connected to the output end of the selective kernel convolution network, and is used to output a second detection result indicating whether the second display interface has a display abnormality according to the fourth feature map and the fifth feature map. The detection network W can implement the function of S3025 and detect the display abnormality of the second display interface. The structure of the detection network W is not limited, for example, it includes a convolution unit, a pooling unit and / or a fully connected unit.

[0186] In one embodiment, the target detection network may also include: Figure 5 and Figure 6 The normalization unit (Batch Normalization, BN), correction unit (Leaky relu), feature fusion unit (concat), etc. are shown.

[0187] Figure 8 and Fig. 9 The target detection model shown in can be applied to the YOLO-V5 model. In the YOLO-V5 model structure, a selective kernel convolution network K is added to the CSP structure to obtain an updated CSP structure. The updated CSP structure is Figure 8 and Fig. 9 The structure except the detection network W is shown in FIG.

[0188] The SKNet network based on the dynamic attention mechanism is introduced into the YOLO-V5 model, and the selective kernel convolution network based on the attention mechanism is added to the CSP structure in the Yolov5 model, thereby improving the accuracy of the target detection model in detecting anomalies.

[0189] In another embodiment, the first display interface is an interface in an application or a system interface.

[0190] The detection method also includes:

[0191] When the first display interface is an interface in an application, the stability of the application is detected by program instrumentation. The instrumentation method can detect whether the application crashes, thereby facilitating the acquisition of operation information about the crash of the application.

[0192] In another embodiment, the method further comprises:

[0193] The first display interface is detected to see if there is a display abnormality, and the detection process may be the same as the detection of the second display interface. Refer to S301 to S304.

[0194] In one embodiment, reference Fig.10 , is a schematic diagram of an abnormality detection device, the device comprising:

[0195] Determination module 1, used to determine a target control in the first display interface; wherein the probability that the user input operation acts on the target control is the highest;

[0196] Display module 2, used to display a second display interface associated with the target control;

[0197] A detection module 3, used to detect whether the second display interface has display abnormality through multiple detection methods;

[0198] The reporting module 4 is used to report abnormal information when the display abnormality occurs on the second display interface.

[0199] In one embodiment, the determination module 1 includes:

[0200] A first acquisition unit, configured to acquire a hierarchical relationship between various interfaces in an application to which the first display interface belongs;

[0201] A determination unit is used to determine a target control according to the feature information of the preset control and the hierarchical relationship through a target reinforcement learning model.

[0202] In one embodiment, the apparatus further comprises:

[0203] A sample acquisition module, used to acquire multiple training sample sets, wherein the training sample sets include: a first display interface sample, a reference control sample, a target control sample, display information of a second display interface sample associated with the target control sample, historical data of each control in each first display interface sample as the target control, and a hierarchical relationship between each interface of the first display interface sample in the application;

[0204] An updating module is used to update the parameters of the value function of the value network in the initial reinforcement learning model using the training sample set to obtain the target reinforcement learning model; wherein the value function is used to update the parameters of the decision function of the decision network in the initial reinforcement learning network model, and the decision network is used to determine the target control.

[0205] In one embodiment, the display information at least includes:

[0206] The display duration of the second display interface sample.

[0207] In one embodiment, the apparatus further comprises:

[0208] The extraction module is used to extract characteristic information of the preset control, and the characteristic information at least includes: position information, icon information, text information and / or unread message mark information.

[0209] In one embodiment, the first display interface is an interface in an application or a system interface;

[0210] The device also includes:

[0211] The stability testing module is used to detect the stability of the application program by program instrumentation when the first display interface is an interface in the application program.

[0212] In one embodiment, the detection module is further used to:

[0213] Detect whether the first display interface has a display abnormality.

[0214] In one embodiment, the display abnormality at least includes:

[0215] Overlapping text or images;

[0216] Image missing;

[0217] Overexposure;

[0218] Plaque;

[0219] White screen;

[0220] Black screen;

[0221] Flower screen;

[0222] and / or, the controls are located abnormally.

[0223] In one embodiment, usually, the anomaly detection scheme is a script detection method for a single scenario and is not universal. It is implemented through testing tools such as Monkey, Instrumentation, Espresso, and Appium. The test interface is random, that is, the controls associated with the tested interface are randomly determined. After the problem is detected, testers are required to perform manual retesting and inspection, and the entire process requires human intervention. After confirming the problem, testers are required to capture the abnormal information log from the system side and submit it to the R&D personnel for analysis. The entire process is relatively cumbersome.

[0224] This embodiment can be applied to various smart terminals, such as automated testing of Android devices. The target control in the first display interface is identified by target detection and set reinforcement learning. The position where the user may click is determined based on the characteristic information of the target control, such as coordinates. This position is where the target control is located, thereby realizing automatic testing that simulates human behavior.

[0225] refer to Fig.11 , is a schematic diagram of another test process. The process includes: determining a target control in a first display interface, detecting whether a second display interface associated with the target control displays abnormally, reporting abnormal information when the display abnormality is detected in the second display interface, and detecting the stability of the application where the first display interface is located.

[0226] Exemplarily, determining a target control in a first display interface includes AI automatic testing. Detecting whether a second display interface associated with the target control displays abnormally includes UI problem detection. Reporting abnormal information includes automatic problem handling.

[0227] Specifically, it can include three parts:

[0228] 1. AI automatic testing (core part):

[0229] Through the first detection model, such as the YOLO model, each control in the first display interface (such as a screenshot of the first display interface) is identified (target detection), such as the coordinates of each control on the Android phone page (identifying the control coordinates), and then the coordinates of each control are input into the reinforcement learning model, and the reinforcement learning model determines the target coordinates (decision coordinates), that is, determines the target control, and generates the next action (click operation). The reinforcement learning model is based on user data learning, simulates user behavior, and the reinforcement learning model is Actor-Critic.

[0230] In one embodiment, the first display interface includes controls such as "WeChat", "Address Book", "Discover", "Me", "Services", "Favorites", "Friends Circle", "Emojis", "Settings" and "Status", and these controls can be identified from the first display interface through the recognition model.

[0231] The reinforcement learning model can be an Actor-Critic network model, including a value network and a policy network. The value network is used to optimize the decision of the policy network. The policy network is updated according to the value output of the value network so that the policy network can make the next decision and determine the target control. The target control is the "WeChat" control, and the second display interface is the interface associated with the "WeChat" control.

[0232] The reinforcement learning network may also include an action sampler, which collects actions output by the policy network and may mark the target control clicked after the action is executed as 1, and mark other controls that are not clicked as 0.

[0233] The proposed system-level solution mainly uses a combination of reinforcement learning algorithm and target detection algorithm in the automated detection stage. First, the target detection algorithm is used to extract the application page information, such as the number of controls in the page, the page path, the control coordinates, etc., and then these page attribute information is input into the multi-layer perceptron network for representation learning. The learned information matrix is ​​then input into the reinforcement learning algorithm to determine the path of the next frame operation, realizing intelligent simulation of click behavior. This solves the problem mentioned in 1 that the existing test tools cannot solve the problem of user login and unstable test coverage.

[0234] 2. Anomaly Detection

[0235] The anomaly detection in this embodiment adopts the Yolov5 algorithm to first mark the abnormal data, then train the model, adjust the optimal hyperparameters, and test the effect of the model. In actual use, the model has a good experimental effect of more than 90%, and can effectively identify various abnormal problems.

[0236] It also includes: stability exceptions are reported to the customized APP by inserting stubs in the system exception method, and then the information is uploaded to the server through HTTP requests.

[0237] 3. Abnormal information processing

[0238] When a display abnormality is detected, the APP reports the abnormal information to the server, which automatically uploads and analyzes the problem.

[0239] The APP is connected to the system to capture abnormal displayed information and enhance logs.

[0240] refer to Fig.12, which is a schematic diagram of a test system, including components corresponding to stability anomaly detection, UI anomaly detection, log enhancement, problem reporting and problem analysis, to realize the functions of the above corresponding embodiments.

[0241] In summary, this patent proposes an AI automatic testing system that can be equipped with multiple components. It can automatically detect stability and UI problems, capture, enhance logs and analyze them. It has strong usability, a wide trial range, and is expandable. The entire process does not require human participation and can solve most of the current automated testing problems.

[0242] The AI ​​automatic testing system serves as a "carrier". On this basis, this embodiment proposes to build automatic detection of stability and UI problems, and at the same time connect to the UI log enhancement tool and the log automatic analysis tool to achieve full automation from discovery to processing of Android platform problems. The specific logic is mainly implemented through the server (equipped with algorithm models, control processes) and APP (system access, detection of stability anomalies, etc.).

[0243] The proposed system-level solution provides system-side abnormal monitoring capabilities in the abnormal information processing module. When the abnormal detection module identifies the problem page, it will immediately capture the relevant problem information under the current page and save it in the Android system log, solving the problem that the existing abnormal detection solution cannot accurately capture on-site information. After capturing the information, the proposed solution will also automatically cluster the abnormal information, extract the screenshots of the problem page, abnormal categories, abnormal information, etc., and display them on the system platform, solving the pain points of manual operation of the existing solution and greatly improving work efficiency.

[0244] The Android system-level universal testing solution based on multiple algorithms such as reinforcement learning and target detection is mainly used for intelligent detection of applications installed on Android devices. It provides a universal automated testing process and anomaly detection solution for various problems existing in applications, supports real-time capture of abnormal problem information, and finally clusters and displays the abnormal information. This system-level solution can be widely used in the application detection link of terminal manufacturers.

[0245] It fills the gap of the current lack of universal Android system-level testing methods. From the test process to anomaly detection, and then to automated information processing after anomaly detection, the entire process does not require manual intervention by testers, thereby improving production efficiency.

[0246] The industry's first universal AI+ fully automated abnormal test detection and analysis tool can cover UI display and stability issues in all scenarios. The testing labor cost is reduced by 15%, the problem analysis and processing efficiency is increased by 30%, and the revenue is significantly improved. AI empowers R&D and testing to achieve closed-loop processing of problem discovery, capture, circulation, and analysis to improve user experience. Pre-intercept internal special compatibility issues, increase test items, and detect three-party compatibility display problems that may be caused by special projects in advance, so as to detect and deal with them early.

[0247] It should be noted that the “first” and “second” in the embodiments of the present disclosure are only for the convenience of description and distinction and have no other specific meanings.

[0248] Fig.13 1 is a block diagram of a terminal device according to an exemplary embodiment. For example, the terminal device may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0249] Reference Fig.13 The terminal device may include one or more of the following components: a processing component 802 , a memory 804 , a power component 806 , a multimedia component 808 , an audio component 810 , an input / output (I / O) interface 812 , a sensor component 814 , and a communication component 816 .

[0250] The processing component 802 generally controls the overall operation of the terminal device, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 802 may include one or more processors 820 to execute instructions to complete all or part of the steps of the above method. In addition, the processing component 802 may include one or more modules to facilitate the interaction between the processing component 802 and other components. For example, the processing component 802 may include a multimedia module to facilitate the interaction between the multimedia component 808 and the processing component 802.

[0251] The memory 804 is configured to store various types of data to support operations on the terminal device. Examples of such data include instructions for any application or method operating on the terminal device, contact data, phone book data, messages, pictures, videos, etc. The memory 804 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0252] The power component 806 provides power to various components of the terminal device. The power component 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the terminal device.

[0253] The multimedia component 808 includes a screen that provides an output interface between the terminal device and the user, and the display component may include a screen, which may be located on the lens. In some embodiments, the multimedia component 808 includes a front camera and / or a rear camera. When the terminal device is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0254] The audio component 810 is configured to output and / or input audio signals. For example, the audio component 810 includes a microphone (MIC), and when the terminal device is in an operation mode, such as a call mode, a recording mode, and a speech recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 804 or sent via the communication component 816. In some embodiments, the audio component 810 also includes a speaker for outputting audio signals.

[0255] I / O interface 812 provides an interface between processing component 802 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, volume buttons, start buttons, and lock buttons.

[0256] The sensor assembly 814 includes one or more sensors for providing various aspects of status assessment for the terminal device. For example, the sensor assembly 814 can detect the open / closed state of the terminal device, the relative positioning of components, such as the display and keypad of the terminal device, and the sensor assembly 814 can also detect the position change of the terminal device or a component of the terminal device, the presence or absence of user contact with the terminal device, the orientation or acceleration / deceleration of the terminal device, and the temperature change of the terminal device. The sensor assembly 814 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 814 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 814 may also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0257] The communication component 816 is configured to facilitate wired or wireless communication between the terminal device and other devices. The terminal device can access a wireless network based on a communication standard, such as Wi-Fi, 4G or 5G, or a combination thereof. In an exemplary embodiment, the communication component 816 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 816 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0258] In an exemplary embodiment, the terminal device may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above method.

[0259] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0260] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. An anomaly detection method, It is characterized in that include: Determine a target control in the first display interface; wherein the probability that the user input operation acts on the target control is the highest; Displaying a second display interface associated with the target control; Detecting whether the second display interface has display abnormality by using multiple detection methods; When the display abnormality occurs on the second display interface, report abnormality information.

2. The method according to claim 1, It is characterized in that The determining a target control in the first display interface includes: Acquire the hierarchical relationship between various interfaces in the application to which the first display interface belongs; Through the target reinforcement learning model, the target control is determined according to the feature information of each preset control in the first display interface and the hierarchical relationship.

3. The method according to claim 2, It is characterized in that The method further comprises: Acquire multiple training sample sets, wherein the training sample sets include: a first display interface sample, a reference control sample, a target control sample, display information of a second display interface sample associated with the target control sample, historical data of each control in each of the first display interface samples as the target control, and a hierarchical relationship between each interface of the first display interface sample in the application; The training sample set is used to update the parameters of the value function of the value network in the initial reinforcement learning model to obtain the target reinforcement learning model; wherein the value function is used to update the parameters of the decision function of the decision network in the initial reinforcement learning network model, and the decision network is used to determine the target control.

4. The method according to claim 3, It is characterized in that The display information at least includes: The display duration of the second display interface sample.

5. The method according to claim 2, It is characterized in that The method further comprises: Extract characteristic information of the preset control, where the characteristic information at least includes: location information, icon information, text information and / or unread message mark information.

6. The method according to claim 1, It is characterized in that The first display interface is an interface in an application or a system interface; The method further comprises: In the case where the first display interface is an interface in an application, the stability of the application is detected by program instrumentation.

7. The method according to claim 1, It is characterized in that The method further comprises: Detect whether the first display interface has a display abnormality.

8. The method according to claim 1, It is characterized in that The display abnormality at least includes: Overlapping text or images; Image missing; Overexposure; Plaque; White screen; Black screen; Flower screen; and / or, the controls are located abnormally.

9. An abnormality detection device, It is characterized in that include: A determination module, configured to determine a target control in the first display interface; wherein the probability that the user input operation acts on the target control is the highest; A display module, used to display a second display interface associated with the target control; A detection module, used to detect whether the second display interface has display abnormality through multiple detection methods; A reporting module is used to report abnormal information when the display abnormality occurs on the second display interface.

10. A terminal device, It is characterized in that include: A processor and a memory for storing executable instructions capable of running on the processor, wherein: When the processor is used to run the executable instructions, the executable instructions execute the method described in any one of claims 1 to 8.

11. A non-transitory computer-readable storage medium, It is characterized in that The non-transitory computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, implement the method of any one of claims 1 to 8.

Images