Interface display anomaly detection method and device, electronic equipment and storage medium

Abstract

The invention relates to the technical field of software detection, and provides an interface display anomaly detection method and device, electronic equipment and a storage medium, and the method comprises the steps: responding to a detected trigger event, and carrying out the delayed interception of a plurality of frames of screen images; determining a target image in a stable state from the multiple frames of screen images according to a comparison result of the multiple frames of screen images; inputting the target image into a pre-trained defect detection model for processing, and determining a candidate abnormal region from the target image; and carrying out overlapping judgment on the candidate abnormal areas so as to generate interface display abnormal information according to overlapping areas among the candidate abnormal areas. By adopting the method, interface display anomaly defect detection can be realized, and the accuracy of interface display anomaly detection is improved.

Description

Technical Field

[0001] The present disclosure relates to the technical field of software detection, and in particular, to a method, apparatus, device, and storage medium for detecting abnormal interface display. Background Art

[0002] UI anomalies are interface display anomalies caused by application software anomalies, compatibility issues between software and hardware environments, etc. Interface display anomalies include icon-over-icon, icon-over-text, text-over-text, etc. Application software is an important part of electronic products. To ensure the adaptation of software upgrades, new software, etc. in new hardware environments, anomaly detection is required to ensure that electronic products meet the quality requirements in terms of software.

[0003] Currently, the detection of interface display anomaly defects is based on manually operating the application software and observing whether there are anomalies in the screen display. During manual detection, small defects cannot be detected, and the manual detection cost is high, and the detection accuracy cannot be guaranteed. Summary of the Invention

[0004] Based on this, it is necessary to provide a method, apparatus, electronic device, and storage medium for detecting abnormal interface display in view of the above technical problems.

[0005] In a first aspect, an embodiment of the present disclosure provides a method for detecting abnormal interface display, including:

[0006] In response to detecting a trigger event, delay and capture multiple frames of screen images;

[0007] Determine a target image in a stable state from the multiple frames of screen images based on the comparison result of the multiple frames of screen images;

[0008] Input the target image into a pre-trained defect detection model for processing, and determine candidate abnormal regions from the target image;

[0009] Perform an overlap determination on the candidate abnormal regions to generate interface display anomaly information based on the overlapping regions between the candidate abnormal regions.

[0010] In a second aspect, an embodiment of the present disclosure provides an apparatus for detecting abnormal interface display, including:

[0011] An acquisition module, configured to delay and capture multiple frames of screen images in response to detecting a trigger event;

[0012] A determination module, configured to determine a target image in a stable state from the multiple frames of screen images based on the comparison result of the multiple frames of screen images;

[0013] A detection module, configured to input the target image into a pre-trained defect detection model for processing, and determine candidate abnormal regions from the target image;

[0014] A generation module, configured to perform an overlap determination on the candidate abnormal regions, so as to generate interface display abnormal information according to the overlapping regions between the candidate abnormal regions.

[0015] In a third aspect, an embodiment of the present disclosure provides an electronic device, including: a processor; a memory for storing executable instructions executable by the processor; the processor is configured to read the executable instructions from the memory and execute the instructions to implement the interface display abnormal detection method described in the first aspect above.

[0016] In a fourth aspect, an embodiment of the present disclosure provides a computer-readable storage medium, where the storage medium stores a computer program, and when the computer program is executed by a processor, the interface display abnormal detection method described in the first aspect above is implemented.

[0017] The technical solution provided by the embodiment of the present disclosure has the following advantages compared with the prior art: in response to detecting a trigger event, multiple frames of screen images are intercepted with a delay, and a target image in a stable state is determined from the multiple frames of screen images through the comparison results of the multiple frames of screen images. Furthermore, the target image is input into a pre-trained defect detection model for processing, candidate abnormal regions are determined from the target image, and an overlap determination is performed on the candidate abnormal regions, so as to generate interface display abnormal information according to the overlapping regions between the candidate abnormal regions. Thus, by combining a multi-frame comparison algorithm with a model detection algorithm, the interference of display abnormalities existing in transition frames on the defect detection result is reduced, the accuracy of interface display abnormal detection is improved, and at the same time, the detection of various interface display abnormal forms in multiple scenarios can be compatible, ensuring defect detection in multiple scenarios, reducing the cost of manual detection, and further improving the detection accuracy of interface display abnormalities for the post-processing related logical judgment of the output results of the defect detection model. Description of the Drawings

[0018] The drawings here are incorporated into the specification and constitute a part of this specification, showing embodiments consistent with the present disclosure, and are used together with the specification to explain the principles of the present disclosure.

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, for those of ordinary skill in the art, other drawings can also be obtained based on these drawings without creative efforts.

[0020] Figure 1 It is a flowchart of an interface display abnormal detection method provided by an embodiment of the present disclosure;

[0021] Figure 2 Schematic flow chart of another interface display anomaly detection method provided by an embodiment of the present disclosure;

[0022] Figure 3 Schematic diagram of a defect detection model provided by an embodiment of the present disclosure;

[0023] Figure 4 Schematic flow chart of a post - processing of defect detection provided by an embodiment of the present disclosure;

[0024] Figure 5 Schematic structural diagram of an interface display anomaly detection device provided by an embodiment of the present disclosure;

[0025] Figure 6 Schematic structural diagram of an electronic device provided by an embodiment of the present disclosure. Detailed implementation manners

[0026] In order to make the objectives, technical solutions and advantages of the present disclosure more clear and understandable, the present disclosure will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present disclosure, and are not used to limit the present disclosure.

[0027] In one embodiment, as Figure 1 shown, an interface display anomaly detection method is provided, and the method includes the following steps:

[0028] Step 102, in response to detecting a trigger event, delay - capture multiple frame screen images.

[0029] In this embodiment, the terminal device detects the trigger event, and the trigger event includes but is not limited to click events and swipe events. Optionally, after the application software is installed, various operations are performed according to the requirements of the test cases, and each operation is captured as a trigger event. When the trigger event is detected, the interface display anomaly detection starts, and multiple frame screen images are delay - captured starting from the trigger time of the trigger event.

[0030] In an embodiment of the present disclosure, delay - capturing multiple frame screen images includes: starting from the trigger time of the trigger event, delay - capturing a first screen image for a first duration and delay - capturing a second screen image for a second duration, where the first duration is less than the second duration.

[0031] As an example, the first duration is 100 ms and the second duration is 350 ms. In this example, taking the trigger time of the trigger event as the starting moment, the screen image 1 and the screen image 2 are captured after delaying for 100 ms and 350 ms respectively. Since the interface rendering takes time and there are a large number of over-rendered frames during this period, multiple screen images are captured after a delay to reduce the interference of the transition frames. Among them, for the interface without animation, the screen image 1 is captured after delaying for 100 ms, and the interface is in a stable state after rendering. For the interface with transition animation, the screen image 2 is captured after delaying for 350 ms, and the interface is in a stable state after rendering.

[0032] In an embodiment of the present disclosure, capturing multiple screen images after a delay includes: starting from the trigger time of the trigger event, capturing the third screen image in sequence at a specified time interval.

[0033] As an example, the specified time interval is 100 ms. In this example, taking the trigger time of the trigger event as the starting moment, the screen image is captured once every 100 ms time interval.

[0034] In an embodiment of the present disclosure, capturing multiple screen images after a delay includes: starting from the trigger time of the trigger event, capturing the first screen image after delaying for the first duration, and capturing the second screen image after delaying for the second duration. Further, after capturing the second screen image, the third screen image is captured at a specified time interval.

[0035] As an example, the first duration is 100 ms, the second duration is 350 ms, and the specified time interval is 100 ms. In this example, taking the trigger time of the trigger event as the starting moment, the screen image 1 and the screen image 2 are captured after delaying for 100 ms and 350 ms respectively, and starting from the moment when the screen image 2 is captured, the screen image is captured once every 100 ms time interval.

[0036] Step 104, determining a target image in a stable state from multiple screen images based on the comparison results of the multiple screen images.

[0037] In this embodiment, after capturing multiple screen images after a delay, adjacent frame screen images are compared based on similarity to determine a target image in a stable state from the multiple screen images according to the comparison results. Among them, the stable state is used to indicate that the interface rendering is completed.

[0038] The comparison process of multiple screen images will be described below.

[0039] In one embodiment of the present disclosure, determining a target image in a stable state from multiple frames of screen images based on the comparison results of the multiple frames of screen images includes: subtracting corresponding pixels of two adjacent frames of screen images in the multiple frames of screen images to generate a difference image; performing binarization processing on the difference image to determine a difference region between the two adjacent frames of screen images; calculating the similarity between the two adjacent frames of screen images according to the area of the difference region and the area of the overall region of the difference image; if the similarity is greater than a similarity threshold, determining the latter frame of the two adjacent frames of screen images as the target image.

[0040] Among them, two adjacent frames of screen images are sequentially determined from the multiple frames of screen images based on the time sequence. For two adjacent frames of screen images, the corresponding pixel points of the two frames of screen images are subtracted and binarized to determine the difference region and the similar region between the two frames of screen images. The overall region of the difference image is composed of the difference region and the similar region. Optionally, the similarity is calculated by the area ratio of the similar region to the overall region. The larger the area ratio, the greater the similarity.

[0041] In this embodiment, after calculating the similarity between two adjacent frames of screen images, if the similarity is less than or equal to the similarity threshold, then a latter frame of screen image and another frame of screen image adjacent to the latter frame of screen image are determined from the multiple frames of screen images based on the time sequence as new adjacent frames of screen images; the steps of calculating the similarity and comparing with the threshold are performed based on the new adjacent frames of screen images until the target image is determined from the multiple frames of screen images.

[0042] As an example, the multiple frames of screen images include a screen image 1 intercepted with a first delay duration and a screen image 2 intercepted with a second delay duration. The corresponding pixel points of the screen image 1 and the screen image 2 are subtracted to obtain a difference image, and the difference image is binarized to obtain a difference region and a similar region. Furthermore, the similarity S is calculated by the area ratio R of the similar region to the overall region, where S = R * R * R. If the similarity is greater than the similarity threshold, it is determined that the screen image 2 is in a stable state, and the screen image 2 is used as the input of the defect detection model for detection.

[0043] In this example, if the similarity is less than or equal to the similarity threshold, it indicates that the screen image 2 is not in a stable state. Subsequently, screen images 3, 4, …, N are captured at specified time intervals, and the subsequent frame screen image i is compared with the previous frame screen image i - 1, where i is greater than or equal to 3. If the similarity is greater than the similarity threshold, it is determined that the screen image i is in a stable state, and the screen image i is used as the input of the defect detection model for detection. If the similarity is less than or equal to the similarity threshold, then i is incremented by 1 and the above steps of comparing adjacent two-frame screen images and comparing similarities are looped until the target image is determined when the similarity is greater than the similarity threshold. Optionally, the above steps are looped. If the target image has not been determined when the next trigger event occurs, an interface display abnormality is returned. Refer to Figure 2 , and the interface display abnormality detection is performed whenever a trigger event is detected, so as to detect the screen display information of each interface of the software.

[0044] Step 106: Input the target image into the pre-trained defect detection model for processing, and determine the candidate abnormal regions from the target image.

[0045] In this embodiment, the input of the defect detection model is the target image, and the output includes the candidate regions in the target image, as well as the confidence levels and region labels of each candidate region, where the region labels include normal labels and abnormal labels. Optionally, input the target image into the pre-trained defect detection model for processing to obtain the candidate regions in the target image, the confidence levels of the candidate regions, and the region labels. Furthermore, for any candidate region, if the confidence level is greater than the confidence level threshold and the region label is an abnormal label, it is determined that the candidate region is a candidate abnormal region; otherwise, it is determined that the candidate region is a normal region.

[0046] Among them, the defect detection model training process includes: obtaining a training data set, the training data set including normal images and defect images displayed on the interface, and annotating the images in the training data set. For example, taking the UI abnormality detection of a mobile terminal as an example, the training data set includes images with normal interface displays and abnormal displays on the mobile terminal interface. Furthermore, import the annotated training data set for model training, and set training parameters to ensure training convergence. The network structure diagram of the defect detection model refers to Figure 3 .

[0047] Step 108: Perform an overlap determination on the candidate abnormal regions to generate interface display abnormality information based on the overlapping regions between the candidate abnormal regions.

[0048] In this embodiment, all candidate abnormal regions are traversed to determine whether there is an overlapping region between any two candidate abnormal regions. If there is an overlapping region between two candidate abnormal regions, abnormal information for interface display is generated based on the overlapping region. If there is no overlapping region between the current candidate abnormal region and other candidate abnormal regions, the current candidate abnormal region is determined to be a normal region. Optionally, if there is an overlapping region between two candidate abnormal regions, abnormal information for interface display is generated based on the outer contour rectangle of the overlapping region.

[0049] As an example, referring to Figure 4 , the output of the defect detection model is an image region queue L1, as well as the confidence levels and region labels of each region in L1. First, screening is performed through a confidence level threshold. If the confidence level of a region in L1 is higher than the confidence level threshold, it is added to the image region queue L2, and the region with a confidence level not higher than the confidence level threshold is determined to be a UI normal region. For the image region queue L2, screening is performed through the region label. If the region label of a region in L2 is an abnormal label, it is added to the image region queue L3. If the region label is a normal label, it is determined to be a UI normal region. For the image region queue L3, it is determined whether there is an overlap in the ranges of any two regions in L3. If there is no overlap, the region is determined to be a UI normal region. If there is an overlap, it is added to the image region queue L4, and the outer rectangle of the overlapping region in L4 is obtained to generate a UI abnormal region.

[0050] According to the technical solution of the embodiment of the present disclosure, in response to detecting a trigger event, multiple frames of screen images are intercepted with a delay. The target image in a stable state is determined from the multiple frames of screen images based on the comparison results of the multiple frames of screen images. Furthermore, the target image is input into a pre-trained defect detection model for processing, candidate abnormal regions are determined from the target image, and overlap determination is performed on the candidate abnormal regions to generate abnormal information for interface display based on the overlapping regions between the candidate abnormal regions. Thus, by combining the multi-frame comparison algorithm with the model detection algorithm, the interference of display abnormalities existing in the transition frames on the defect detection result is reduced, the accuracy of interface display abnormality detection is improved, and at the same time, the detection of various interface display abnormality forms in multiple scenarios can be compatible, ensuring defect detection in multiple scenarios and reducing the cost of manual detection. In addition, the relevant logical judgment for post-processing the output result of the defect detection model further improves the detection accuracy of interface display abnormalities.

[0051] It should be understood that although Figure 1 the steps in the flowchart of Figure 1At least a part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed and completed at the same moment, but can be executed at different moments. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed alternately or in turns with at least a part of other steps or sub-steps or stages of other steps.

[0052] In one embodiment, as Figure 5 shown, an interface display anomaly detection device is provided, including: an acquisition module 52, a determination module 54, a detection module 56, and a generation module 58.

[0053] The acquisition module 52 is configured to delay and capture multiple frame screen images in response to detecting a trigger event;

[0054] The determination module 54 is configured to determine a target image in a stable state from the multiple frame screen images based on the comparison result of the multiple frame screen images;

[0055] The detection module 56 is configured to input the target image into a pre-trained defect detection model for processing, and determine candidate anomaly regions from the target image;

[0056] The generation module 58 is configured to perform an overlap determination on the candidate anomaly regions, and generate interface display anomaly information based on the overlapping regions between the candidate anomaly regions.

[0057] In one embodiment, the acquisition module 52 is specifically configured to: starting from the trigger time of the trigger event, delay for a first duration to capture a first screen image, and delay for a second duration to capture a second screen image; wherein, the first duration is less than the second duration; and / or, capture third screen images in sequence at a specified time interval.

[0058] In one embodiment, the first duration is 100 ms and the second duration is 350 ms.

[0059] In one embodiment, the determination module 54 is specifically configured to: perform pixel subtraction on two adjacent frame screen images among the multiple frame screen images to generate a difference image; perform binarization processing on the difference image to determine the difference region between the two adjacent frame screen images; calculate the similarity between the two adjacent frame screen images according to the area of the difference region and the area of the overall region of the difference image; if the similarity is greater than the similarity threshold, determine the latter frame screen image among the two adjacent frame screen images as the target image.

[0060] In one embodiment, the determination module 54 is further configured to: if the similarity is less than or equal to the similarity threshold, determine a subsequent frame of the screen image and another frame of the screen image adjacent to the subsequent frame of the screen image from multiple frames of screen images in chronological order, so as to serve as two new adjacent frames of screen images; perform the steps of calculating the similarity and comparing with the threshold based on the two new adjacent frames of screen images until the target image is determined from the multiple frames of screen images.

[0061] In one embodiment, the detection module 56 is specifically configured to: input the target image into a pre-trained defect detection model for processing to obtain candidate regions in the target image, the confidence of the candidate regions, and region labels; for any candidate region, if the confidence is greater than the confidence threshold and the region label is an abnormal label, determine the candidate region as a candidate abnormal region.

[0062] In one embodiment, the generation module 58 is specifically configured to: traverse multiple candidate abnormal regions and determine whether there is an overlapping region between any two candidate abnormal regions; if there is an overlapping region between two candidate abnormal regions, generate interface display abnormal information according to the outer contour rectangle of the overlapping region.

[0063] For the specific definition of the interface display abnormality detection device, reference may be made to the definition of the interface display abnormality detection method in the foregoing text. It has functional modules corresponding to the execution of the method and beneficial effects, which will not be elaborated here. Each module in the above interface display abnormality detection device can be implemented in whole or in part by software, hardware, and their combination. The above modules can be embedded in the processor of the electronic device in hardware form or independent of it, or stored in the memory of the electronic device in software form, so that the processor can call and execute the operations corresponding to the above respective modules.

[0064] In one embodiment, an electronic device is provided. The electronic device may be a terminal or a smart device, and its internal structure diagram may be as Figure 6As shown in the figure. The electronic device includes a processor, a memory, a communication interface, a display screen, and an input device connected via a system bus. Among them, the processor of the electronic device is used to provide computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The communication interface of the electronic device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be achieved through WIFI, a carrier network, near-field communication (NFC), or other technologies. When the computer program is executed by the processor, it implements an interface display anomaly detection method. The display screen of the electronic device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the electronic device can be a touch layer covering the display screen, or a button, a trackball, or a touchpad provided on the housing of the electronic device, or an external keyboard, touchpad, or mouse, etc.

[0065] Those skilled in the art can understand that Figure 6 the structure shown in the figure is only a block diagram of some structures related to the solution of the present disclosure, and does not constitute a limitation on the electronic device to which the solution of the present disclosure is applied. The specific electronic device may include more or fewer components than those shown in the figure, or combine some components, or have a different component arrangement.

[0066] In one embodiment, the interface display anomaly detection device provided by the present disclosure can be implemented in the form of a computer program, and the computer program can run on an electronic device as shown in Figure 6 the figure. Each program module constituting the interface display anomaly detection device can be stored in the memory of the electronic device. For example, Figure 5 the computer program composed of each program module shown in the figure enables the processor to execute the steps in the interface display anomaly detection method of each embodiment of the present disclosure described in this specification.

[0067] In one embodiment, an electronic device is provided, including a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, the following steps are implemented: in response to detecting a trigger event, delaying and capturing multiple frames of screen images, determining a target image in a stable state from the multiple frames of screen images through the comparison result of the multiple frames of screen images, and then inputting the target image into a pre-trained defect detection model for processing, determining candidate anomaly regions from the target image, and performing an overlap determination on the candidate anomaly regions to generate interface display anomaly information based on the overlapping regions between the candidate anomaly regions.

[0068] An electronic device according to an embodiment of the present disclosure implements the above steps when a processor executes a computer program. By combining a multi-frame comparison algorithm with a model detection algorithm, the interference of display anomalies in transitional frames to the defect detection result is reduced, the accuracy of interface display anomaly detection is improved, and at the same time, it can be compatible with the detection of various interface display anomaly forms in multiple scenarios, ensuring defect detection in multiple scenarios and reducing the cost of manual detection. In addition, the relevant logical judgment for post-processing the output result of the defect detection model further improves the detection accuracy of interface display anomalies.

[0069] In one embodiment, the following steps are implemented when a processor executes a computer program: starting from the trigger time of the trigger event, intercepting a first screen image after a first time delay, and intercepting a second screen image after a second time delay; wherein, the first time delay is less than the second time delay; and / or, intercepting third screen images in sequence at a specified time interval.

[0070] In one embodiment, the following steps are implemented when a processor executes a computer program: performing pixel subtraction on two adjacent screen images among multiple frame screen images to generate a difference image; performing binarization processing on the difference image to determine the difference region between the two adjacent screen images; calculating the similarity between the two adjacent screen images according to the area of the difference region and the area of the overall region of the difference image; if the similarity is greater than the similarity threshold, determining the latter screen image among the two adjacent screen images as the target image.

[0071] In one embodiment, the following steps are implemented when a processor executes a computer program: if the similarity is less than or equal to the similarity threshold, determining the latter screen image and another screen image adjacent to the latter screen image from the multiple frame screen images based on the time sequence as the new two adjacent screen images; performing the steps of calculating the similarity and comparing with the threshold based on the new two adjacent screen images until the target image is determined from the multiple frame screen images.

[0072] In one embodiment, the following steps are implemented when a processor executes a computer program: inputting the target image into a pre-trained defect detection model for processing to obtain candidate regions, confidence levels of the candidate regions, and region labels in the target image; for any candidate region, if the confidence level is greater than the confidence threshold and the region label is an abnormal label, determining the candidate region as a candidate abnormal region.

[0073] In one embodiment, the following steps are implemented when a processor executes a computer program: traversing multiple candidate abnormal regions to determine whether there is an overlapping region between any two candidate abnormal regions; if there is an overlapping region between two candidate abnormal regions, generating interface display anomaly information according to the outer contour rectangle of the overlapping region.

[0074] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented: in response to detecting a trigger event, delay and capture multiple frame screen images, determine a target image in a stable state from the multiple frame screen images based on the comparison result of the multiple frame screen images, and then input the target image into a pre-trained defect detection model for processing, determine candidate abnormal regions from the target image, perform an overlap determination on the candidate abnormal regions, and generate interface display abnormal information according to the overlapping regions between the candidate abnormal regions.

[0075] In one embodiment, when the computer program is executed by a processor, the following steps can also be implemented: starting from the trigger time of the trigger event, delay for a first duration to capture a first screen image, and delay for a second duration to capture a second screen image; wherein, the first duration is less than the second duration; and / or, capture third screen images in sequence at a specified time interval.

[0076] In one embodiment, when the computer program is executed by a processor, the following steps can also be implemented: perform pixel subtraction on two adjacent frame screen images among the multiple frame screen images to generate a difference image; perform binarization processing on the difference image to determine the difference region between the two adjacent frame screen images; calculate the similarity between the two adjacent frame screen images according to the area of the difference region and the area of the overall region of the difference image; if the similarity is greater than the similarity threshold, determine the latter frame screen image among the two adjacent frame screen images as the target image.

[0077] In one embodiment, when the computer program is executed by a processor, the following steps can also be implemented: if the similarity is less than or equal to the similarity threshold, determine the latter frame screen image and another frame screen image adjacent to the latter frame screen image from the multiple frame screen images based on the time sequence as new adjacent two frame screen images; perform the steps of calculating the similarity and comparing with the threshold based on the new adjacent two frame screen images until the target image is determined from the multiple frame screen images.

[0078] In one embodiment, when the computer program is executed by a processor, the following steps can also be implemented: input the target image into a pre-trained defect detection model for processing to obtain candidate regions, confidence levels of the candidate regions, and region labels in the target image; for any candidate region, if the confidence level is greater than the confidence level threshold and the region label is an abnormal label, determine the candidate region as a candidate abnormal region.

[0079] In one embodiment, when the computer program is executed by a processor, the following steps can also be implemented: traverse multiple candidate abnormal regions, and determine whether there is an overlapping region between any two candidate abnormal regions; if there is an overlapping region between two candidate abnormal regions, generate interface display abnormal information according to the outer contour rectangle of the overlapping region.

[0080] According to the computer-readable storage medium of the embodiments of the present disclosure, by combining a multi-frame comparison algorithm with a model detection algorithm, the interference of display anomalies existing in transitional frames to the defect detection result is reduced, the accuracy of interface display anomaly detection is improved, and at the same time, the detection of various interface display anomaly forms in multiple scenarios can be compatible, ensuring defect detection in multiple scenarios and reducing the cost of manual detection. In addition, for the post-processing related logical judgment of the output result of the defect detection model, the detection accuracy of interface display anomalies is further improved.

[0081] Those of ordinary skill in the art can understand that all or part of the processes of implementing the methods in the above embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods. Among them, any reference to a memory, database, or other medium used in the embodiments provided by the present disclosure can include at least one of non-volatile and volatile memories. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, or optical memory, etc. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static random access memory (SRAM) and dynamic random access memory (DRAM), etc.

[0082] The technical features of the above embodiments can be combined arbitrarily. For the sake of concise description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as the scope described in this specification.

[0083] The above-described embodiments merely represent several implementation manners of the present disclosure. Their descriptions are relatively specific and detailed, but they should not be construed as limiting the scope of the invention patent. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present disclosure, several modifications and improvements can still be made, and these all belong to the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure patent should be subject to the appended claims.

Claims

1. A method for detecting abnormal interface display, characterized in that, Including: In response to detecting a trigger event, delay and capture multiple frame screen images; Determine a target image in a stable state from the multiple frame screen images based on the comparison result of the multiple frame screen images; Input the target image into a pre-trained defect detection model for processing, and determine candidate abnormal regions from the target image; Perform an overlap determination on the candidate abnormal regions to generate interface display abnormal information based on the overlapping regions between the candidate abnormal regions.

2. The method according to claim 1, wherein The delaying and capturing multiple frame screen images includes: Starting from the trigger time of the trigger event, delay for a first duration to capture a first screen image, and delay for a second duration to capture a second screen image; wherein, the first duration is less than the second duration; And / or, capture third screen images in sequence at a specified time interval.

3. The method according to claim 2, wherein The first duration is 100 ms, and the second duration is 350 ms.

4. The method according to claim 1, wherein The determining a target image in a stable state from the multiple frame screen images based on the comparison result of the multiple frame screen images includes: Perform pixel subtraction on two adjacent frame screen images in the multiple frame screen images to generate a difference image; Perform binarization processing based on the difference image to determine the difference region between the two adjacent frame screen images; Calculate the similarity between the two adjacent frame screen images according to the area of the difference region and the area of the overall region of the difference image; If the similarity is greater than the similarity threshold, determine the latter frame screen image in the two adjacent frame screen images as the target image.

5. The method according to claim 4, characterized in that After calculating the similarity between the two adjacent frame screen images, the method further includes: If the similarity is less than or equal to the similarity threshold, determine the latter frame screen image and another frame screen image adjacent to the latter frame screen image from the multiple frame screen images based on the time sequence as new adjacent two frame screen images; Perform the steps of calculating similarity and threshold comparison based on the new adjacent two frame screen images until the target image is determined from the multiple frame screen images.

6. The method according to claim 1, wherein The inputting the target image into a pre-trained defect detection model for processing and determining candidate abnormal regions from the target image includes: Input the target image into a pre-trained defect detection model for processing to obtain candidate regions, confidence levels of the candidate regions, and region labels in the target image; For any candidate region, if the confidence level is greater than the confidence level threshold and the region label is an abnormal label, determine the candidate region as the candidate abnormal region.

7. The method according to claim 6, wherein The performing an overlap determination on the candidate abnormal regions to generate interface display abnormal information based on the overlapping regions between the candidate abnormal regions includes: Traverse multiple candidate abnormal regions to determine whether there is an overlapping region between any two candidate abnormal regions; If there is an overlapping region between the two candidate abnormal regions, generate the interface display abnormal information according to the outer contour rectangle of the overlapping region.

8. An interface display anomaly detection device, characterized in that, Including: An acquisition module, configured to delay and capture multiple frame screen images in response to detecting a trigger event; A determination module, configured to determine a target image in a stable state from the multi-frame screen images according to the comparison result of the multi-frame screen images; A detection module, configured to input the target image into a pre-trained defect detection model for processing, and determine candidate abnormal regions from the target image; A generation module, configured to perform an overlap determination on the candidate abnormal regions, so as to generate interface display abnormal information according to the overlapping regions between the candidate abnormal regions.

9. An electronic device, comprising a memory and a processor, the memory storing a computer program, characterized in that, When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images