A screen fault detection method, apparatus, electronic device, and readable storage medium.

Abstract

This application provides a screen fault detection method, apparatus, electronic device, and readable storage medium. The method includes: recording a reference video containing the screen under test while a reference image is played on a screen under test that is not faulty; recording a test video containing the screen under test while the reference image is played on the screen under test during a fault detection phase; detecting a video frame recorded while the reference image is played from the reference video as a reference image; detecting a video frame recorded while the reference image is played from the test video as a test image; calculating perspective transformation parameters of the test image relative to the reference image; performing perspective transformation on each frame of the test image using the perspective transformation parameters; comparing the difference between the screen under test area in the reference image and the screen under test area in the perspective-transformed test image; and determining the screen fault detection result based on the difference. This method improves the efficiency of screen fault detection.

Description

Technical Field

[0001] This application relates to the field of fault detection technology, and in particular to a screen fault detection method, apparatus, electronic device, and readable storage medium. Background Technology

[0002] Currently, video information has permeated all aspects of our lives, including work, study, training, medical care, meetings, and entertainment, where it is used as an information carrier. Video playback devices, as a common type of device for displaying video information, are widely used. However, the screens on these devices inevitably suffer damage during use, such as black and white blocks, shadows, bright lines, black lines, white spots, black spots, and screen distortion. Therefore, quickly detecting damaged screens and promptly maintaining or replacing them represents a significant market demand.

[0003] Traditional methods for detecting screen damage (faults), such as using a colorimeter to inspect the screen, require not only purchasing an expensive colorimeter, but also training technicians or hiring external professionals to use the instrument correctly. This results in high testing costs, and the screens can only be tested one by one, making batch testing impossible. Furthermore, the testing time is long and time-consuming. Summary of the Invention

[0004] In view of this, the purpose of this application is to provide a screen fault detection method, device, electronic device and readable storage medium that eliminates the need for professional instruments to detect the screen, reduces manual intervention and improves the efficiency of screen fault detection.

[0005] In a first aspect, embodiments of this application provide a screen fault detection method, including:

[0006] While the screen under test is playing each reference image sequentially according to a preset playback duration and preset playback order, a reference video containing the screen under test is recorded.

[0007] During the fault detection phase, as the screen under test plays each of the reference images sequentially according to the preset playback duration and the preset playback order, a test video containing the screen under test is recorded.

[0008] For each of the reference images, a video frame recorded while playing the reference image is detected from the continuous video frames contained in the reference video to obtain the reference image corresponding to the reference image; and a video frame recorded while playing the reference image is detected from the continuous video frames contained in the test video to obtain the test image corresponding to the reference image.

[0009] Calculate the perspective transformation parameters of the test image relative to the reference image, and use the perspective transformation parameters to perform perspective transformation on each frame of the test image respectively;

[0010] The differences between the screen region to be detected in the reference image corresponding to each of the reference images and the screen region to be detected in the perspective-transformed test image are compared to determine the screen fault detection result based on the differences corresponding to each of the reference images.

[0011] In conjunction with the first aspect, embodiments of this application provide a first possible implementation of the first aspect, wherein, for each reference image, detecting a video frame recorded while playing the reference image from a series of consecutive video frames included in the reference video to obtain a reference image corresponding to the reference image; and detecting a video frame recorded while playing the reference image from a series of consecutive video frames included in the test video to obtain a test image corresponding to the reference image, includes:

[0012] For each video in the reference video and the test video, the first abrupt change frame when playing the first reference image is detected from the consecutive video frames contained in the video, and the video frames before the first abrupt change frame in the video are cropped to obtain the reference video and test video with the first frame aligned.

[0013] Calculate the product of the preset playback duration and the video frame rate, and use the product result as the target interval frame number; calculate the ratio of the target interval frame number to 2, and use the ratio result as the target frame number.

[0014] From the consecutive video frames contained in the reference video after the first frame is aligned, a video frame with the target number of frames is selected as the first frame reference image. Then, a video frame is selected every target interval frame number as the reference image for each frame to obtain the reference image corresponding to each reference image.

[0015] From the consecutive video frames contained in the test video after the first frame is aligned, a video frame with the target number of frames is selected as the first test image. Then, a video frame is selected every target interval frame number as the test image for each frame, so as to obtain the test image corresponding to each reference image.

[0016] In conjunction with the first possible implementation of the first aspect, embodiments of this application provide a second possible implementation of the first aspect, wherein detecting the first abrupt change frame when playing the first reference image from consecutive video frames contained in each of the reference video and the test video includes:

[0017] The minimum value between the total number of video frames in the reference video and the total number of video frames in the test video is determined, and this minimum value is taken as the target total number of frames.

[0018] From both the reference video and the test video, extract video frames representing the target total number of frames to obtain a reference video and a test video with the total number of frames aligned.

[0019] For each video in the reference video and test video with total frame number alignment, calculate the inter-frame similarity between any two adjacent video frames in the video to obtain the inter-frame similarity sequence, and use the first-order difference algorithm to calculate the first-order jump sequence of the inter-frame similarity sequence.

[0020] A preset proportion of the maximum first-order jump value in the first-order jump sequence is selected as the first threshold. The first first-order jump value in the first-order jump sequence that exceeds the first threshold is searched. The second video frame of the two video frames corresponding to the first-order jump value is taken as the first mutation frame when playing the first reference image.

[0021] In conjunction with the second possible implementation of the first aspect, this application provides a third possible implementation of the first aspect, wherein calculating the inter-frame similarity between any two adjacent video frames in the video includes:

[0022] The inter-frame similarity between any two adjacent frames in the video can be calculated using the following formula:

[0023]

[0024] in, This is the brightness comparison function; This is a contrast comparison function; For structural comparison functions; , , These are preset coefficients; and Represents two adjacent video frames; For video frames and Inter-frame similarity;

[0025]

[0026]

[0027] in, It is a video frame The average value of the pixels; It is a video frame The average pixel value; L is the preset value; K1 is an empirical coefficient;

[0028]

[0029]

[0030] in, It is a video frame The standard deviation of pixel values; It is a video frame The standard deviation of the pixel values; K2 is an empirical coefficient;

[0031]

[0032]

[0033] in, It is a video frame and The covariance of pixel values.

[0034] In conjunction with the first aspect, this application provides a fourth possible implementation of the first aspect, wherein calculating the perspective transformation parameters of the test image relative to the reference image includes:

[0035] Select one frame of the reference image, and perform grayscale processing on the reference image and all test images to obtain a first reference grayscale image and multiple frames of the first test grayscale image;

[0036] Feature extraction is performed on the first reference grayscale image to obtain multiple first feature points on the first reference grayscale image; and feature extraction is performed on each frame of the first test grayscale image to obtain multiple second feature points on each frame of the first test grayscale image.

[0037] For each first feature point, the Hamming distance-based brute-force matching algorithm based on K-nearest neighbors is used to calculate the Hamming distance between the first feature point and each second feature point. The two smallest Hamming distances are selected from all the Hamming distances corresponding to the first feature point and are respectively used as the first Hamming distance and the second Hamming distance; the first Hamming distance is less than or equal to the second Hamming distance.

[0038] Calculate the ratio between the first Hamming distance and the second Hamming distance. If the ratio is less than a second threshold, it means that the second feature point corresponding to the first Hamming distance is unique to the first feature point, and the second feature point is used as the matching feature point of the first feature point. If the ratio is greater than or equal to the second threshold, it means that the second feature point corresponding to the first Hamming distance is not unique to the first feature point, and the first feature point has no matching feature point.

[0039] After determining whether each of the first feature points has a matching feature point, the total number of all matching feature points is counted. If the total number is higher than a third threshold, the homography transformation matrix of the first test grayscale image relative to the first reference grayscale image is calculated based on the first coordinates of the first feature point with matching feature points on the first reference grayscale image and the second coordinates of each matching feature point on its respective first test grayscale image. This homography transformation matrix is ​​then used as the perspective transformation parameter of the test image relative to the reference image.

[0040] In conjunction with the first aspect, this application provides a fifth possible implementation of the first aspect, wherein comparing the differences between the screen region to be detected in the reference image corresponding to each of the reference images and the screen region to be detected in the perspective-transformed test image, to determine the screen fault detection result based on the differences corresponding to each of the reference images, includes:

[0041] The two reference images with the largest brightness difference are selected from all the reference images as the first reference image and the second reference image. Based on the brightness difference between the reference image corresponding to the first reference image and the reference image corresponding to the second reference image, the position area of ​​the screen to be detected on the reference image is determined, and the screen mask of the screen to be detected is obtained.

[0042] Select the reference image with the lowest brightness from all the reference images as the target reference image. For the reference image corresponding to the target reference image and the test image after perspective change, calculate the average brightness of each pixel in the reference image to obtain a first average brightness value, and calculate the average brightness of each pixel in the test image after perspective change to obtain a second average brightness value.

[0043] Based on the first average brightness value and the second average brightness value, a brightness correction coefficient is determined. The brightness correction coefficient is then used to correct the brightness of each test image after perspective change, resulting in each test image after brightness correction.

[0044] Based on the screen mask, the screen area to be detected in each of the reference images and the screen area to be detected in each of the brightness-corrected test images are determined. The difference between the screen area to be detected in the reference image corresponding to each reference image and the screen area to be detected in the brightness-corrected test image is compared, so as to determine the screen fault detection result based on the difference corresponding to each reference image.

[0045] In conjunction with the fifth possible implementation of the first aspect, this application provides a sixth possible implementation of the first aspect, wherein selecting the two reference images with the largest brightness difference from all the reference images as the first reference image and the second reference image, determining the position region of the screen to be detected on the reference image based on the brightness difference between the reference image corresponding to the first reference image and the reference image corresponding to the second reference image, and obtaining the screen mask of the screen to be detected, includes:

[0046] Each test image after perspective transformation is processed into grayscale to obtain a second test grayscale image for each test image; the second test grayscale image contains the brightness value of each pixel;

[0047] For each of the second test grayscale images, the brightness value of each pixel located in the image edge region of the second test grayscale image is compared with a fourth threshold. If the brightness value of a pixel located in the image edge region is less than or equal to the fourth threshold, a first symbol is used to mark that pixel; if the brightness value of a pixel located in the image edge region is greater than the fourth threshold, a second symbol is used to mark that pixel; and the second symbol is used to mark each pixel located in the image center region of the second test grayscale image to obtain the mask bitmap corresponding to the second test grayscale image; wherein, the position marked with the first symbol in the mask bitmap is the position filled with black during the perspective change, and the position marked with the second symbol is not the position filled with black during the perspective change;

[0048] Based on the mask bitmaps of all the second test grayscale images, a common region is determined for each test image after perspective transformation; wherein each pixel in the common region is marked with the second symbol in each of the mask bitmaps;

[0049] Using the common area, each of the reference images and each of the perspective-transformed test images are cropped, retaining only a portion of the image in the common area, to obtain each cropped reference image and test image;

[0050] The two reference images with the largest brightness difference are selected from all the reference images as the first reference image and the second reference image, and the cropped reference image corresponding to the first reference image is used as the first reference image, and the cropped reference image corresponding to the second reference image is used as the second reference image.

[0051] For each pixel in the first reference image and the second reference image, the brightness difference at that pixel is calculated. If the brightness difference is greater than the fifth threshold, the second symbol is used to mark the pixel. If the brightness difference is less than or equal to the fifth threshold, the first symbol is used to mark the pixel, so as to obtain the screen mask of the screen to be detected composed of the first symbol and the second symbol.

[0052] In conjunction with the fifth possible implementation of the first aspect, this application provides a seventh possible implementation of the first aspect, wherein determining a brightness correction coefficient based on the first average brightness value and the second average brightness value, and using the brightness correction coefficient to correct the brightness of each test image after perspective change, to obtain each test image after brightness correction, includes:

[0053] Calculate the ratio of the first average brightness value to the second average brightness value, and use the ratio result as the brightness correction coefficient;

[0054] For each test image after perspective change, the brightness correction coefficient is multiplied by the pixel value of each pixel in the test image to obtain the new pixel value of each pixel after brightness correction. The new pixel value of each pixel is used as the pixel value of each pixel in the brightness-corrected test image to obtain each test image after brightness correction.

[0055] In conjunction with the fifth possible implementation of the first aspect, this application provides an eighth possible implementation of the first aspect, wherein the step of determining the screen region to be detected in each of the reference images and determining the screen region to be detected in each brightness-corrected test image based on the screen mask, comparing the difference between the screen region to be detected in the reference image corresponding to each reference image and the screen region to be detected in the brightness-corrected test image, and determining the screen fault detection result based on the difference corresponding to each of the reference images, includes:

[0056] For each reference image and the brightness-corrected test image corresponding to the reference image, grayscale processing is performed on the reference image and the test image to obtain the second reference grayscale image and the third test grayscale image corresponding to the reference image.

[0057] Based on the screen mask, the first screen region to be detected in the second reference grayscale image and the second screen region to be detected in the third test grayscale image are determined, and the grayscale values ​​of corresponding positions in the first screen region to be detected and the second screen region to be detected are subtracted and the absolute value is taken.

[0058] If the absolute value is greater than the sixth threshold, the second symbol is used to mark the position; if the absolute value is less than or equal to the sixth threshold, the first symbol is used to mark the position, thus obtaining the fault marking map corresponding to the reference image; wherein, the area marked with the second symbol in the fault marking map represents the fault area;

[0059] The fault marker maps corresponding to all the reference images are superimposed to obtain the overall fault marker map.

[0060] Secondly, embodiments of this application also provide a screen fault detection device, comprising:

[0061] The first recording module is used to record a reference video containing the screen under test while each reference image is played sequentially according to a preset playback duration and preset playback order on the screen under test that has not experienced a fault.

[0062] The second recording module is used to record a test video containing the screen under test during the fault detection phase, in which the screen under test plays each of the reference images in sequence according to the preset playback duration and the preset playback order.

[0063] The detection module is configured to, for each reference image, detect one video frame recorded while playing the reference image from the continuous video frames contained in the reference video, and obtain the reference image corresponding to the reference image; and detect one video frame recorded while playing the reference image from the continuous video frames contained in the test video, and obtain the test image corresponding to the reference image.

[0064] The calculation module is used to calculate the perspective transformation parameters of the test image relative to the reference image, so as to perform perspective transformation on each frame of the test image using the perspective transformation parameters;

[0065] The comparison module is used to compare the differences between the screen region to be detected in the reference image corresponding to each of the reference images and the screen region to be detected in the perspective-transformed test image, so as to determine the screen fault detection result based on the differences corresponding to each of the reference images.

[0066] Thirdly, embodiments of this application also provide an electronic device, including: a processor, a memory, and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor communicates with the memory via the bus, and when the machine-readable instructions are executed by the processor, the steps in any of the possible implementations of the first aspect described above are performed.

[0067] Fourthly, embodiments of this application also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the steps in any of the possible implementations of the first aspect described above.

[0068] This application provides a screen fault detection method, apparatus, electronic device, and readable storage medium. In this method, the testing personnel only need to record a reference video containing the screen under test while the screen under test is playing each reference image sequentially according to a preset playback duration and preset playback order during the fault detection phase. Then, during the fault detection phase, a test video containing the screen under test is recorded while the screen under test is playing each reference image sequentially according to a preset playback duration and preset playback order. Subsequently, the reference video and test video can be automatically analyzed and processed using this method to automatically determine the screen fault detection result.

[0069] In this embodiment, the testing personnel only need to record two videos (a reference video and a test video), eliminating the need for specialized training and the use of professional instruments (such as a colorimeter). This reduces manual intervention and lowers testing costs. Furthermore, the computer in this embodiment can automatically analyze and process multiple reference and test videos from multiple screens under testing in batches, improving testing efficiency.

[0070] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0071] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0072] Figure 1 A flowchart of a screen fault detection method provided in an embodiment of this application is shown;

[0073] Figure 2 This illustration shows a frame-capturing process for a reference image provided in an embodiment of this application;

[0074] Figure 3 This paper shows a schematic diagram of the structure of a screen fault detection device provided in an embodiment of this application;

[0075] Figure 4A schematic diagram of the structure of an electronic device provided in an embodiment of this application is shown. Detailed Implementation

[0076] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely represents selected embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0077] Traditional screen damage (fault) detection methods are mostly quite professional and require the participation of professionals. For example, using a colorimeter to inspect the screen requires not only purchasing an expensive colorimeter, but also special training for technicians or hiring external professionals to use the instrument correctly. The detection cost is high, and it can only be tested one screen at a time, making batch testing impossible. The testing time is also long and time-consuming.

[0078] However, some venues utilize a large number of screens, especially large ones, such as conference rooms, exhibition halls, plazas, and cinemas. Given the sheer number of these large screens, achieving low-cost, automated, and rapid fault detection has become a primary concern.

[0079] Based on this, embodiments of this application provide a screen fault detection method, apparatus, electronic device, and readable storage medium. This eliminates the need for specialized training of testing personnel and the use of professional instruments (such as colorimeters), reducing manual intervention and testing costs. Furthermore, it allows for automated batch analysis and processing of reference and test videos from multiple screens under test, improving testing efficiency. The following description uses embodiments to illustrate this.

[0080] To facilitate understanding of this embodiment, a screen fault detection method disclosed in this application will first be described in detail. For example... Figure 1 As shown, the method includes the following steps S101-S105:

[0081] S101: While the screen under test is playing each reference image in a preset playback duration and preset playback order without any malfunction, a reference video containing the screen under test is recorded.

[0082] In this embodiment, the screen to be tested refers to the screen that needs to be fault detected. The screen refers to the display screen on the video playback device. The screen to be tested includes, for example, mobile phone screens, tablet computer screens, laptop screens, desktop computer screens, television screens, outdoor LED screens, cinema screens, etc.

[0083] This embodiment is divided into two stages: the first stage is the preparation stage (the stage where the screen under test has not yet malfunctioned), and the second stage is the fault detection stage. The preparation stage can be the stage when the screen under test has just been purchased or is about to be put into use. The screen under test in this stage is generally new and unused. New screens are mostly fault-free and normal (i.e., functioning normally). To further ensure that the screen under test in the preparation stage is fault-free, it can be further tested manually or through other testing methods to ensure that the screen under test in the preparation stage is fault-free. The fault detection stage generally refers to the stage where the screen under test is periodically tested for faults after it has been in use for a period of time.

[0084] Step S101 is the preparation stage. In step S101, for the screen to be tested that has not experienced any malfunctions, the screen to be tested is controlled to play each reference image in sequence according to the preset playback duration and preset playback order. During the process of the screen to be tested that has not experienced any malfunctions playing each reference image in sequence according to the preset playback duration and preset playback order, the camera is controlled to record a reference video containing the screen to be tested.

[0085] There are multiple reference images, specifically including: a completely black image, a completely white image, and images of the three primary colors of red, green, and blue at 20%, 40%, 60%, 80%, and 100% brightness, for a total of 17 images.

[0086] The images (15 images in total) show the pure red, green, and blue primary colors at 20%, 40%, 60%, 80%, and 100% brightness. Specifically, these refer to: a red image at 20% brightness, a red image at 40% brightness, a red image at 60% brightness, a red image at 80% brightness, and a red image at 100% brightness; a green image at 20% brightness, a green image at 40% brightness, a green image at 60% brightness, a green image at 80% brightness, and a green image at 100% brightness; and a blue image at 20% brightness, a blue image at 40% brightness, a blue image at 60% brightness, a blue image at 80% brightness, and a blue image at 100% brightness.

[0087] The preset playback duration refers to the playback duration of each reference image. The playback duration for each reference image is the same and is the preset playback duration. For example, if the screen to be inspected plays one reference image every 2 seconds, the preset playback duration is 2 seconds. The preset playback order refers to the playback order of the reference images. The playback order of the reference images is consistent during both the preparation and fault detection phases.

[0088] When controlling the camera to record a reference video containing the screen to be detected, recording typically begins 2 seconds before the first reference image is played on the screen to be detected. For example, ideally, when the preset playback duration is 2 seconds and there are 17 reference images, the reference video needs to be recorded for 36 seconds.

[0089] In this embodiment, when recording a reference video containing the screen to be inspected using a camera, the camera needs to be aimed directly at the screen to be inspected for recording (shooting). The screen to be inspected should be complete during shooting, and its position in the video should be as large as possible to improve the fault detection and recognition rate. The resulting video is called the reference video. The reference video only needs to be recorded once and saved. It will be used as a comparison standard in each subsequent periodic fault detection phase.

[0090] S102: During the fault detection phase, while the screen under test plays each reference image sequentially according to the preset playback duration and preset playback order, a test video containing the screen under test is recorded.

[0091] Step S102 is the fault detection stage. In step S102, for the screen to be tested in the fault detection stage, the screen to be tested is controlled to play each reference image in sequence according to the preset playback duration and preset playback order of the reference images. During the process of the screen to be tested in the fault detection stage playing each reference image in sequence according to the preset playback duration and preset playback order, the camera is controlled to record a test video containing the screen to be tested.

[0092] Typically, when recording test videos, the scene where the screen to be tested is located should be the same as when the reference video was recorded, and the camera should also be in the same position. Each test video should be recorded independently for each fault detection stage. In other words, for different fault detection stages, a separate test video for that fault detection stage needs to be recorded.

[0093] S103: For each reference image, detect one video frame recorded while playing the reference image from the continuous video frames contained in the reference video to obtain the reference image corresponding to the reference image; and detect one video frame recorded while playing the reference image from the continuous video frames contained in the test video to obtain the test image corresponding to the reference image.

[0094] In step S103, taking a reference video as an example, when recording the reference video, since each reference image is played for a preset playback duration (e.g., 2 seconds), the recorded reference video will have multiple consecutive video frames in which the screen to be detected plays the same reference image.

[0095] For example, assuming the preset playback duration is 2 seconds, there are 17 reference images, the reference video is recorded for 36 seconds, and the video frame rate is 30 fps (30 frames per second), then there are a total of 30 × 36 = 1080 frames in the reference video, and each reference image is played for 2 × 30 = 60 frames.

[0096] For each reference image, one video frame recorded while playing the reference image is selected from the consecutive video frames contained in the reference video. The content displayed on the screen to be detected in this video frame is consistent with the content in the reference image; this video frame is then used as the reference image. For example, for a "completely black image" as the reference image, the content displayed on the screen to be detected in the corresponding reference image is also completely black. Each reference image corresponds to one reference image, and the number of reference images is the same as the number of reference images.

[0097] Similarly, from the continuous video frames contained in the test video, select one video frame recorded while playing the reference image. If the content displayed on the screen to be tested in this video frame matches the content in the reference image, then this video frame is used as the test image for the reference image. For example, for a "completely black image" as the reference image, the content displayed on the screen to be tested in the corresponding test image is also completely black. Each reference image corresponds to one test image, and the number of test images is the same as the number of reference images.

[0098] In one possible implementation, when performing step S103, the following steps S1031-S1034 can be specifically performed:

[0099] S1031: For each video in the reference video and test video, detect the first mutation frame when the first reference image is played from the continuous video frames contained in the video, and crop the video frames before the first mutation frame in the video to obtain the reference video and test video with the first frame aligned.

[0100] In step S1031, the first abrupt change frame when playing the first reference image is detected from the consecutive video frames contained in the reference video, and the video frames before the first abrupt change frame in the reference video are cropped. Similarly, the first abrupt change frame when playing the first reference image is detected from the consecutive video frames contained in the test video, and the video frames before the first abrupt change frame in the test video are cropped. This yields a reference video and a test video with first-frame alignment.

[0101] In this embodiment, whether it is a reference video or a test video, since the video frames are continuous, when the reference image starts playing and displaying, it will continue for multiple frames (for example, a reference image is displayed for 2 seconds and 60 frames are played). The first mutation frame does not refer to any frame within these 2 seconds (that is, not any frame among these 60 frames), but the very first frame that appears, which is the critical point from "not" to "yes".

[0102] In other words, the first mutation frame refers to the video frame in a series of video frames where, when the "first reference image" is played, the image content of the "first reference image" can be clearly and distinctly identified. That is, the first mutation frame is the first video frame in a series of video frames whose screen display content matches the content of the first reference image.

[0103] In one possible implementation, when performing step S1031, for each video in the reference video and the test video, detecting the first abrupt change frame from the consecutive video frames contained in that video when playing the first reference image, the specific steps S10311-S10314 can be performed as follows:

[0104] S10311: Determine the minimum value between the total number of video frames in the reference video and the total number of video frames in the test video, and use this minimum value as the target total number of frames.

[0105] In step S10311, assume the reference video is r[i], i=1,2,...,m. The test video is t[j], j=1,2,...,n. Here, i represents the number of frames in the reference video, m represents the total number of frames in the reference video, j represents the number of frames in the test video, and n represents the total number of frames in the test video. The minimum value between m and n, k=min(m,n), is taken as the target total number of frames.

[0106] S10312: Extract video frames from both the reference video and the test video to obtain a reference video and a test video with the total number of frames aligned.

[0107] In step S10312, the first k frames of both the reference video and the test video are extracted to obtain the reference video (r[i], i=1,2,...,k) and the test video (t[j], j=1,2,...,k) with the total number of frames aligned.

[0108] In this embodiment, step S103 mainly consists of two steps: frame number alignment and time alignment. Steps S10311-S10312 described above are the specific process of frame number alignment.

[0109] However, after processing in steps S10211-S10212, the reference video and test video have the same total frame count and length, but the start recording time may differ slightly. For example, the test video may start slightly earlier or later, making precise time alignment impossible. Therefore, in this embodiment, the first abrupt change frame is used as the alignment time point. Assuming the "completely black image" in the reference image is the first reference image played, when the first reference image "completely black image" starts playing, the screen to be detected will abruptly change to a "completely black image." By detecting the first abrupt change frame in the reference video and test video after total frame count alignment, and using this first abrupt change frame as the alignment time point, time alignment is achieved.

[0110] In this embodiment, the first mutation frame in the reference video and test video after total frame number alignment is detected through steps S10313-S10314, so as to achieve time alignment between the reference video and the test video.

[0111] S10313: For each video in the reference video and test video with total frame number alignment, calculate the inter-frame similarity between any two adjacent video frames in the video to obtain the inter-frame similarity sequence, and use the first-order difference algorithm to calculate the first-order transition sequence of the inter-frame similarity sequence.

[0112] In step S10313, the inter-frame structural similarity index algorithm (SSIM) is used to calculate the inter-frame similarity between any two adjacent video frames in the reference video, thus obtaining the inter-frame similarity sequence corresponding to the reference video. Similarly, the inter-frame structural similarity index algorithm (SSIM) is used to calculate the inter-frame similarity between any two adjacent video frames in the test video, thus obtaining the inter-frame similarity sequence corresponding to the test video.

[0113] In this embodiment, for each video in the reference video and test video aligned with the total number of frames, the inter-frame similarity between any two adjacent video frames in the video can be calculated using the following formula:

[0114]

[0115] in, This is the brightness comparison function; This is a contrast comparison function; For structural comparison functions; , , The preset coefficient is usually taken as... =1; and Represents two adjacent video frames; For video frames and Inter-frame similarity.

[0116]

[0117]

[0118] in, It is a video frame The average value of the pixels; It is a video frame The average value of the pixel values; L is a preset value, specifically, L is the maximum value of the pixel values, usually L=255; K1 is an empirical coefficient, usually K1=0.01.

[0119]

[0120]

[0121] in, It is a video frame The standard deviation of pixel values; It is a video frame The standard deviation of the pixel values; K2 is an empirical coefficient, usually taken as K2=0.03.

[0122]

[0123]

[0124] in, It is a video frame and The covariance of pixel values.

[0125] The inter-frame similarity sequence (SSIM sequence) corresponding to the reference video contains the inter-frame similarity between any two adjacent video frames in the reference video (r[i], i=1,2,...,k) after total frame alignment. The inter-frame similarity sequence (SSIM sequence) corresponding to the test video contains the inter-frame similarity between any two adjacent video frames in the test video (t[j], j=1,2,...,k) after total frame alignment.

[0126] For the inter-frame similarity sequence corresponding to the reference video, the following first-order difference algorithm is used to calculate the first-order jump sequence of the inter-frame similarity sequence:

[0127]

[0128] in, This represents the r-th inter-frame similarity in the inter-frame similarity sequence corresponding to the reference video; This represents the first-order jump value corresponding to the inter-frame similarity in the first-order jump sequence of the reference video. This represents the (r+1)th inter-frame similarity in the inter-frame similarity sequence corresponding to the reference video. The number of inter-frame similarities in the inter-frame similarity sequence corresponding to the reference video is usually one less than the total number of frames in the reference video. For example, if the total number of frames in the reference video is 100, the number of inter-frame similarities in the inter-frame similarity sequence corresponding to the reference video is 99.

[0129] Similarly, for the inter-frame similarity sequence corresponding to the test video, the following first-order difference algorithm is used to calculate the first-order jump sequence of the inter-frame similarity sequence:

[0130]

[0131] in, This represents the t-th inter-frame similarity in the inter-frame similarity sequence corresponding to the test video; This represents the first-order jump value corresponding to the t-th frame similarity in the first-order jump sequence corresponding to the test video. This represents the (t+1)th inter-frame similarity in the inter-frame similarity sequence corresponding to the test video. The number of inter-frame similarities in the inter-frame similarity sequence corresponding to the test video is usually one less than the total number of frames in the test video. For example, if the total number of frames in the test video is 100, the number of inter-frame similarities in the inter-frame similarity sequence corresponding to the test video is 99.

[0132] S10314: Select a preset proportion of the maximum first-order jump value in the first-order jump sequence as the first threshold, search for the first first-order jump value in the first-order jump sequence that exceeds the first threshold, and take the second video frame of the two video frames corresponding to the first-order jump value as the first mutation frame when playing the first reference image.

[0133] In step S10314, the preset ratio can be 50%.

[0134] For the first-order transition sequence corresponding to the reference video, half of the maximum first-order transition value in the sequence is selected as the first threshold. Whenever the inter-frame first-order transition value between two adjacent frames exceeds this first threshold, it indicates a significant change in the image displayed on the screen to be detected, signifying a switch of the reference image. Therefore, by searching for the first first-order transition value exceeding the first threshold in the sequence, we can pinpoint the moment when the screen to be detected in the reference video begins playing the first reference image (e.g., a "completely black image").

[0135] Since the first-order jump value refers to the first-order jump value between two adjacent frames, the next video frame in the two adjacent video frames corresponding to the first first-order jump value that exceeds the first threshold is taken as the first mutation frame when the first reference image is played in the reference video, thus determining the first mutation frame in the reference video.

[0136] For the test video, the same method is used: half of the maximum first-order jump value in the corresponding first-order jump sequence is selected as the first threshold. By searching for the first first-order jump value in the sequence that exceeds the first threshold, this is the moment when the screen to be tested in the test video begins to play the first reference image (such as a "completely black image"). The next video frame between the two adjacent frames corresponding to the first first-order jump value that exceeds the first threshold is taken as the first abrupt change frame when the first reference image is played in the test video, thus determining the first abrupt change frame in the test video.

[0137] In step S1031, after determining the first mutation frame in the reference video, the video frames preceding the first mutation frame in the reference video (after aligning the total number of frames) are cropped. Similarly, after determining the first mutation frame in the test video, the video frames preceding the first mutation frame in the test video (after aligning the total number of frames) are cropped. This yields the reference video and test video with first frame alignment.

[0138] Then, the reference video and test video with first-frame alignment are made to have the same number of frames. That is, the minimum of the total number of video frames in the reference video with first-frame alignment and the total number of video frames in the test video with first-frame alignment is determined again, and this minimum value is taken as the new target total number of frames. From both the reference video and test video with first-frame alignment, video frames up to the new target total number of frames are extracted to obtain reference videos and test videos with aligned total frame counts and first frames.

[0139] In this way, the first video frame in both the reference video and the test video with first frame alignment is the first video frame of the first reference image (such as a "completely black image"), and each subsequent video frame is automatically aligned with the time, thus achieving time alignment between the test video and the reference video.

[0140] In this embodiment, after achieving time alignment between the test video and the reference video, keyframes (i.e., the reference image and test image corresponding to each reference image) are obtained at intermediate moments based on the preset playback duration of the reference image. This avoids instability during the switching process between the two reference images and interference caused by temporary changes in lighting. Specifically, in this embodiment, through steps S1032-S1034, the test image and reference image are extracted from the time-aligned test video and reference video, respectively.

[0141] S1032: Calculate the product of the preset playback duration and the video frame rate, use the product result as the target interval frame number, and calculate the ratio of the target interval frame number to 2, use the ratio result as the target frame number.

[0142] In step S1032, for example, if the preset playback duration is 2 seconds, and the video frame rate of the test video and the reference video are the same, both are fps=30 (30 frames per second), then the target interval frame count is 60 frames, and the target frame count is 30 frames.

[0143] S1033: From the continuous video frames contained in the reference video after the first frame is aligned, select a video frame with the target number of frames as the first frame reference image. Then, select a video frame every target interval frame number as the reference image for each frame to obtain the reference image corresponding to each reference image.

[0144] In step S1033, following the example above, such as... Figure 2 As shown, the first frame reference image is the 30th frame of the reference video after the first frame is aligned. Then, one frame is taken every 60 frames, for a total of 17 frames (because there are 17 reference images) as reference images.

[0145] S1034: From the continuous video frames contained in the test video after the first frame alignment, select a video frame with the target number of frames as the first frame test image. Then, select a video frame every target interval frame number as the test image for each frame to obtain the test image corresponding to each reference image.

[0146] In step S1034, following the example above, the first test image is the 30th video frame in the test video after the first frame is aligned. Then, one frame is taken every 60 frames, for a total of 17 video frames as test images.

[0147] In the example above, after processing, both the reference video and the test video each yield 17 video frames, corresponding to 17 baseline images. The 17 video frames selected from the reference video are called the reference image sequence, and the 17 video frames selected from the test video are called the test image sequence.

[0148] Since all reference images are recorded simultaneously by a single camera, they are spatially aligned by default. Similarly, since all test images are recorded simultaneously by a single camera, they are spatially aligned by default. However, because the test video and reference video are not recorded simultaneously, the test images and reference images are usually spatially misaligned. Therefore, in this embodiment, after obtaining the reference image and test image corresponding to each baseline image, it is necessary to spatially align the baseline image and test image through step S104.

[0149] S104: Calculate the perspective transformation parameters of the test image relative to the reference image, and use the perspective transformation parameters to perform perspective transformation on each frame of the test image.

[0150] In step S104, since the reference images are spatially aligned, the perspective transformation parameters of all test images relative to one of the reference images are first calculated. Subsequently, these perspective transformation parameters are used to perform perspective transformation on each frame of the test image, ensuring that the perspective-transformed test images are spatially registered (i.e., spatially aligned) with the reference images. After spatial registration, the reference image and the test images are at the same pixel position, corresponding to the same location in space. This eliminates the deviation caused by camera position jitter.

[0151] In one possible implementation, when performing step S104 to calculate the perspective transformation parameters of the test image relative to the reference image, the specific steps S1041-S1045 can be performed as follows:

[0152] S1041: Select one frame of reference image, perform grayscale processing on the reference image and all test images to obtain a first reference grayscale image and multiple frames of first test grayscale images.

[0153] In step S1041, a reference image is randomly selected from multiple reference images, or the first reference image can be selected (for example, the reference image displayed on the screen to be detected in the first reference image is a "completely black image").

[0154] The selected reference frame is processed into a grayscale image to obtain a first reference grayscale image. Similarly, all test images are processed into grayscale images to obtain a first test grayscale image corresponding to each test frame.

[0155] S1042: Perform feature extraction on the first reference grayscale image to obtain multiple first feature points on the first reference grayscale image; and perform feature extraction on each frame of the first test grayscale image to obtain multiple second feature points on each frame of the first test grayscale image.

[0156] In step S1042, the ORB feature extraction algorithm is used to extract features from the first reference grayscale image, extracting multiple first feature points on the first reference grayscale image, for example, extracting 3000 first feature points.

[0157] Furthermore, the ORB feature extraction algorithm is used to extract multiple second feature points on each frame of the test image.

[0158] S1043: For each first feature point, use the K-nearest neighbor brute-force matching algorithm based on Hamming distance to calculate the Hamming distance between the first feature point and each second feature point, and select the two smallest Hamming distances from all Hamming distances corresponding to the first feature point as the first Hamming distance and the second Hamming distance respectively; the first Hamming distance is less than or equal to the second Hamming distance.

[0159] In step S1043, for each first feature point extracted from the first reference grayscale image, a matching degree calculation is performed between it and each second feature point extracted from each of the first test grayscale images. The matching degree calculation uses a K-nearest neighbor algorithm based on Hamming distance to calculate the Hamming distance between the first feature point and each second feature point in each of the first test grayscale images. The calculated Hamming distance characterizes the similarity in position and brightness between the first and second feature points; a smaller Hamming distance indicates higher similarity, and vice versa.

[0160] Select the two smallest Hamming distances from all Hamming distances corresponding to the first feature point, and take them as the first Hamming distance a and the second Hamming distance b, respectively; the first Hamming distance a is less than or equal to the second Hamming distance b.

[0161] S1044: Calculate the ratio between the first Hamming distance and the second Hamming distance. If the ratio is less than the second threshold, it means that the second feature point corresponding to the first Hamming distance is unique to the first feature point, and the second feature point is used as the matching feature point of the first feature point. If the ratio is greater than or equal to the second threshold, it means that the second feature point corresponding to the first Hamming distance is not unique to the first feature point, and the first feature point has no matching feature point.

[0162] In step S1044, it is assumed that the first Hamming distance a is the Hamming distance between the first feature point W and the second feature point Q1, and the second Hamming distance b is the Hamming distance between the first feature point W and the second feature point Q2.

[0163] Calculate the ratio a / b between the first Hamming distance a and the second Hamming distance b. This ratio a / b, for the first feature point W, can characterize whether the second feature point Q1 corresponding to the first Hamming distance is unique to the first feature point W compared to other second feature points.

[0164] Generally, the smaller the ratio a / b, the better. If the ratio a / b equals 1, it means that the two second feature points Q1 and Q2 are the same as the first feature point W. In this case, the second feature point is not unique and may be an error value, so it should be discarded.

[0165] If the ratio a / b is less than the second threshold (usually 0.75), it means that the second feature point Q1 corresponding to the first Hamming distance a is unique to the first feature point W. The second feature point Q1 is then used as the matching feature point of the first feature point W and added to the matching point list.

[0166] If the ratio a / b is greater than or equal to the second threshold (e.g., 0.75), it means that the second feature point Q1 corresponding to the first Hamming distance a is not unique to the first feature point W. In this case, it is considered that the first feature point W has no matching feature point, and the second feature point Q1 is discarded and not added to the matching point list.

[0167] S1045: After determining whether each first feature point has a matching feature point, count the total number of all matching feature points. If the total number is higher than the third threshold, calculate the homography transformation matrix of the first test grayscale image relative to the first reference grayscale image based on the first coordinates of the first feature point with matching feature points on the first reference grayscale image and the second coordinates of each matching feature point on its respective first test grayscale image. Use this homography transformation matrix as the perspective transformation parameter of the test image relative to the reference image.

[0168] In step S1045, the total number of matching feature points is less than or equal to the total number of first feature points, because each first feature point has at most one matching feature point. After determining whether each first feature point has a matching feature point, the total number of matching feature points in the matching point list is counted. If the total number is higher than a third threshold (e.g., 100), the matching is considered successful. At this time, based on the first coordinates of the first feature points with matching feature points on the first reference grayscale image, and the second coordinates of each matching feature point on its respective first test grayscale image, the homography transformation matrix H of the first test grayscale image relative to the first reference grayscale image is calculated, and this homography transformation matrix H is used as the perspective transformation parameter of the test image relative to the reference image.

[0169] If the total number is lower than the third threshold (e.g., 100), the matching is considered to have failed, and the test video recorded in step S102 is invalid.

[0170] In this embodiment, only one homography transformation matrix H (i.e., perspective transformation parameter) needs to be calculated. This homography transformation matrix H (i.e., perspective transformation parameter) is used to perform a perspective transformation on each test image extracted in step S103, so that the perspective-transformed test image is used as the test image spatially aligned with the reference image.

[0171] S105: Compare the differences between the screen region to be detected in the reference image corresponding to each reference image and the screen region to be detected in the perspective-transformed test image, so as to determine the screen fault detection result based on the differences corresponding to each reference image.

[0172] In one possible implementation, when performing step S105, the following steps S1051-S1055 can be specifically performed:

[0173] S1051: Select the two reference images with the largest brightness difference from all reference images as the first reference image and the second reference image. Based on the brightness difference between the reference image corresponding to the first reference image and the reference image corresponding to the second reference image, determine the position area of ​​the screen to be detected on the reference image and obtain the screen mask of the screen to be detected.

[0174] In step S1051, since this embodiment detects whether the screen to be detected is faulty, areas that are not the screen to be detected are not considered. By extracting the screen mask of the screen to be detected, interference can be reduced and the detection accuracy can be improved.

[0175] In the embodiment of step S101, the reference images include a completely black image, a completely white image, and images of the pure red, green, and blue primary colors at brightness levels of 20%, 40%, 60%, 80%, and 100%, totaling 17 images. Therefore, in step S1051, the two reference images with the greatest brightness difference are the completely black image and the completely white image, respectively. The completely black image and the completely white image are used as the first reference image and the second reference image, respectively.

[0176] Since the brightness of other areas remains basically unchanged when the screen to be tested displays a completely black image and a completely white image, the position area of ​​the screen to be tested on the reference image can be determined based on the brightness difference between the reference image corresponding to the completely black image and the reference image corresponding to the completely white image, thus obtaining a screen mask used to represent the position area of ​​the screen to be tested on the reference image.

[0177] In one possible implementation, after performing perspective transformation (including translation, scaling, and rotation) on each frame of the test image in step S104, the missing areas in the test image are filled with black. Therefore, in step S1051, when determining the screen mask of the screen to be detected, it is necessary to first extract the common area in each frame of the test image after perspective transformation, exclude the areas filled with black, and then extract the screen area to be detected from the common area, thereby determining the screen mask of the screen to be detected. Based on this, when performing step S1051, it can be specifically executed according to the following steps S10511-S10516:

[0178] S10511: Perform grayscale processing on each test image after perspective transformation to obtain a second test grayscale image for each test image; the second test grayscale image contains the brightness value of each pixel.

[0179] In step S10511, each test image after spatial alignment (i.e., after perspective transformation) is processed into grayscale to obtain each second test grayscale image after grayscale processing. Only the brightness value of each pixel is retained in the second test grayscale image.

[0180] S10512: For each second test grayscale image, compare the brightness value of each pixel located in the image edge region with the fourth threshold. If the brightness value of a pixel located in the image edge region is less than or equal to the fourth threshold, mark the pixel with the first symbol; if the brightness value of a pixel located in the image edge region is greater than the fourth threshold, mark the pixel with the second symbol; and mark each pixel located in the image center region of the second test grayscale image with the second symbol to obtain the mask bitmap corresponding to the second test grayscale image; wherein, the position marked with the first symbol in the mask bitmap is the position filled with black during the perspective change, and the position marked with the second symbol is not the position filled with black during the perspective change.

[0181] In step S10512, a very small fourth threshold is set (after spatial alignment, the pixel brightness of the black-filled boundary is zero), and this fourth threshold is generally set to 0.

[0182] The second test grayscale image is divided into an image edge region and an image center region. Under normal circumstances, the screen to be tested should be located in the image center region, and the black fill position that appears after perspective change should be located in the image edge region.

[0183] If the brightness value of a pixel located in the edge region of the second test grayscale image is less than or equal to the fourth threshold, then the pixel is considered to be a position filled with black. In this case, the first symbol is used to mark it, and the first symbol can be "0".

[0184] If the brightness value of a pixel located in the edge region of the second test grayscale image is greater than the fourth threshold, it indicates that the pixel is not a position filled with black. In this case, the second symbol is used to mark it, and the second symbol can be "1".

[0185] Since the black fill position that appears after perspective change is usually located in the edge area of ​​the image, the pixels in the center area of ​​the second test grayscale image are marked with the second symbol to obtain the mask bitmap corresponding to the second test grayscale image; wherein, the area marked with the first symbol "0" in the mask bitmap is the area filled with black during the perspective change, and the area marked with the second symbol "1" is not the area filled with black during the perspective change.

[0186] S10513: Based on the mask bitmaps of all the second test grayscale images, determine the common region of each test image after perspective transformation; wherein each pixel in the common region is marked with a second symbol in each mask bitmap.

[0187] In step S10513, each second test grayscale image corresponds to a mask bitmap. An AND operation is performed on the same positions among all mask bitmaps. If the position is marked with the second symbol "1" in all mask bitmaps, the value of that position is set to "1"; otherwise, it is set to "0". The regions with a value of "1" are extracted to obtain the common region of each frame of the second test grayscale images.

[0188] S10514: Using the common area, crop each reference image and each test image after perspective transformation, retaining only a portion of the image in the common area, to obtain each cropped reference image and test image.

[0189] In step S10514, the common area is used to crop the reference images of each frame and the test images after perspective transformation, retaining only a portion of the image in the common area.

[0190] In this embodiment, after cropping the reference image and test image to retain only the common area through steps S10511-S10514, the screen area to be detected is extracted through steps S10515-S10516 to determine the screen mask of the screen to be detected.

[0191] S10515: Select the two reference images with the largest brightness difference from all reference images as the first reference image and the second reference image, and use the cropped reference image corresponding to the first reference image as the first reference image, and use the cropped reference image corresponding to the second reference image as the second reference image.

[0192] In step S10515, the two reference images with the greatest brightness difference are a completely black image and a completely white image, respectively. The completely black image and the completely white image are used as the first reference image and the second reference image, respectively.

[0193] The reference image cropped by steps S10511-S10514 corresponding to the all-black image is used as the first reference image, and the reference image cropped by steps S10511-S10514 corresponding to the all-white image is used as the second reference image.

[0194] S10516: For each pixel in the first reference image and the second reference image, calculate the brightness difference at that pixel. If the brightness difference is greater than the fifth threshold, mark the pixel with a second symbol. If the brightness difference is less than or equal to the fifth threshold, mark the pixel with a first symbol, so as to obtain a screen mask of the screen to be detected composed of the first symbol and the second symbol.

[0195] In step S10516, the fifth threshold is determined experimentally. For pixels at the same location in the first and second reference images, the absolute value of the brightness difference at that pixel is calculated. If the absolute value of the brightness difference is greater than the fifth threshold, the pixel is marked with the second symbol "1"; if the absolute value of the brightness difference is less than or equal to the fifth threshold, the pixel is marked with the second symbol "0". This results in a screen mask for the screen to be detected, composed of the first and second symbols. The area marked with the second symbol "1" in the screen mask is the area of ​​the screen to be detected.

[0196] In this embodiment, considering that the test video and reference video were captured at different times, potentially spanning a significant period, and that the environments of the two recordings may differ slightly in terms of lighting, window opening or closing, etc., the overall brightness of the recorded test video and reference video will deviate. This, in turn, will lead to a deviation in the overall brightness of the test image and reference image. Therefore, after obtaining the screen mask of the screen to be tested through steps S10511-S10516, the next step is to perform brightness correction on the test image after perspective change through steps S1052-S1053 to ensure that the brightness of the test image is consistent with the brightness of the reference image.

[0197] It is worth noting that the overall brightness of each frame of the reference image extracted from the reference video should be consistent, and the overall brightness of each frame of the test image extracted from the test video should be consistent.

[0198] S1052: Select the reference image with the lowest brightness from all reference images as the target reference image. For the reference image corresponding to the target reference image and the test image after perspective change, calculate the average brightness of each pixel in the reference image to obtain the first average brightness value, and calculate the average brightness of each pixel in the test image after perspective change to obtain the second average brightness value.

[0199] In step S1052, the reference image with the lowest brightness among all reference images is a completely black image. When the screen to be tested plays a completely black image, the screen has no brightness output. It can be considered that the image brightness of the reference image and the test image after perspective change is affected by the ambient brightness.

[0200] At this point, the average brightness of each pixel in the reference image corresponding to the all-black image is calculated to obtain the first average brightness value, and the average brightness of each pixel in the test image after perspective transformation corresponding to the all-black image is calculated to obtain the second average brightness value.

[0201] S1053: Determine the brightness correction coefficient based on the first average brightness value and the second average brightness value, and use the brightness correction coefficient to correct the brightness of each test image after perspective change, so as to obtain each test image after brightness correction.

[0202] In step S1052, the ratio of the first average brightness value to the second average brightness value is calculated, and the ratio result is used as the brightness correction coefficient. For each test image after perspective change, the brightness correction coefficient is multiplied by the pixel value of each pixel in the test image to obtain the new pixel value of each pixel after brightness correction. The new pixel value of each pixel is used as the pixel value of each pixel in the test image after brightness correction to obtain each test image after brightness correction.

[0203] In this embodiment, only the brightness of the test image after perspective change needs to be corrected so that the overall brightness of the test image after brightness correction is consistent with the overall brightness of the test image.

[0204] S1054: Based on the screen mask, determine the screen area to be detected in each reference image and the screen area to be detected in each test image after brightness correction. Compare the differences between the screen area to be detected in the reference image corresponding to each reference image and the screen area to be detected in the test image after brightness correction, so as to determine the screen fault detection result based on the differences corresponding to each reference image.

[0205] In step S1054, the screen region to be detected in each reference image can be determined based on the area marked with the second symbol "1" in the screen mask. Furthermore, the screen region to be detected in each brightness-corrected test image can also be determined based on the area marked with the second symbol "1" in the screen mask.

[0206] In one possible implementation, when performing step S1054, steps S10541-S10544 can be specifically performed according to the following steps:

[0207] S10541: For each reference image and the brightness-corrected test image corresponding to the reference image, perform grayscale processing on the reference image and the test image to obtain the second reference grayscale image and the third test grayscale image corresponding to the reference image.

[0208] In step S10541, the reference image and the brightness-corrected test image corresponding to the same reference image are grouped together, with two frames in each group. Grayscale processing is performed on the reference image and the test image in the same group to obtain a second reference grayscale image and a third test grayscale image.

[0209] S10542: Based on the screen mask, determine the first screen region to be detected in the second reference grayscale image and the second screen region to be detected in the third test grayscale image, and subtract the grayscale values ​​of corresponding positions in the first screen region to be detected and the absolute value.

[0210] In step S10542, the screen region to be detected in the second reference grayscale image can be determined based on the area marked with the second symbol "1" in the screen mask. And the screen region to be detected in the third test grayscale image can be determined based on the area marked with the second symbol "1" in the screen mask.

[0211] S10543: If the absolute value is greater than the sixth threshold, the second symbol is used to mark the position; if the absolute value is less than or equal to the sixth threshold, the first symbol is used to mark the position, thus obtaining the fault marking map corresponding to the reference image; wherein, the area marked with the second symbol in the fault marking map represents the fault area.

[0212] In step S10543, the sixth threshold can be set based on experience and environmental stability, for example, it can be divided into three levels: high sensitivity, medium sensitivity, and low sensitivity. The larger the sixth threshold, the lower the sensitivity and the higher the false alarm rate; the smaller the sixth threshold, the higher the sensitivity and the higher the false alarm rate.

[0213] For the same location in both the first and second screen regions to be detected, the grayscale values ​​are subtracted, and the absolute value is taken. If the absolute value is greater than a sixth threshold, the location is marked with the second symbol "1". If the absolute value is less than or equal to the sixth threshold, the location is marked with the first symbol "0", resulting in a fault marking map corresponding to the reference image. With 17 reference images, there are 17 fault marking maps. The area marked with the second symbol "1" in the fault marking map represents the fault area.

[0214] S10544: Overlay the fault marker maps corresponding to all reference images to obtain the overall fault marker map.

[0215] In this embodiment, fault marker maps corresponding to each reference image are superimposed. In all fault marker maps, if a position is marked with the second symbol "0", then the value of that position is set to "0"; otherwise, it is set to "1", thus obtaining the overall fault marker map. That is, the overall fault marker map is a mask map composed of "0" and "1". In the overall fault marker map, the area with a value of "1" represents the fault area.

[0216] After obtaining the overall fault marking map, a reference image is selected, for example, the first frame reference image (in which the baseline image displayed on the screen to be inspected is a completely black image). The fault area is determined based on the overall fault marking map, and then a third marker is used to mark the fault area on that frame reference image. The reference image with the marked fault area is then displayed on the user's (inspection personnel's) screen for viewing. Simultaneously, the location of the fault can also be output to the database for verification by the inspection personnel. The third marker is a prominent color, such as red.

[0217] Based on the same technical concept, embodiments of this application also provide a screen fault detection device, such as... Figure 3 As shown, the device includes:

[0218] The first recording module 301 is used to record a reference video containing the screen under test while each reference image is played sequentially according to a preset playback duration and preset playback order on the screen under test that has not experienced a fault.

[0219] The second recording module 302 is used to record a test video containing the screen under test during the fault detection phase, in which the screen under test plays each of the reference images in sequence according to the preset playback duration and the preset playback order.

[0220] The detection module 303 is used to detect, for each reference image, one video frame recorded when playing the reference image from the continuous video frames contained in the reference video, to obtain the reference image corresponding to the reference image; and to detect, one video frame recorded when playing the reference image from the continuous video frames contained in the test video, to obtain the test image corresponding to the reference image.

[0221] Calculation module 304 is used to calculate the perspective transformation parameters of the test image relative to the reference image, so as to perform perspective transformation on each frame of the test image using the perspective transformation parameters;

[0222] The comparison module 305 is used to compare the difference between the screen area to be detected in the reference image corresponding to each of the reference images and the screen area to be detected in the test image after perspective transformation, so as to determine the screen fault detection result based on the difference corresponding to each of the reference images.

[0223] Optionally, when the detection module 303 is used to detect, for each reference image, a video frame recorded while playing the reference image from the continuous video frames included in the reference video to obtain a reference image corresponding to the reference image; and to detect, from the continuous video frames included in the test video, a video frame recorded while playing the reference image to obtain a test image corresponding to the reference image, specifically:

[0224] For each video in the reference video and the test video, the first abrupt change frame when playing the first reference image is detected from the consecutive video frames contained in the video, and the video frames before the first abrupt change frame in the video are cropped to obtain the reference video and test video with the first frame aligned.

[0225] Calculate the product of the preset playback duration and the video frame rate, and use the product result as the target interval frame number; calculate the ratio of the target interval frame number to 2, and use the ratio result as the target frame number.

[0226] From the consecutive video frames contained in the reference video after the first frame is aligned, a video frame with the target number of frames is selected as the first frame reference image. Then, a video frame is selected every target interval frame number as the reference image for each frame to obtain the reference image corresponding to each reference image.

[0227] From the consecutive video frames contained in the test video after the first frame is aligned, a video frame with the target number of frames is selected as the first test image. Then, a video frame is selected every target interval frame number as the test image for each frame, so as to obtain the test image corresponding to each reference image.

[0228] Optionally, when the detection module 303 detects the first abrupt change frame when playing the first reference image from consecutive video frames contained in each of the reference video and the test video, it is specifically used to:

[0229] The minimum value between the total number of video frames in the reference video and the total number of video frames in the test video is determined, and this minimum value is taken as the target total number of frames.

[0230] From both the reference video and the test video, extract video frames representing the target total number of frames to obtain a reference video and a test video with the total number of frames aligned.

[0231] For each video in the reference video and test video with total frame number alignment, calculate the inter-frame similarity between any two adjacent video frames in the video to obtain the inter-frame similarity sequence, and use the first-order difference algorithm to calculate the first-order jump sequence of the inter-frame similarity sequence.

[0232] A preset proportion of the maximum first-order jump value in the first-order jump sequence is selected as the first threshold. The first first-order jump value in the first-order jump sequence that exceeds the first threshold is searched. The second video frame of the two video frames corresponding to the first-order jump value is taken as the first mutation frame when playing the first reference image.

[0233] Optionally, when calculating the inter-frame similarity between any two adjacent video frames in the video, the detection module 303 is specifically used for:

[0234] The inter-frame similarity between any two adjacent frames in the video can be calculated using the following formula:

[0235]

[0236] in, This is the brightness comparison function; This is a contrast comparison function; For structural comparison functions; , , These are preset coefficients; and Represents two adjacent video frames; For video frames and Inter-frame similarity;

[0237]

[0238]

[0239] in, It is a video frame The average value of the pixels; It is a video frame The average pixel value; L is the preset value; K1 is an empirical coefficient;

[0240]

[0241]

[0242] in, It is a video frame The standard deviation of pixel values; It is a video frame The standard deviation of the pixel values; K2 is an empirical coefficient;

[0243]

[0244]

[0245] in, It is a video frame and The covariance of pixel values.

[0246] Optionally, when calculating the perspective transformation parameters of the test image relative to the reference image, the calculation module 304 is specifically used for:

[0247] Select one frame of the reference image, and perform grayscale processing on the reference image and all test images to obtain a first reference grayscale image and multiple frames of the first test grayscale image;

[0248] Feature extraction is performed on the first reference grayscale image to obtain multiple first feature points on the first reference grayscale image; and feature extraction is performed on each frame of the first test grayscale image to obtain multiple second feature points on each frame of the first test grayscale image.

[0249] For each first feature point, the Hamming distance-based brute-force matching algorithm based on K-nearest neighbors is used to calculate the Hamming distance between the first feature point and each second feature point. The two smallest Hamming distances are selected from all the Hamming distances corresponding to the first feature point and are respectively used as the first Hamming distance and the second Hamming distance; the first Hamming distance is less than or equal to the second Hamming distance.

[0250] Calculate the ratio between the first Hamming distance and the second Hamming distance. If the ratio is less than a second threshold, it means that the second feature point corresponding to the first Hamming distance is unique to the first feature point, and the second feature point is used as the matching feature point of the first feature point. If the ratio is greater than or equal to the second threshold, it means that the second feature point corresponding to the first Hamming distance is not unique to the first feature point, and the first feature point has no matching feature point.

[0251] After determining whether each of the first feature points has a matching feature point, the total number of all matching feature points is counted. If the total number is higher than a third threshold, the homography transformation matrix of the first test grayscale image relative to the first reference grayscale image is calculated based on the first coordinates of the first feature point with matching feature points on the first reference grayscale image and the second coordinates of each matching feature point on its respective first test grayscale image. This homography transformation matrix is ​​then used as the perspective transformation parameter of the test image relative to the reference image.

[0252] Optionally, when the comparison module 305 compares the differences between the screen region to be detected in the reference image corresponding to each of the reference images and the screen region to be detected in the perspective-transformed test image, in order to determine the screen fault detection result based on the differences corresponding to each of the reference images, it is specifically used for:

[0253] The two reference images with the largest brightness difference are selected from all the reference images as the first reference image and the second reference image. Based on the brightness difference between the reference image corresponding to the first reference image and the reference image corresponding to the second reference image, the position area of ​​the screen to be detected on the reference image is determined, and the screen mask of the screen to be detected is obtained.

[0254] Select the reference image with the lowest brightness from all the reference images as the target reference image. For the reference image corresponding to the target reference image and the test image after perspective change, calculate the average brightness of each pixel in the reference image to obtain a first average brightness value, and calculate the average brightness of each pixel in the test image after perspective change to obtain a second average brightness value.

[0255] Based on the first average brightness value and the second average brightness value, a brightness correction coefficient is determined. The brightness correction coefficient is then used to correct the brightness of each test image after perspective change, resulting in each test image after brightness correction.

[0256] Based on the screen mask, the screen area to be detected in each of the reference images and the screen area to be detected in each of the brightness-corrected test images are determined. The difference between the screen area to be detected in the reference image corresponding to each reference image and the screen area to be detected in the brightness-corrected test image is compared, so as to determine the screen fault detection result based on the difference corresponding to each reference image.

[0257] Optionally, when the comparison module 305 selects the two reference images with the largest brightness difference from all the reference images as the first reference image and the second reference image, and determines the position region of the screen to be detected on the reference image based on the brightness difference between the reference image corresponding to the first reference image and the reference image corresponding to the second reference image, and obtains the screen mask of the screen to be detected, it is specifically used for:

[0258] Each test image after perspective transformation is processed into grayscale to obtain a second test grayscale image for each test image; the second test grayscale image contains the brightness value of each pixel;

[0259] For each of the second test grayscale images, the brightness value of each pixel located in the image edge region of the second test grayscale image is compared with a fourth threshold. If the brightness value of a pixel located in the image edge region is less than or equal to the fourth threshold, a first symbol is used to mark that pixel; if the brightness value of a pixel located in the image edge region is greater than the fourth threshold, a second symbol is used to mark that pixel; and the second symbol is used to mark each pixel located in the image center region of the second test grayscale image to obtain the mask bitmap corresponding to the second test grayscale image; wherein, the position marked with the first symbol in the mask bitmap is the position filled with black during the perspective change, and the position marked with the second symbol is not the position filled with black during the perspective change;

[0260] Based on the mask bitmaps of all the second test grayscale images, a common region is determined for each test image after perspective transformation; wherein each pixel in the common region is marked with the second symbol in each of the mask bitmaps;

[0261] Using the common area, each of the reference images and each of the perspective-transformed test images are cropped, retaining only a portion of the image in the common area, to obtain each cropped reference image and test image;

[0262] The two reference images with the largest brightness difference are selected from all the reference images as the first reference image and the second reference image, and the cropped reference image corresponding to the first reference image is used as the first reference image, and the cropped reference image corresponding to the second reference image is used as the second reference image.

[0263] For each pixel in the first reference image and the second reference image, the brightness difference at that pixel is calculated. If the brightness difference is greater than the fifth threshold, the second symbol is used to mark the pixel. If the brightness difference is less than or equal to the fifth threshold, the first symbol is used to mark the pixel, so as to obtain the screen mask of the screen to be detected composed of the first symbol and the second symbol.

[0264] Optionally, when the comparison module 305 determines a brightness correction coefficient based on the first average brightness value and the second average brightness value, and uses the brightness correction coefficient to correct the brightness of each test image after perspective change to obtain each test image after brightness correction, it is specifically used for:

[0265] Calculate the ratio of the first average brightness value to the second average brightness value, and use the ratio result as the brightness correction coefficient;

[0266] For each test image after perspective change, the brightness correction coefficient is multiplied by the pixel value of each pixel in the test image to obtain the new pixel value of each pixel after brightness correction. The new pixel value of each pixel is used as the pixel value of each pixel in the brightness-corrected test image to obtain each test image after brightness correction.

[0267] Optionally, the comparison module 305, when used to determine the screen region to be detected in each of the reference images and the screen region to be detected in each brightness-corrected test image based on the screen mask, and to compare the difference between the screen region to be detected in the reference image corresponding to each reference image and the screen region to be detected in the brightness-corrected test image, in order to determine the screen fault detection result based on the differences corresponding to each of the reference images, is specifically used for:

[0268] For each reference image and the brightness-corrected test image corresponding to the reference image, grayscale processing is performed on the reference image and the test image to obtain the second reference grayscale image and the third test grayscale image corresponding to the reference image.

[0269] Based on the screen mask, the first screen region to be detected in the second reference grayscale image and the second screen region to be detected in the third test grayscale image are determined, and the grayscale values ​​of corresponding positions in the first screen region to be detected and the second screen region to be detected are subtracted and the absolute value is taken.

[0270] If the absolute value is greater than the sixth threshold, the second symbol is used to mark the position; if the absolute value is less than or equal to the sixth threshold, the first symbol is used to mark the position, thus obtaining the fault marking map corresponding to the reference image; wherein, the area marked with the second symbol in the fault marking map represents the fault area;

[0271] The fault marker maps corresponding to all the reference images are superimposed to obtain the overall fault marker map.

[0272] Figure 4 A schematic diagram of an electronic device provided in this application embodiment includes: a processor 401, a memory 402, and a bus 403. The memory 402 stores machine-readable instructions executable by the processor 401. When the electronic device runs the above-described information processing method, the processor 401 and the memory 402 communicate through the bus 403. The processor 401 executes the machine-readable instructions to perform the steps of the method described in Embodiment 1.

[0273] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the steps described in Embodiment 1.

[0274] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the devices, electronic devices, and computer-readable storage media described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0275] In the several embodiments provided in this application, it should be understood that the disclosed methods, apparatuses, electronic devices, and computer-readable storage media can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Additionally, the displayed or discussed mutual couplings, direct couplings, or communication connections may be through some communication interfaces; indirect couplings or communication connections between devices or modules may be electrical, mechanical, or other forms.

[0276] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0277] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0278] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a processor-executable, non-volatile, computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0279] Finally, it should be noted that the above-described embodiments are merely specific implementations of this application, used to illustrate the technical solutions of this application, and not to limit them. The scope of protection of this application is not limited thereto. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features, within the scope of the technology disclosed in this application. Such modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of protection of the claims.

Claims

1. A screen fault detection method, characterized in that, include: While the screen under test is playing each reference image sequentially according to a preset playback duration and preset playback order, a reference video containing the screen under test is recorded. During the fault detection phase, as the screen under test plays each of the reference images sequentially according to the preset playback duration and the preset playback order, a test video containing the screen under test is recorded. For each of the reference images, a video frame recorded while playing the reference image is detected from the continuous video frames contained in the reference video to obtain the reference image corresponding to the reference image; and a video frame recorded while playing the reference image is detected from the continuous video frames contained in the test video to obtain the test image corresponding to the reference image. Calculate the perspective transformation parameters of the test image relative to the reference image, and use the perspective transformation parameters to perform perspective transformation on each frame of the test image respectively; The differences between the screen region to be detected in the reference image corresponding to each of the reference images and the screen region to be detected in the perspective-transformed test image are compared to determine the screen fault detection result based on the differences corresponding to each of the reference images. For each reference image, detecting a video frame recorded while playing the reference image from a series of consecutive video frames in the reference video to obtain a reference image corresponding to that reference image; and detecting a video frame recorded while playing the reference image from a series of consecutive video frames in the test video to obtain a test image corresponding to that reference image, includes: For each video in the reference video and the test video, the first abrupt change frame when playing the first reference image is detected from the consecutive video frames contained in the video, and the video frames before the first abrupt change frame in the video are cropped to obtain the reference video and test video with the first frame aligned. Calculate the product of the preset playback duration and the video frame rate, and use the product result as the target interval frame number; calculate the ratio of the target interval frame number to 2, and use the ratio result as the target frame number. From the consecutive video frames contained in the reference video after the first frame is aligned, a video frame with the target number of frames is selected as the first frame reference image. Then, a video frame is selected every target interval frame number as the reference image for each frame to obtain the reference image corresponding to each reference image. From the consecutive video frames contained in the test video after the first frame is aligned, a video frame with the target number of frames is selected as the first test image. Then, a video frame is selected every target interval frame number as the test image for each frame, so as to obtain the test image corresponding to each reference image.

2. The method according to claim 1, characterized in that, The step of detecting the first abrupt change frame when playing the first reference image from consecutive video frames contained in each of the reference video and the test video includes: The minimum value between the total number of video frames in the reference video and the total number of video frames in the test video is determined, and this minimum value is taken as the target total number of frames. From both the reference video and the test video, extract video frames representing the target total number of frames to obtain a reference video and a test video with the total number of frames aligned. For each video in the reference video and test video with total frame number alignment, calculate the inter-frame similarity between any two adjacent video frames in the video to obtain the inter-frame similarity sequence, and use the first-order difference algorithm to calculate the first-order jump sequence of the inter-frame similarity sequence. A preset proportion of the maximum first-order jump value in the first-order jump sequence is selected as the first threshold. The first first-order jump value in the first-order jump sequence that exceeds the first threshold is searched. The second video frame of the two video frames corresponding to the first-order jump value is taken as the first mutation frame when playing the first reference image.

3. The method according to claim 2, characterized in that, The calculation of the inter-frame similarity between any two adjacent video frames in the video includes: The inter-frame similarity between any two adjacent frames in the video can be calculated using the following formula: in, This is the brightness comparison function; This is a contrast comparison function; For structural comparison functions; , , These are preset coefficients; and Represents two adjacent video frames; For video frames and Inter-frame similarity; in, It is a video frame The average value of the pixels; It is a video frame The average pixel value; L is the preset value; K1 is an empirical coefficient; in, It is a video frame The standard deviation of pixel values; It is a video frame The standard deviation of the pixel values; K2 is an empirical coefficient; in, It is a video frame and The covariance of pixel values.

4. The method according to claim 1, characterized in that, The calculation of the perspective transformation parameters of the test image relative to the reference image includes: Select one frame of the reference image, and perform grayscale processing on the reference image and all test images to obtain a first reference grayscale image and multiple frames of the first test grayscale image; Feature extraction is performed on the first reference grayscale image to obtain multiple first feature points on the first reference grayscale image; and feature extraction is performed on each frame of the first test grayscale image to obtain multiple second feature points on each frame of the first test grayscale image. For each first feature point, the Hamming distance-based brute-force matching algorithm based on K-nearest neighbors is used to calculate the Hamming distance between the first feature point and each second feature point. The two smallest Hamming distances are selected from all the Hamming distances corresponding to the first feature point and are respectively used as the first Hamming distance and the second Hamming distance; the first Hamming distance is less than or equal to the second Hamming distance. Calculate the ratio between the first Hamming distance and the second Hamming distance. If the ratio is less than a second threshold, it means that the second feature point corresponding to the first Hamming distance is unique to the first feature point, and the second feature point is used as the matching feature point of the first feature point. If the ratio is greater than or equal to the second threshold, it means that the second feature point corresponding to the first Hamming distance is not unique to the first feature point, and the first feature point has no matching feature point. After determining whether each of the first feature points has a matching feature point, the total number of all matching feature points is counted. If the total number is higher than a third threshold, the homography transformation matrix of the first test grayscale image relative to the first reference grayscale image is calculated based on the first coordinates of the first feature point with matching feature points on the first reference grayscale image and the second coordinates of each matching feature point on its respective first test grayscale image. This homography transformation matrix is ​​then used as the perspective transformation parameter of the test image relative to the reference image.

5. The method according to claim 1, characterized in that, The step of comparing the differences between the screen region to be detected in the reference image corresponding to each of the reference images and the screen region to be detected in the perspective-transformed test image, in order to determine the screen fault detection result based on the differences corresponding to each of the reference images, includes: The two reference images with the largest brightness difference are selected from all the reference images as the first reference image and the second reference image. Based on the brightness difference between the reference image corresponding to the first reference image and the reference image corresponding to the second reference image, the position area of ​​the screen to be detected on the reference image is determined, and the screen mask of the screen to be detected is obtained. Select the reference image with the lowest brightness from all the reference images as the target reference image. For the reference image corresponding to the target reference image and the test image after perspective change, calculate the average brightness of each pixel in the reference image to obtain a first average brightness value, and calculate the average brightness of each pixel in the test image after perspective change to obtain a second average brightness value. Based on the first average brightness value and the second average brightness value, a brightness correction coefficient is determined. The brightness correction coefficient is then used to correct the brightness of each test image after perspective change, resulting in each test image after brightness correction. Based on the screen mask, the screen area to be detected in each of the reference images and the screen area to be detected in each of the brightness-corrected test images are determined. The difference between the screen area to be detected in the reference image corresponding to each reference image and the screen area to be detected in the brightness-corrected test image is compared, so as to determine the screen fault detection result based on the difference corresponding to each reference image.

6. The method according to claim 5, characterized in that, The step of selecting the two reference images with the largest brightness difference from all the reference images as the first reference image and the second reference image, and determining the position region of the screen to be detected on the reference image based on the brightness difference between the reference image corresponding to the first reference image and the reference image corresponding to the second reference image, to obtain the screen mask of the screen to be detected, includes: Each test image after perspective transformation is processed into grayscale to obtain a second test grayscale image for each test image; the second test grayscale image contains the brightness value of each pixel; For each of the second test grayscale images, the brightness value of each pixel located in the image edge region of the second test grayscale image is compared with a fourth threshold. If the brightness value of a pixel located in the image edge region is less than or equal to the fourth threshold, a first symbol is used to mark that pixel; if the brightness value of a pixel located in the image edge region is greater than the fourth threshold, a second symbol is used to mark that pixel; and the second symbol is used to mark each pixel located in the image center region of the second test grayscale image to obtain the mask bitmap corresponding to the second test grayscale image; wherein, the position marked with the first symbol in the mask bitmap is the position filled with black during the perspective change, and the position marked with the second symbol is not the position filled with black during the perspective change; Based on the mask bitmaps of all the second test grayscale images, a common region is determined for each test image after perspective transformation; wherein each pixel in the common region is marked with the second symbol in each of the mask bitmaps; Using the common area, each of the reference images and each of the perspective-transformed test images are cropped, retaining only a portion of the image in the common area, to obtain each cropped reference image and test image; The two reference images with the largest brightness difference are selected from all the reference images as the first reference image and the second reference image, and the cropped reference image corresponding to the first reference image is used as the first reference image, and the cropped reference image corresponding to the second reference image is used as the second reference image. For each pixel in the first reference image and the second reference image, the brightness difference at that pixel is calculated. If the brightness difference is greater than the fifth threshold, the second symbol is used to mark the pixel. If the brightness difference is less than or equal to the fifth threshold, the first symbol is used to mark the pixel, so as to obtain the screen mask of the screen to be detected composed of the first symbol and the second symbol.

7. The method according to claim 5, characterized in that, The step involves determining a brightness correction coefficient based on the first average brightness value and the second average brightness value, and then using the brightness correction coefficient to correct the brightness of each test image after perspective change, resulting in brightness-corrected test images, including: Calculate the ratio of the first average brightness value to the second average brightness value, and use the ratio result as the brightness correction coefficient; For each test image after perspective change, the brightness correction coefficient is multiplied by the pixel value of each pixel in the test image to obtain the new pixel value of each pixel after brightness correction. The new pixel value of each pixel is used as the pixel value of each pixel in the brightness-corrected test image to obtain each test image after brightness correction.

8. The method according to claim 5, characterized in that, The step of determining the screen region to be detected in each reference image and the screen region to be detected in each brightness-corrected test image based on the screen mask, and comparing the difference between the screen region to be detected in the reference image corresponding to each reference image and the screen region to be detected in the brightness-corrected test image, to determine the screen fault detection result based on the difference corresponding to each reference image, includes: For each reference image and the brightness-corrected test image corresponding to the reference image, grayscale processing is performed on the reference image and the test image to obtain the second reference grayscale image and the third test grayscale image corresponding to the reference image. Based on the screen mask, the first screen region to be detected in the second reference grayscale image and the second screen region to be detected in the third test grayscale image are determined, and the grayscale values ​​of corresponding positions in the first screen region to be detected and the second screen region to be detected are subtracted and the absolute value is taken. If the absolute value is greater than the sixth threshold, the second symbol is used to mark the position; if the absolute value is less than or equal to the sixth threshold, the first symbol is used to mark the position, thus obtaining the fault marking map corresponding to the reference image; wherein, the area marked with the second symbol in the fault marking map represents the fault area; The fault marker maps corresponding to all the reference images are superimposed to obtain the overall fault marker map.

9. A screen fault detection device, characterized in that, include: The first recording module is used to record a reference video containing the screen under test while each reference image is played sequentially according to a preset playback duration and preset playback order on the screen under test that has not experienced a fault. The second recording module is used to record a test video containing the screen under test during the fault detection phase, in which the screen under test plays each of the reference images in sequence according to the preset playback duration and the preset playback order. The detection module is configured to, for each reference image, detect one video frame recorded while playing the reference image from the continuous video frames contained in the reference video, and obtain the reference image corresponding to the reference image; and detect one video frame recorded while playing the reference image from the continuous video frames contained in the test video, and obtain the test image corresponding to the reference image. The calculation module is used to calculate the perspective transformation parameters of the test image relative to the reference image, so as to perform perspective transformation on each frame of the test image using the perspective transformation parameters; The comparison module is used to compare the differences between the screen region to be detected in the reference image corresponding to each of the reference images and the screen region to be detected in the perspective-transformed test image, so as to determine the screen fault detection result based on the differences corresponding to each of the reference images. The detection module, when used for each reference image to detect, from the continuous video frames contained in the reference video, one video frame recorded while playing the reference image, to obtain the reference image corresponding to that reference image; and when used for detecting, from the continuous video frames contained in the test video, one video frame recorded while playing the reference image, to obtain the test image corresponding to that reference image, is specifically used for: For each video in the reference video and the test video, the first abrupt change frame when playing the first reference image is detected from the consecutive video frames contained in the video, and the video frames before the first abrupt change frame in the video are cropped to obtain the reference video and test video with the first frame aligned. Calculate the product of the preset playback duration and the video frame rate, and use the product result as the target interval frame number; calculate the ratio of the target interval frame number to 2, and use the ratio result as the target frame number. From the consecutive video frames contained in the reference video after the first frame is aligned, a video frame with the target number of frames is selected as the first frame reference image. Then, a video frame is selected every target interval frame number as the reference image for each frame to obtain the reference image corresponding to each reference image. From the consecutive video frames contained in the test video after the first frame is aligned, a video frame with the target number of frames is selected as the first test image. Then, a video frame is selected every target interval frame number as the test image for each frame, so as to obtain the test image corresponding to each reference image.

10. An electronic device, characterized in that, include: The device includes a processor, a memory, and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is in operation, the processor communicates with the memory via the bus, and the machine-readable instructions, when executed by the processor, perform the steps of the method as described in any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, performs the steps of the method as described in any one of claims 1 to 8.

Images