Display picture detection method and device, equipment, medium and product

Abstract

The invention discloses a display picture detection method and device, equipment, a medium and a product, relates to the technical field of image recognition, and utilizes an image classification model to analyze multiple frames of images so as to determine an initial classification result. And respectively extracting an edge feature map, a diagonal gradient feature map, an optical flow feature map and a difference feature map from the multiple frames of images. And based on the edge relative displacement between the edge feature maps of the adjacent frames and the gradient relative displacement between the diagonal gradient feature maps of the adjacent frames, determining a first identification result whether the display picture of the image flickers or not. And according to the initial classification result, the optical flow structure similarity corresponding to the multiple frames of optical flow feature maps and the difference structure similarity corresponding to the multiple frames of difference feature maps, determining a second identification result whether the display picture of the image is lagged or not. Different types of feature maps in multiple frames of images are analyzed, so that the abnormal problem of the display picture can be timely and accurately detected, and the maintainability of the system is improved.

Description

Technical Field

[0001] This application relates to the field of image recognition technology, and in particular to a method, apparatus, device, medium and product for detecting display images. Background Technology

[0002] The local display on a general-purpose server needs to be processed and output to the monitor via the integrated graphics card within the Baseboard Management Controller (BMC). The video memory used by the integrated graphics card typically uses a portion of the BMC's system memory. However, in a BMC system, system memory is often used not only for the host's video memory but also for the BMC's ARM CPU's computation, networking, and operating system kernel operation. Therefore, under high load on the BMC system, abnormal video memory access can occur.

[0003] Currently, the common approach to display anomalies is to record the monitor screen and use display anomaly detection algorithms to identify issues such as screen tearing or blackouts. However, this method of capturing the display screen using recording equipment is only a passive detection method, and the detection accuracy often depends on the performance of the capturing device. If the capturing device is of poor performance, the recognition accuracy will be poor, easily leading to identification errors.

[0004] It is evident that how to detect abnormalities in the display screen in a timely and accurate manner is a problem that needs to be solved by those skilled in the art. Summary of the Invention

[0005] This application provides a method, apparatus, device, medium, and product for detecting display images, in order to at least solve the problem of poor recognition accuracy of display images in related technologies, which easily leads to recognition errors.

[0006] This application provides a method for detecting a display screen, including: An image classification model is used to analyze multiple frames of images to determine the initial classification results; the initial classification results include distorted screen, black screen, and normal display. Edge feature maps, diagonal gradient feature maps, optical flow feature maps, and difference feature maps are extracted from multiple frames of images, respectively. The first recognition result of the image is determined based on the relative displacement of edges between edge feature maps of adjacent frames and the relative displacement of gradients between diagonal gradient feature maps of adjacent frames; wherein, the first recognition result includes screen flickering. Based on the initial classification results, the optical flow structure similarity corresponding to the multi-frame optical flow feature maps, and the difference structure similarity corresponding to the multi-frame difference feature maps, the second recognition result of the image is determined; wherein, the second recognition result includes display screen stuttering.

[0007] This application also provides a detection device for displaying images, including an analysis unit, an extraction unit, a first determination unit, and a second determination unit; The analysis unit is used to analyze multiple frames of images using an image classification model to determine the initial classification results; the initial classification results include distorted screen, black screen, and normal display. The extraction unit is used to extract edge feature maps, diagonal gradient feature maps, optical flow feature maps, and difference feature maps from multiple frames of images, respectively. The first determining unit is used to determine a first recognition result of the image based on the relative edge displacement between edge feature maps of adjacent frames and the relative gradient displacement between diagonal gradient feature maps of adjacent frames; wherein the first recognition result includes screen flickering. The second determining unit is used to determine the second recognition result of the image based on the initial classification result, the optical flow structure similarity corresponding to the multi-frame optical flow feature maps, and the difference structure similarity corresponding to the multi-frame difference feature maps; wherein, the second recognition result includes display screen stuttering.

[0008] This application also provides an electronic device, including: a memory for storing a computer program; and a processor for executing the computer program to implement the steps of any of the above-described display screen detection methods.

[0009] This application also provides a computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, it implements the steps of any of the above-described display screen detection methods.

[0010] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of any of the above-described display screen detection methods.

[0011] This application utilizes an image classification model to analyze multiple frames of images to determine initial classification results, including screen flickering, black screen, and normal display. Different types of features reflect different image characteristics. To accurately and comprehensively detect display anomalies, edge feature maps, diagonal gradient feature maps, optical flow feature maps, and difference feature maps can be extracted from multiple frames of images. Based on the relative edge displacement between edge feature maps of adjacent frames and the relative gradient displacement between diagonal gradient feature maps of adjacent frames, a first recognition result is determined, including screen flickering. Based on the initial classification result, the optical flow structure similarity corresponding to the optical flow feature maps of multiple frames, and the difference structure similarity corresponding to the difference feature maps of multiple frames, a second recognition result is determined, including screen stuttering. In this technical solution, based on the initial classification result determined by the image classification model, by analyzing different types of feature maps in the multiple frames of images currently displayed on the screen, abnormal display problems can be detected in a timely and accurate manner during system operation, improving system maintainability. Attached Figure Description

[0012] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 A flowchart illustrating a method for detecting a display screen according to an embodiment of this application; Figure 2 A flowchart illustrating a method for identifying screen flickering, as provided in this application embodiment; Figure 3 A flowchart illustrating a method for identifying display screen stuttering provided in this application embodiment; Figure 4 A framework diagram of a display screen detection system provided in an embodiment of this application; Figure 5 This is a schematic diagram of the structure of a display screen detection device provided in an embodiment of this application. Detailed Implementation

[0014] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.

[0015] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., in this application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0016] The local display on a general-purpose server needs to be processed by the integrated graphics card within the BMC before being output to the monitor. The integrated graphics card typically uses the BMC's system memory for video memory. Therefore, under high load on the BMC system, access to video memory can become abnormal, such as read-memory non-response errors or write-memory non-response errors. This results in the display controller being unable to obtain valid data, and the host data failing to update the video memory in a timely manner, causing display abnormalities on the Video Graphics Array (VGA) interface and affecting the screen's visual quality.

[0017] When read / write errors occur in the BMC system, they often manifest as a sudden surge in memory bandwidth usage, and the timing of these errors is unpredictable. Because the host system's display updates are not correlated with the BMC system's internal calculations, the timing of these anomalies is highly random. Traditional methods using recording devices to capture the display screen can only passively detect the display and cannot identify anomalies in real time. If the recording device has poor performance, the recognition accuracy will also be poor, easily leading to recognition errors. Alternatively, an additional piece of video memory can be added for video memory data verification and real-time analysis of video memory content. Adding extra video memory undoubtedly increases video memory bandwidth consumption, which can exacerbate problems under high load, and the additional dedicated video memory also increases costs.

[0018] Therefore, the embodiments of this application provide a method, apparatus, device, medium and product for detecting display screens, which identifies abnormal images based on artificial intelligence (AI) convolutional neural networks. By sharing convolutional neural network feature maps, it can automatically identify single-frame abnormalities and continuous frame abnormalities, and can detect abnormalities in the display screen in real time.

[0019] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0020] Figure 1A flowchart of a method for detecting a display screen provided in this application embodiment includes: S101: Analyze multiple frames of images using an image classification model to determine the initial classification result.

[0021] The initial classification results can include screen distortion, black screen, and normal display.

[0022] Image classification models can employ hardware-accelerated shared neural networks. By inputting multiple frames of images into this shared neural network for classification, the classification results based on pre-trained parameters can be obtained, confirming whether each frame contains classification results such as distorted images, black images, or normal images.

[0023] The image classification model is only 10k in size, making it easy to integrate and implement in embedded devices.

[0024] The hardware-accelerated shared convolutional neural network architecture consists of three convolutional layers, one pooling layer, one fully connected layer, and an output layer.

[0025] For the three convolutional layers, the first convolutional layer uses a network structure with fixed convolutional kernels, specifically a 16*3*3*3 kernel, followed by batch normalization (BN) and the first type of activation function (ReLU6). The dimension of the input data changes from w*h*3 to (w / 2)*(h / 2)*16; where w represents the width of the data, h represents the height of the data, and the last number in the dimension represents the number of elements. The convolution through the fixed kernel in the first convolutional layer yields 16 (w / 2)*(h / 2) feature maps.

[0026] The second convolutional layer is formed by adding a depthwise separable convolutional kernel, BN, and a second cumulative activation function (Harmwish). The data dimension changes from (w / 2)*(h / 2)*(16) to (w / 4)*(h / 4)*(32).

[0027] The third convolutional layer is formed by adding a depthwise separable convolutional kernel, BN, and a second type of activation function. The data dimension changes from (w / 4)*(h / 4)*(32) to (w / 4)*(h / 4)*(64).

[0028] Finally, average pooling is applied to obtain a 1*1*64 dimension, which is then passed through a fully connected layer to obtain the three-class classification output. The three-class classification output is then output through the output layer.

[0029] S102: Extract edge feature maps, diagonal gradient feature maps, optical flow feature maps, and difference feature maps from multiple frames of images.

[0030] The first layer of the shared neural network has 16 fixed convolutional kernels of 3x3x3 dimensions, including four 3x3x3 image edge detection operators (Sobel), four optical flow detection kernels, four diagonal gradient kernels, and four difference kernels. The image edge detection operators are used to detect edges in the three color (RGB) channels.

[0031] In practical applications, edge feature maps can be extracted from multiple frames of images using the image edge detection operator of the image classification model; diagonal gradient feature maps can be extracted from multiple frames of images using the diagonal gradient kernel of the image classification model; optical flow feature maps can be extracted from multiple frames of images using the optical flow detection kernel of the image classification model; and difference feature maps can be extracted from multiple frames of images using the difference kernel of the image classification model.

[0032] When analyzing multiple frames of images using an image classification model, the four types of feature maps extracted by the fixed convolution kernel of the first layer of the image classification model can be stored in a set buffer.

[0033] S103: Based on the relative edge displacement between edge feature maps of adjacent frames and the relative gradient displacement between diagonal gradient feature maps of adjacent frames, the first recognition result of the image is determined.

[0034] The first identification result may include screen flickering.

[0035] Considering that in practical applications, the changes in three consecutive image frames will involve a gradual process, and this gradual process will affect the human eye's perception, we use A, A1, and A2 to represent the three image frames for ease of description. A and A1 are adjacent frames, and A1 and A2 are adjacent frames. When there are changes in the image, the motion direction of key image parts between two adjacent frames is consistent. For example, the motion change direction between adjacent frames A, A1 and A1, A2 will be consistent. If the motion direction between adjacent frames A, A1 and A1, A2 is inconsistent, and this phenomenon occurs multiple times, it indicates that the display screen is flickering. The human eye will perceive the screen as jumping and flickering. Therefore, by statistically analyzing whether the motion direction of key image parts is consistent over a period of time, we can detect abnormal display problems such as image flickering and jumping to a certain extent.

[0036] Edge feature maps and diagonal gradient feature maps can more intuitively reflect the direction of motion transformation in an image. The direction of motion transformation can be represented by relative displacement. Therefore, in this embodiment, by calculating the relative displacement between edge feature maps of adjacent frames and the relative gradient displacement between diagonal gradient feature maps of adjacent frames, it is possible to identify whether the motion directions of adjacent frames are consistent. To distinguish the relative displacements corresponding to different types of feature maps, the relative displacement between edge feature maps can be called edge relative displacement, and the relative displacement between diagonal gradient feature maps can be called gradient relative displacement.

[0037] If the motion directions of multiple adjacent frames are inconsistent, it indicates that the display screen is flickering, and the first recognition result can be "display screen flickering". If the motion directions of multiple adjacent frames are inconsistent, it indicates that the display screen is not flickering, and the first recognition result can be "display screen not flickering".

[0038] S104: Based on the initial classification results, the optical flow structure similarity corresponding to the multi-frame optical flow feature maps, and the difference structure similarity corresponding to the multi-frame difference feature maps, the second recognition result of the image is determined.

[0039] The second identification result may include display screen lag.

[0040] Typically, the host updates a new frame of video within one frame's time. One frame's time can include vertical sync time (vsync time) and field display enable time (ver enable time).

[0041] Within a single frame, the VGA controller reads the video memory and outputs the display. Normally, the CPU updates the next frame within the vsync time. If the next frame isn't updated within the vsync time, it will be updated within the ver enable time of that frame. Otherwise, screen tearing will occur, where part of the video memory contains the previous frame and part contains the current frame. Screen tearing happens when the read / write pointers overlap within a single frame.

[0042] Taking three consecutive frames as an example, let p1, p2, and p3 represent the three consecutive frames. Calculate D1 = p2 - p1, D2 = p3 – p2, and D3 = p3 - p1. If D1, D2, and D3 are all greater than zero, it indicates that there are changes in the image. If D3 = D1 + D2, then the update speed of the three consecutive frames is considered too slow.

[0043] In practical applications, optical flow feature maps and difference feature maps can more intuitively reflect the changes in an image. When the display is playing normally, the structural similarity values ​​of consecutive frames are often small. A large structural similarity value indicates that the image is changing too slowly.

[0044] Therefore, in this embodiment, by analyzing the structural similarity corresponding to multiple frames of optical flow feature maps and the structural similarity corresponding to multiple frames of difference feature maps, and combining this with the initial classification results to determine whether there is a screen flickering issue, it can be determined whether there is a display stuttering problem. For ease of distinction, the structural similarity corresponding to multiple frames of optical flow feature maps can be called optical flow structural similarity, and the structural similarity corresponding to multiple frames of difference feature maps can be called difference structural similarity.

[0045] When the structural similarity is greater than the set threshold, it indicates that the screen changes too slowly. Based on this, if the initial classification result is a distorted screen, it indicates that the display screen is lagging.

[0046] As can be seen from the above technical solution, an image classification model is used to analyze multiple frames of images to determine the initial classification result, which includes screen flickering, black screen, and normal display. Different types of features can reflect different characteristics of the image. To accurately and comprehensively detect anomalies in the display, edge feature maps, diagonal gradient feature maps, optical flow feature maps, and difference feature maps can be extracted from multiple frames of images. Based on the relative edge displacement between the edge feature maps of adjacent frames and the relative gradient displacement between the diagonal gradient feature maps of adjacent frames, the first recognition result of the image is determined, which includes screen flickering. Based on the initial classification result, the optical flow structure similarity corresponding to the optical flow feature maps of multiple frames, and the difference structure similarity corresponding to the difference feature maps of multiple frames, the second recognition result of the image is determined, which includes screen stuttering. In this technical solution, based on the initial classification result determined by the image classification model, by analyzing different types of feature maps in the multiple frames of images currently displayed on the monitor, abnormal problems in the display can be detected in a timely and accurate manner during system operation, improving the maintainability of the system.

[0047] Figure 2 A flowchart of a method for identifying screen flickering provided in this application embodiment, the method including: S201: Traverse the edge feature maps of adjacent frames according to the principle of minimum difference to determine the best matching pixel block pair of the edge.

[0048] In this embodiment of the application, in order to determine the relative displacement more accurately, it is necessary to first determine the best matching pixel block pair between adjacent frames. The distance between the best matching pixel block pair can more accurately reflect the direction of motion change of the image.

[0049] The fixed convolutional kernel in the first layer contains four image edge detection operators, so four edge feature maps can be extracted for each frame of the image. To make it easier to distinguish the four edge feature maps extracted from a frame of the image, they can be referred to as the first edge feature map, the second edge feature map, the third edge feature map, and the fourth edge feature map, respectively.

[0050] In the specific implementation, the first edge feature map of the first frame image and the first edge feature map of the second frame image can be traversed according to the first window to determine the first best-matching pixel block pair with the smallest absolute difference value. The second edge feature map within the first best-matching pixel block pair can be traversed according to the second window to determine the second best-matching pixel block pair with the smallest absolute difference value. The third edge feature map within the second best-matching pixel block pair can be traversed according to the third window to determine the third best-matching pixel block pair with the smallest absolute difference value. The fourth edge feature map within the third best-matching pixel block pair can be traversed according to the fourth window to determine the fourth best-matching pixel block pair with the smallest absolute difference value.

[0051] For each adjacent edge feature map, the implementation method for determining the best matching pixel block pair based on window traversal of the edge feature maps is similar. In this embodiment, the process of determining the best matching pixel block pair of the first edge is described as an example.

[0052] In a specific implementation, a first window image can be selected from the first edge feature map of the first frame image according to the first window; based on the set search range, the window is moved on the first edge feature map of the second frame image to obtain multiple second window images; based on the deviation between all pixels of the first window image and all pixels of each second window image, multiple absolute difference values ​​are determined; the first window image and the second window image with the smallest absolute difference value are taken as the first edge best matching pixel block pair.

[0053] The calculation of the absolute difference value involves performing a difference operation on the grayscale values ​​of all pixels in the first window image and the grayscale values ​​of all pixels in the target second window image to obtain multiple difference values; where the target second window image is any one of the second window images. The absolute values ​​of all differences are summed to obtain the absolute difference value corresponding to the target second window.

[0054] Taking two image frames as an example, to determine the motion transformation direction of two adjacent frames, in practical applications, we can read the four edge feature maps generated by the Sobel operator convolution in the first layer of the network from the first frame image, denoted as m11, m12, m13, and m14. These four edge feature maps can be stored in an internal buffer. Then, we extract the four edge feature maps from the consecutive second frame image, denoted as m21, m22, m23, and m24.

[0055] After obtaining the edge feature maps of adjacent frames, the best matching pixel block pairs between the two adjacent frame edge feature maps can be determined by the global successive average block matching algorithm (SAD). The basic idea of ​​SAD is to calculate the sum of the absolute values ​​of the differences, that is, to sum the absolute values ​​of the differences between the corresponding values ​​of each pixel block.

[0056] The SAD algorithm is expressed by the formula: SAD(u,v)=Sum{|Left(u,v)-Right(u,v)|}; Here, (u, v) represents the pixels of a pixel block. Left represents the left image, and Right represents the right image. The edge feature map of the first frame can be used as the left image, and the edge feature map of the second frame can be used as the right image.

[0057] In practical applications, two frames of images can be input, scanned sequentially, and an anchor point can be selected. The basic process of the SAD algorithm is as follows: (1) Construct a small window, similar to a convolution kernel; (2) Cover the left image with the window and select all pixels within the covered area; (3) Cover the right image with the same window and select the pixels within the covered area; (4) Subtract the right covered area from the left covered area and calculate the sum of the absolute values ​​of the grayscale differences of all pixels; (5) Move the window of the right image and repeat the processing of (3)-(4). There is a search range during the movement process, and the process will exit if it exceeds this range; (6) Find the window with the smallest SAD value within this range, that is, find the best matching pixel block of the anchor point of the left image.

[0058] Taking the four edge feature maps of the first frame and the four edge feature maps of the second frame as examples, we can traverse m11 and m21 in 32*32 pixel blocks. When a pixel block b11 and b21 are matched, it is considered that the best matching pixel block pair b11 and b21 has been found in m11. Then, we continue to match m12 and m22 within this pixel block b11 in 16*16 blocks to find the best matching pixel block pair b12 and b22. Then, we continue to match m13 and m23 in 8*8 blocks to find the best matching pixel block pair b13 and b23. Finally, we match m14 and m24 in 4*4 blocks to find the best matching pixel block pair b14 and b24.

[0059] S202: Based on the pixels of the best-matching pixel block of the edge in the previous frame and the pixels of the best-matching pixel block of the edge in the current frame, determine the relative displacement of the edges between the edge feature maps of adjacent frames.

[0060] In this embodiment, the best-matching pixel blocks corresponding to the edge feature maps can be symmetrically paired as edge best-matching pixel block pairs. The relative displacement between the edge feature maps of adjacent frames is called the edge relative displacement.

[0061] After determining the best-matching pixel block pair, the difference between the pixel value of the best-matching pixel block in the previous frame and the pixel value of the best-matching pixel block in the current frame can be used as the relative displacement of the edge.

[0062] Based on the example of S201 above, the top-left vertex A(Xa,Yb) of the 4*4 b14 pixel block can be selected as the pixel point of the best-matching pixel block of the previous frame, and the top-left vertex B(Xb,Yb) of the b24 pixel block can be selected as the pixel point of the best-matching pixel block of the current frame. Then, the relative displacements Ta(|Xa-Xb|,|Ya-Yb|) of the edges are calculated, recorded, and stored in the internal buffer.

[0063] S203: Traverse the diagonal gradient feature maps of adjacent frames according to the principle of minimum difference to determine the best matching pixel block pair of diagonal gradient.

[0064] The fixed convolutional kernel in the first layer contains four diagonal gradient kernels, so for a single frame of image, four diagonal gradient feature maps can be extracted. For ease of distinction, these four diagonal gradient feature maps can be referred to as the first diagonal gradient feature map, the second diagonal gradient feature map, the third diagonal gradient feature map, and the fourth diagonal gradient feature map, respectively.

[0065] In the specific implementation, the first diagonal gradient feature maps of the first frame and the second frame can be traversed using the fifth window to determine the best-matching pixel block pair with the smallest absolute difference value. The second diagonal gradient feature maps within the best-matching pixel block pair with the first diagonal gradient can be traversed using the sixth window to determine the best-matching pixel block pair with the smallest absolute difference value. The third diagonal gradient feature maps within the best-matching pixel block pair with the second diagonal gradient can be traversed using the seventh window to determine the best-matching pixel block pair with the smallest absolute difference value. The fourth diagonal gradient feature maps within the best-matching pixel block pair with the third diagonal gradient can be traversed using the eighth window to determine the best-matching pixel block pair with the smallest absolute difference value.

[0066] Taking the determination of the best-matching pixel block pair for the first diagonal gradient as an example, a fifth window image can be selected from the first diagonal gradient feature map of the first frame image. Based on the set search range, the window is moved on the first diagonal gradient feature map of the second frame image to obtain multiple sixth window images. Multiple absolute difference values ​​are determined based on the deviations between all pixels in the fifth window image and all pixels in each of the sixth window images. The fifth window image and the sixth window image with the smallest absolute difference value are selected as the best-matching pixel block pair for the first diagonal gradient.

[0067] The calculation of the absolute difference value involves performing a difference operation on the grayscale values ​​of all pixels in the fifth window image and the grayscale values ​​of all pixels in the target sixth window image to obtain multiple difference values; where the target sixth window image is any one of the six sixth window images. The absolute values ​​of all differences are summed to obtain the absolute difference value corresponding to the target sixth window.

[0068] The principle behind determining the best-matching pixel block pairs for diagonal gradients is the same as that for determining the best-matching pixel block pairs for edges. For specific implementation details, please refer to the description in S201, which will not be repeated here.

[0069] S204: Based on the pixels of the best-matching pixel block of the diagonal gradient in the previous frame and the pixels of the best-matching pixel block of the diagonal gradient in the current frame, determine the relative gradient displacement between the diagonal gradient feature maps of adjacent frames.

[0070] In this embodiment, the best-matching pixel blocks corresponding to the diagonal gradient feature maps can be symmetrically paired as diagonal gradient best-matching pixel block pairs. The relative displacement between the diagonal gradient feature maps of adjacent frames is called the gradient relative displacement.

[0071] The principle behind calculating gradient relative displacement is the same as that for calculating edge relative displacement. For specific implementation details, please refer to the description in S203, which will not be repeated here.

[0072] S205: Determine the first recognition result of the image based on the angle between the relative displacement of the edges of multiple adjacent frames and the relative displacement of the gradients of multiple adjacent frames.

[0073] Let Ta represent the relative displacement at the edge and Tb represent the relative displacement at the gradient. The relative displacement at the edge and the relative displacement at the gradient can be weighted and summed to obtain the motion transformation direction vector, i.e., T = 0.7 * Ta + 0.3 Tb, where T represents the motion transformation direction vector.

[0074] In this embodiment, the relative edge displacement of the first adjacent frame and the relative gradient displacement of the first adjacent frame are weighted and summed to obtain the first motion transformation direction vector. The relative edge displacement of the second adjacent frame and the relative gradient displacement of the second adjacent frame are weighted and summed to obtain the second motion transformation direction vector. If the angle between the first motion transformation direction vector and the second motion transformation direction vector is greater than a set vector threshold, a first recognition result of screen flickering is output.

[0075] In practical implementation, the included angle can be calculated using the following formula: ; Where T0 represents the first motion transformation direction vector, T1 represents the second motion transformation direction vector, and θ represents the angle between the first motion transformation direction vector and the second motion transformation direction vector.

[0076] The vector threshold can be set to 135°. If θ > 135°, flickering behavior is considered to have occurred.

[0077] To reduce false positives, before outputting the first identification result of screen flickering, the process includes: incrementing the flickering behavior count by one each time the angle between the first motion change direction vector and the second motion change direction vector exceeds a set vector threshold; resetting the flickering behavior count to zero if the count falls below the set threshold within a set time; and executing the step of outputting the first identification result of screen flickering if the count is greater than or equal to the set threshold within a set time.

[0078] In practical applications, a counter cnt can be set, and a behavior threshold h can be set. The cnt count is incremented by 1 for each occurrence of flickering behavior. If the number of cnt counts is greater than the threshold h within a set 1s=60 frames, it is considered that an abnormal behavior has occurred. If the number of cnt counts is less than the threshold within a period of 1s=60 frames, cnt is reset to zero.

[0079] In this embodiment, the best matching pixel block pairs of the edge feature maps of adjacent frames and the best matching pixel block pairs of the diagonal gradient feature maps of adjacent frames are determined based on the principle of minimum difference. The relative displacement is then confirmed based on the best matching pixel block pairs. The relative displacement reflects the direction of motion between adjacent frame images. By identifying whether there is a jump in the direction of motion, the presence of flickering behavior in the image can be accurately determined.

[0080] Figure 3 A flowchart of a method for identifying display screen stuttering provided in this application embodiment, the method including: S301: Normalize the optical flow frame difference map corresponding to the optical flow feature map of multiple consecutive frames to obtain multiple normalized optical flow frame difference maps.

[0081] The fixed convolutional kernel in the first layer contains four optical flow detection kernels. Therefore, for a single frame of image, four optical flow feature maps can be extracted, which can be referred to as the first optical flow feature map, the second optical flow feature map, the third optical flow feature map, and the fourth optical flow feature map, respectively.

[0082] In practical applications, multiple first optical flow feature maps of consecutive frames of images can be analyzed using the image frame difference method to determine multiple first optical flow frame difference maps. The pixel values ​​of each of these first optical flow frame difference maps are subtracted from the average pixel value of all first optical flow frame difference maps to obtain multiple normalized first optical flow frame difference maps. Similarly, multiple second optical flow feature maps of consecutive frames of images can be analyzed using the image frame difference method to determine multiple second optical flow frame difference maps; the pixel values ​​of each second optical flow frame difference map are then subtracted from the average pixel value of all second optical flow frame difference maps to obtain multiple normalized second optical flow frame difference maps. Finally, multiple third optical flow feature maps of consecutive frames of images can be analyzed using the image frame difference method to determine multiple third optical flow frame difference maps; the pixel values ​​of each third optical flow frame difference map are then subtracted from the average pixel value of all third optical flow frame difference maps to obtain multiple normalized third optical flow frame difference maps. Based on the image frame difference method, multiple fourth optical flow feature maps of consecutive frames are analyzed to determine multiple fourth optical flow frame difference maps. The pixel values ​​of multiple fourth optical flow frame difference maps are subtracted from the pixel average value of all fourth optical flow frame difference maps to obtain multiple normalized fourth optical flow frame difference maps.

[0083] For each frame of image, four optical flow feature maps and four difference feature maps can be extracted and stored in an internal buffer for use in detecting stuttering changes in the next frame. For ease of description, M5, M6, M7, and M8 can be used to represent the four optical flow feature maps, and M13, M14, M15, and M16 can be used to represent the four difference feature maps.

[0084] To determine the frame difference maps, taking multiple first optical flow feature maps (M5) as an example, when the first frame arrives, extract the four optical flow feature maps and four difference feature maps of the current frame and store them in an internal buffer. When the second frame arrives, extract the four optical flow feature maps and four difference feature maps and store them in the internal buffer. When the third frame arrives, extract the four optical flow feature maps and four difference feature maps. Calculate the frame difference map between the first optical flow feature maps of the first and third frames, denoted as d1_13. Then calculate the frame difference map between the first and second optical flow feature maps, denoted as d1_12. Calculate the frame difference map between the first and third optical flow feature maps, denoted as d1_23. And so on, the frame difference maps for the other three optical flow feature maps can be calculated. This will result in 4*3=12 frame difference maps.

[0085] For each set of optical flow feature maps, a normalization operation is performed on the corresponding frame difference map. The normalization operation can be achieved by subtracting the average value from the pixel value di within each frame difference map, i.e. ; Where m*n is the total number of pixels, and dinew represents the normalized optical flow frame difference map. For pixels whose result is less than 0, assign a value of 0.

[0086] S302: The structural similarity algorithm is used to process multiple normalized optical flow frame difference maps to obtain the optical flow structural similarity.

[0087] In practical applications, a structural similarity algorithm can be used to process multiple normalized first optical flow frame difference maps to obtain a first structural similarity; a structural similarity algorithm can be used to process multiple normalized second optical flow frame difference maps to obtain a second structural similarity; a structural similarity algorithm can be used to process multiple normalized third optical flow frame difference maps to obtain a third structural similarity; and a structural similarity algorithm can be used to process multiple normalized fourth optical flow frame difference maps to obtain a fourth structural similarity.

[0088] The average of the first, second, third, and fourth structural similarities is taken as the optical flow structural similarity.

[0089] Based on the above example, after normalizing d1_13, d1_12, and d1_23, structural similarity (SSIM) can be calculated from these three normalized optical flow frame difference maps. Similarly, the structural similarity of the other three optical flow feature maps, M6, M7, and M8, can be calculated. The average of the four structural similarities is then used as the final structural similarity of the optical flow.

[0090] S303: Normalize the differential frame difference maps corresponding to the differential feature maps of multiple consecutive frames of images to obtain multiple normalized differential frame difference maps.

[0091] The fixed convolutional kernel in the first layer contains four difference kernels. Therefore, for a single frame of image, four difference feature maps can be extracted, which can be called the first difference feature map, the second difference feature map, the third difference feature map, and the fourth difference feature map, respectively.

[0092] In practical applications, multiple first difference feature maps of consecutive frames of images can be analyzed using the image frame difference method to determine multiple first difference frame difference maps. The pixel values ​​of each of these first difference frame difference maps are then subtracted from the average pixel value of all first difference frame difference maps to obtain multiple normalized first difference frame difference maps. Similarly, multiple second difference feature maps of consecutive frames of images can be analyzed using the image frame difference method to determine multiple second difference frame difference maps. The pixel values ​​of each of these second difference frame difference maps are then subtracted from the average pixel value of all second difference frame difference maps to obtain multiple normalized second difference frame difference maps. Finally, multiple third difference feature maps of consecutive frames of images can be analyzed using the image frame difference method to determine multiple third difference frame difference maps. The pixel values ​​of each of these third difference frame difference maps are then subtracted from the average pixel value of all third difference frame difference maps to obtain multiple normalized third difference frame difference maps. Based on the image frame difference method, multiple fourth difference feature maps of consecutive frames of images are analyzed to determine multiple fourth difference frame difference maps. The pixel values ​​of multiple fourth difference frame difference maps are subtracted from the pixel average of all fourth difference frame difference maps to obtain multiple normalized fourth difference frame difference maps.

[0093] The implementation principle for calculating the normalized differential frame difference map is the same as that for calculating the normalized optical flow frame difference map. For specific implementation details, please refer to the description in S301, which will not be repeated here.

[0094] Similarly, the normalized differential frame difference maps corresponding to the four differential feature maps M13, M14, M15, and M16 can be calculated using the same method.

[0095] S304: Use a structural similarity algorithm to process multiple normalized differential frame difference maps to obtain differential structural similarity.

[0096] In practical applications, structural similarity algorithms can be used to process multiple normalized first-difference frame difference maps to obtain the first difference structural similarity. The same algorithm is then used to process multiple normalized second-difference frame difference maps to obtain the second difference structural similarity. The same algorithm is used to process multiple normalized third-difference frame difference maps to obtain the third difference structural similarity. Finally, the same algorithm is used to process multiple normalized fourth-difference frame difference maps to obtain the fourth difference structural similarity. The average of the first, second, third, and fourth difference structural similarities is taken as the final difference structural similarity.

[0097] The implementation principle for calculating the difference structure similarity is the same as that for calculating the optical flow structure similarity. For specific implementation details, please refer to the description in S302, which will not be repeated here.

[0098] S305: If the optical flow structure similarity and differential structure similarity meet the similarity requirements, and the initial classification result is a distorted screen, output the second recognition result with a stuttering display.

[0099] In the embodiments of this application, the optical flow structure similarity and the differential structure similarity can be weighted and summed to obtain the final structure similarity.

[0100] In practical implementation, the final structural similarity value can be calculated according to different weight parameters, such as S=0.6*S 光流 +0.4*S 差分 Where S represents the final structural similarity, S 光流 S represents the similarity of optical flow structures. 差分 This represents the difference structure similarity.

[0101] If the final structural similarity is greater than the set threshold and the initial classification result is a garbled screen, a second recognition result with a choppy display can be output.

[0102] In this embodiment, a frame difference map is calculated between optical flow feature maps of multiple consecutive frames, as well as a frame difference map is calculated between differential feature maps of multiple consecutive frames. Based on the similarity of the normalized frame difference maps, it is determined whether there is a slow update problem between consecutive frames. If slow updates and screen tearing occur, it can be determined that there is a stuttering problem in the image.

[0103] Figure 4This is a framework diagram of a display screen detection system provided in an embodiment of this application. The display screen detection system includes an image classification model, two buffers, a pixel matching module, a motion direction detection module, a control module, a frame difference calculation module, a similarity calculation module, and a detection result analysis module. For the two buffers, one buffer can be used to cache edge feature maps and diagonal gradient feature maps; the other buffer can be used to cache optical flow feature maps and difference feature maps.

[0104] The image classification model outputs three-class classification results and shares data from internal convolutional layers. The three-class classification results include three categories: distorted screen, black screen, and normal screen. The internal convolutional layer data includes edge feature maps, diagonal gradient feature maps, optical flow feature maps, and difference feature maps. The control module controls the global hardware logic and the execution of the internal state machine. The pixel matching module determines the best matching pixel block pairs and confirms the motion vector. The motion direction detection module confirms motion behavior between consecutive frames and checks for vector jumps. The frame difference calculation module calculates the frame difference map between feature maps to obtain a normalized result. The similarity calculation module calculates the similarity of the frame difference maps to check for slow update issues between consecutive frames. The detection result analysis module outputs the final detection result. If no flickering or stuttering anomalies are detected, it outputs the initial classification result; if flickering or stuttering anomalies are detected, it combines the calculations from the internal modules to output the final detection result.

[0105] The display detection system includes various functional modules that can be deployed in the BMC, enabling timely detection of abnormal display problems during BMC system operation.

[0106] In this embodiment of the application, the initial classification result, the first identification result, and the second identification result can be summarized to determine the final identification result; if there is an anomaly in the final identification result, the final identification result and its corresponding system log are fed back to the controller.

[0107] By feeding system logs back to the controller, problems can be located and analyzed more easily and quickly. The final identification results are promptly fed back to the BMC system, allowing for optimized system output, improving system stability and availability, and enhancing product performance. This solution supports real-time detection and output of synchronized system logs, increasing system maintainability. It does not consume video memory bandwidth, ensuring system stability and smoothness. Furthermore, in practical applications, when anomalies are detected, the load on the BMC system can be adjusted promptly. When the load is too high, the system load can be reduced in a timely manner by changing the scheduling strategy, thereby ensuring stable and smooth display.

[0108] Figure 5A schematic diagram of a display screen detection device provided in an embodiment of this application includes an analysis unit 51, an extraction unit 52, a first determination unit 53, and a second determination unit 54; Analysis unit 51 is used to analyze multiple frames of images using an image classification model to determine the initial classification result; wherein the initial classification result includes screen distortion, black screen, and normal display; Extraction unit 52 is used to extract edge feature maps, diagonal gradient feature maps, optical flow feature maps and difference feature maps from multiple frames of images, respectively; The first determining unit 53 is used to determine a first recognition result of the image based on the relative edge displacement between edge feature maps of adjacent frames and the relative gradient displacement between diagonal gradient feature maps of adjacent frames; wherein the first recognition result includes screen flickering. The second determining unit 54 is used to determine the second recognition result of the image based on the initial classification result, the optical flow structure similarity corresponding to the multi-frame optical flow feature map, and the difference structure similarity corresponding to the multi-frame difference feature map; wherein, the second recognition result includes display screen stuttering.

[0109] In some embodiments, the extraction unit is used to extract edge feature maps from multiple frames of images using the image edge detection operator of the image classification model; extract diagonal gradient feature maps from multiple frames of images using the diagonal gradient kernel of the image classification model; extract optical flow feature maps from multiple frames of images using the optical flow detection kernel of the image classification model; and extract difference feature maps from multiple frames of images using the difference kernel of the image classification model.

[0110] In some embodiments, the first determining unit includes an edge matching subunit, an edge displacement subunit, a diagonal gradient matching subunit, a diagonal gradient displacement subunit, and a result determining subunit; The edge matching subunit is used to traverse the edge feature maps of adjacent frames according to the principle of minimum difference in order to determine the best matching pixel block pair for the edge. The edge displacement subunit is used to determine the relative edge displacement between the edge feature maps of adjacent frames based on the pixels of the best-matching pixel block of the previous frame and the pixels of the best-matching pixel block of the current frame. The diagonal gradient matching subunit is used to traverse the diagonal gradient feature maps of adjacent frames according to the principle of minimum difference in order to determine the best matching pixel block pair of diagonal gradient. The diagonal gradient displacement subunit is used to determine the relative gradient displacement between the diagonal gradient feature maps of adjacent frames based on the pixels of the best-matching pixel block of the diagonal gradient in the previous frame and the pixels of the best-matching pixel block of the diagonal gradient in the current frame. The result determination subunit is used to determine the first recognition result of the image based on the angle between the relative displacement of the edges of multiple adjacent frames and the relative displacement of the gradients of multiple adjacent frames.

[0111] In some embodiments, the edge matching subunit is configured to traverse the first edge feature map of the first frame image and the first edge feature map of the second frame image according to the first window, in order to determine the first best matching pixel block pair with the smallest absolute difference value. The second edge feature map within the first edge best-matching pixel block pair is traversed according to the second window to determine the second edge best-matching pixel block pair with the smallest absolute difference value. The third edge feature map within the second edge best-matching pixel block pair is traversed according to the third window to determine the third edge best-matching pixel block pair with the smallest absolute difference value. The fourth edge feature map within the third edge best-matching pixel block pair is determined by traversing the fourth edge feature map within the fourth window to identify the fourth edge best-matching pixel block pair with the smallest absolute difference value.

[0112] In some embodiments, the edge matching subunit is configured to select a first window map according to the first edge feature map of the first frame image; Based on the set search range, the window is moved on the first edge feature map of the second frame image to obtain multiple second window maps; Based on the deviations between all pixels in the first window image and all pixels in each of the second window images, multiple absolute difference values ​​are determined. The first window image and the second window image with the smallest absolute difference value are used as the first edge best matching pixel block pair.

[0113] In some embodiments, the edge matching subunit is used to perform a difference operation on the gray values ​​of all pixels in the first window image and the gray values ​​of all pixels in the target second window image to obtain multiple difference values; wherein, the target second window image is any one of the second window images; Sum the absolute values ​​of all differences to obtain the absolute difference value corresponding to the second window of the target.

[0114] In some embodiments, the diagonal gradient matching subunit is used to traverse the first diagonal gradient feature map of the first frame image and the first diagonal gradient feature map of the second frame image according to the fifth window, so as to determine the first diagonal gradient best matching pixel block pair with the smallest absolute difference value. The second diagonal gradient feature map within the first diagonal gradient best-matching pixel block pair is determined by traversing the second diagonal gradient best-matching pixel block pair within the first diagonal gradient best-matching pixel block pair using the sixth window; The third diagonal gradient feature map within the second diagonal gradient best-matching pixel block pair is determined by traversing the third diagonal gradient best-matching pixel block pair within the seventh window to identify the third diagonal gradient best-matching pixel block pair with the smallest absolute difference value. The fourth diagonal gradient feature map within the third diagonal gradient best-matching pixel block pair is determined by traversing the fourth diagonal gradient best-matching pixel block pair within the eighth window.

[0115] In some embodiments, the diagonal gradient matching subunit is used to select a fifth window map according to the first diagonal gradient feature map of the first frame image; Based on the set search range, the window is moved on the first diagonal gradient feature map of the second frame image to obtain multiple sixth window maps. Based on the deviations between all pixels in the fifth window image and all pixels in each sixth window image, multiple absolute difference values ​​are determined; The fifth window image and the sixth window image with the smallest absolute difference value are used as the first diagonal gradient best-matched pixel block pair.

[0116] In some embodiments, the diagonal gradient matching subunit is used to perform a difference operation on the gray values ​​of all pixels in the fifth window image and the gray values ​​of all pixels in the target sixth window image to obtain multiple difference values; wherein, the target sixth window image is any one of the sixth window images; Sum the absolute values ​​of all differences to obtain the absolute difference value corresponding to the sixth window of the target.

[0117] In some embodiments, the result determination subunit is used to perform a weighted summation of the edge relative displacement of the first adjacent frame and the gradient relative displacement of the first adjacent frame to obtain the first motion transformation direction vector. The relative displacement of the edge of the second adjacent frame and the relative displacement of the gradient of the second adjacent frame are weighted and summed to obtain the second motion transformation direction vector; If the angle between the first motion transformation direction vector and the second motion transformation direction vector is greater than a set vector threshold, the first recognition result with flickering display screen will be output.

[0118] In some embodiments, an accumulation unit and a clearing unit are also included; The accumulation unit increments the flashing behavior count by one each time the angle between the first motion transformation direction vector and the second motion transformation direction vector is greater than a set vector threshold. If the flashing behavior count is greater than or equal to the set behavior threshold within a set time, the result determination subunit is triggered to execute the operation step of outputting the first recognition result of the flashing screen. The zeroing unit is used to reset the flashing behavior count value to zero if the flashing behavior count value is less than the set behavior threshold within a set time.

[0119] In some embodiments, the second determining unit includes a first normalization subunit, a first processing subunit, a second normalization subunit, a second processing subunit, and an output subunit; The first normalization subunit is used to normalize the optical flow frame difference map corresponding to the optical flow feature map of multiple consecutive frames of images to obtain multiple normalized optical flow frame difference maps. The first processing subunit is used to process multiple normalized optical flow frame difference maps using a structural similarity algorithm to obtain optical flow structural similarity. The second normalization subunit is used to normalize the difference frame difference map corresponding to the difference feature map of multiple consecutive frames of images to obtain multiple normalized difference frame difference maps. The second processing subunit is used to process multiple normalized differential frame difference maps using a structural similarity algorithm to obtain differential structural similarity. The output subunit is used to output a second recognition result that causes screen stuttering when the optical flow structure similarity and differential structure similarity meet the similarity requirements and the initial classification result is a garbled screen.

[0120] In some embodiments, the first normalization subunit is used to analyze multiple first optical flow feature maps of consecutive frames of images based on the image frame difference method to determine multiple first optical flow frame difference maps. The pixel values ​​of multiple first optical flow frame difference maps are subtracted from the pixel average of all first optical flow frame difference maps to obtain multiple normalized first optical flow frame difference maps. Based on the image frame difference method, multiple second optical flow feature maps of consecutive multi-frame images are analyzed to determine multiple second optical flow frame difference maps; The pixel values ​​of multiple second optical flow frame difference maps are subtracted from the average pixel value of all second optical flow frame difference maps to obtain multiple normalized second optical flow frame difference maps. Based on the image frame difference method, multiple third optical flow feature maps of consecutive multi-frame images are analyzed to determine multiple third optical flow frame difference maps; The pixel values ​​of multiple third optical flow frame difference maps are subtracted from the pixel average of all third optical flow frame difference maps to obtain multiple normalized third optical flow frame difference maps. Based on the image frame difference method, multiple fourth optical flow feature maps of consecutive multi-frame images are analyzed to determine multiple fourth optical flow frame difference maps; The pixel values ​​of multiple fourth optical flow frame difference maps are subtracted from the average pixel value of all fourth optical flow frame difference maps to obtain multiple normalized fourth optical flow frame difference maps.

[0121] In some embodiments, the first processing subunit is used to process multiple normalized first optical flow frame difference maps using a structural similarity algorithm to obtain a first structural similarity. The structural similarity algorithm is used to process multiple normalized second optical flow frame difference maps to obtain the second structural similarity. The structural similarity algorithm is used to process multiple normalized third optical flow frame difference maps to obtain the third structural similarity. The structural similarity algorithm is used to process multiple normalized fourth optical flow frame difference maps to obtain the fourth structural similarity. The average of the first, second, third, and fourth structural similarities is taken as the optical flow structural similarity.

[0122] In some embodiments, the output subunit is used to perform a weighted summation of the optical flow structure similarity and the differential structure similarity to obtain the final structure similarity; If the final structural similarity is greater than the set threshold and the initial classification result is a garbled screen, the output will show a second recognition result with a choppy display.

[0123] In some embodiments, a summarizing unit and a feedback unit are also included; The summarization unit is used to summarize the initial classification results, the first identification results, and the second identification results to determine the final identification results; The feedback unit is used to send the final identification result and its corresponding system log back to the controller if there is an anomaly in the final identification result.

[0124] For a description of the features in the embodiment corresponding to the display screen detection device, please refer to the relevant description in the embodiment corresponding to the display screen detection method, which will not be repeated here.

[0125] As can be seen from the above technical solution, an image classification model is used to analyze multiple frames of images to determine the initial classification result, which includes screen flickering, black screen, and normal display. Different types of features can reflect different characteristics of the image. To accurately and comprehensively detect anomalies in the display, edge feature maps, diagonal gradient feature maps, optical flow feature maps, and difference feature maps can be extracted from multiple frames of images. Based on the relative edge displacement between the edge feature maps of adjacent frames and the relative gradient displacement between the diagonal gradient feature maps of adjacent frames, the first recognition result of the image is determined, which includes screen flickering. Based on the initial classification result, the optical flow structure similarity corresponding to the optical flow feature maps of multiple frames, and the difference structure similarity corresponding to the difference feature maps of multiple frames, the second recognition result of the image is determined, which includes screen stuttering. In this technical solution, based on the initial classification result determined by the image classification model, by analyzing different types of feature maps in the multiple frames of images currently displayed on the monitor, abnormal problems in the display can be detected in a timely and accurate manner during system operation, improving the maintainability of the system.

[0126] Embodiments of this application also provide an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to perform the steps in any of the above-described embodiments of the display screen detection method.

[0127] Embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above-described methods for detecting display screens when it is run.

[0128] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0129] The embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above-described methods for detecting display screens.

[0130] Embodiments of this application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in any of the above-described methods for detecting display screens.

[0131] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0132] The foregoing has provided a detailed description of a display screen detection method, apparatus, device, medium, and product. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are only intended to aid in understanding the method and core ideas of this application. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of this application.

Claims

1. A method for detecting a displayed image, characterized in that, include: An image classification model is used to analyze multiple frames of images to determine the initial classification result; wherein, the initial classification result includes screen distortion, black screen, and normal display; Edge feature maps, diagonal gradient feature maps, optical flow feature maps, and difference feature maps are extracted from multiple frames of images, respectively. Based on the relative displacement of edges between edge feature maps of adjacent frames and the relative displacement of gradients between diagonal gradient feature maps of adjacent frames, a first recognition result of the image is determined; wherein, the first recognition result includes screen flickering. Based on the initial classification results, the optical flow structure similarity corresponding to the multi-frame optical flow feature maps, and the difference structure similarity corresponding to the multi-frame difference feature maps, a second recognition result for the image is determined; wherein, the second recognition result includes display screen stuttering.

2. The method for detecting the displayed image according to claim 1, characterized in that, The extraction of edge feature maps, diagonal gradient feature maps, optical flow feature maps, and difference feature maps from multiple frames of images includes: Using the image edge detection operator of the image classification model, edge feature maps are extracted from multiple frames of images; Using the diagonal gradient kernel of the image classification model, diagonal gradient feature maps are extracted from multiple frames of images; Optical flow feature maps are extracted from multiple frames of images using the optical flow detection kernel of the image classification model. The difference feature map is extracted from multiple frames of images using the difference kernel of the image classification model.

3. The method for detecting the displayed image according to claim 1, characterized in that, The first recognition result of the image is determined based on the relative edge displacement between edge feature maps of adjacent frames and the relative gradient displacement between diagonal gradient feature maps of adjacent frames, including: The edge feature maps of adjacent frames are traversed according to the principle of minimum difference to determine the best matching pixel block pair of the edge; Based on the pixels of the best-matching pixel block of the edge in the previous frame and the pixels of the best-matching pixel block of the edge in the current frame, the relative displacement of the edge between the edge feature maps of adjacent frames is determined. The diagonal gradient feature maps of adjacent frames are traversed according to the principle of minimum difference to determine the best matching pixel block pair of diagonal gradient. Based on the pixels of the best-matching pixel block of the diagonal gradient in the previous frame and the pixels of the best-matching pixel block of the diagonal gradient in the current frame, the relative gradient displacement between the diagonal gradient feature maps of adjacent frames is determined. The first recognition result of the image is determined based on the angle between the relative displacement of the edges of multiple adjacent frames and the relative displacement of the gradients of multiple adjacent frames.

4. The method for detecting the display screen according to claim 3, characterized in that, The step of traversing the edge feature maps of adjacent frames according to the principle of minimum difference to determine the best matching pixel block pair includes: By traversing the first edge feature map of the first frame image and the first edge feature map of the second frame image through the first window, the best matching pixel block pair of the first edge with the smallest absolute difference value is determined; The second edge feature map within the first best-matching pixel block pair is traversed through the second window to determine the second best-matching pixel block pair with the smallest absolute difference value. The third edge feature map within the second edge best-matching pixel block pair is traversed according to the third window to determine the third edge best-matching pixel block pair with the smallest absolute difference value. The fourth edge feature map within the third edge best-matching pixel block pair is traversed using the fourth window to determine the fourth edge best-matching pixel block pair with the smallest absolute difference value.

5. The method for detecting the display screen according to claim 4, characterized in that, The step of traversing the first edge feature map of the first frame image and the first edge feature map of the second frame image according to the first window to determine the first edge best matching pixel block pair with the smallest absolute difference value includes: The first window image is selected from the first edge feature map of the first frame image according to the first window; Based on the set search range, the window is moved on the first edge feature map of the second frame image to obtain multiple second window maps; Based on the deviation between all pixels in the first window image and all pixels in each of the second window images, multiple absolute difference values ​​are determined; The first window image and the second window image with the smallest absolute difference value are taken as the first edge best matching pixel block pair.

6. The method for detecting a display screen according to claim 5, characterized in that, The method determines multiple absolute difference values ​​based on the deviations between all pixels in the first window image and all pixels in each of the second window images, including: A difference operation is performed on the grayscale values ​​of all pixels in the first window image and the grayscale values ​​of all pixels in the target second window image to obtain multiple difference values; wherein, the target second window image is any one of the second window images; The absolute values ​​of all differences are summed to obtain the absolute difference value corresponding to the second window of the target.

7. The method for detecting the display screen according to claim 3, characterized in that, The step of traversing the diagonal gradient feature maps of adjacent frames according to the principle of minimum difference to determine the best matching pixel block pair of diagonal gradients includes: By traversing the first diagonal gradient feature map of the first frame image and the first diagonal gradient feature map of the second frame image through the fifth window, the best matching pixel block pair with the smallest absolute difference value of the first diagonal gradient is determined. The second diagonal gradient feature map within the first diagonal gradient best-matching pixel block pair is determined by traversing the second diagonal gradient best-matching pixel block pair within the first diagonal gradient best-matching pixel block pair using the sixth window; The third diagonal gradient feature map within the second diagonal gradient best-matching pixel block pair is determined by traversing the third diagonal gradient best-matching pixel block pair within the seventh window to identify the third diagonal gradient best-matching pixel block pair with the smallest absolute difference value. The fourth diagonal gradient feature map within the third diagonal gradient best-matching pixel block pair is determined by traversing the fourth diagonal gradient best-matching pixel block pair within the eighth window.

8. The method for detecting a display screen according to claim 7, characterized in that, The step of traversing the first diagonal gradient feature map of the first frame image and the first diagonal gradient feature map of the second frame image according to the fifth window to determine the best matching pixel block pair of the first diagonal gradient with the smallest absolute difference value includes: The fifth window image is selected from the first diagonal gradient feature map of the first frame image according to the fifth window described above; Based on the set search range, the window is moved on the first diagonal gradient feature map of the second frame image to obtain multiple sixth window maps; Based on the deviation between all pixels in the fifth window image and all pixels in each of the sixth window images, multiple absolute difference values ​​are determined; The fifth window image and the sixth window image with the smallest absolute difference value are used as the first diagonal gradient best-matched pixel block pair.

9. The method for detecting a display screen according to claim 8, characterized in that, Based on the deviation between all pixels in the fifth window image and all pixels in each of the sixth window images, multiple absolute difference values ​​are determined, including: A difference operation is performed on the grayscale values ​​of all pixels in the fifth window image and the grayscale values ​​of all pixels in the target sixth window image to obtain multiple difference values; wherein, the target sixth window image is any one of the sixth window images; Sum the absolute values ​​of all differences to obtain the absolute difference value corresponding to the sixth window of the target.

10. The method for detecting a display screen according to claim 3, characterized in that, The determination of the first recognition result of the image based on the angle between the relative edge displacements of multiple adjacent frames and the relative gradient displacements of multiple adjacent frames includes: The relative displacement of the edges of the first adjacent frame and the relative displacement of the gradient of the first adjacent frame are weighted and summed to obtain the first motion transformation direction vector; The relative displacement of the edge of the second adjacent frame and the relative displacement of the gradient of the second adjacent frame are weighted and summed to obtain the second motion transformation direction vector; If the angle between the first motion transformation direction vector and the second motion transformation direction vector is greater than a set vector threshold, the first recognition result of the flickering display screen will be output.

11. The method for detecting a display screen according to claim 10, characterized in that, Before the first recognition result, which is displayed on the screen, is shown, the following is also included: Each time the angle between the first motion transformation direction vector and the second motion transformation direction vector is greater than the set vector threshold, the flashing behavior count value is incremented by one. If the flashing behavior count value is less than the set behavior threshold within a set time, the flashing behavior count value will be cleared to zero. If the blinking behavior count value is greater than or equal to the set behavior threshold within a set time, the operation step of the first recognition result of the blinking of the output display screen is executed.

12. The method for detecting a display screen according to claim 1, characterized in that, The second recognition result of the image is determined based on the initial classification result, the optical flow structure similarity corresponding to the multi-frame optical flow feature maps, and the difference structure similarity corresponding to the multi-frame difference feature maps, including: Normalize the optical flow frame difference maps corresponding to the optical flow feature maps of multiple consecutive frames of images to obtain multiple normalized optical flow frame difference maps. The structural similarity algorithm is used to process multiple normalized optical flow frame difference maps to obtain the optical flow structural similarity. Normalize the difference frame difference maps corresponding to the difference feature maps of multiple consecutive frames of images to obtain multiple normalized difference frame difference maps. A structural similarity algorithm is used to process multiple normalized differential frame difference maps to obtain differential structural similarity. If the optical flow structure similarity and the difference structure similarity meet the similarity requirements, and the initial classification result is a distorted screen, a second recognition result with stuttering display is output.

13. The method for detecting a display screen according to claim 12, characterized in that, The normalization operation on the optical flow frame difference maps corresponding to the optical flow feature maps of consecutive multiple frames of images to obtain multiple normalized optical flow frame difference maps includes: Based on the image frame difference method, multiple first optical flow feature maps of consecutive multi-frame images are analyzed to determine multiple first optical flow frame difference maps; The pixel values ​​of multiple first optical flow frame difference maps are subtracted from the pixel average of all first optical flow frame difference maps to obtain multiple normalized first optical flow frame difference maps. Based on the image frame difference method, multiple second optical flow feature maps of consecutive multi-frame images are analyzed to determine multiple second optical flow frame difference maps; The pixel values ​​of multiple second optical flow frame difference maps are subtracted from the average pixel value of all second optical flow frame difference maps to obtain multiple normalized second optical flow frame difference maps. Based on the image frame difference method, multiple third optical flow feature maps of consecutive multi-frame images are analyzed to determine multiple third optical flow frame difference maps; The pixel values ​​of multiple third optical flow frame difference maps are subtracted from the pixel average of all third optical flow frame difference maps to obtain multiple normalized third optical flow frame difference maps. Based on the image frame difference method, multiple fourth optical flow feature maps of consecutive multi-frame images are analyzed to determine multiple fourth optical flow frame difference maps; The pixel values ​​of multiple fourth optical flow frame difference maps are subtracted from the average pixel value of all fourth optical flow frame difference maps to obtain multiple normalized fourth optical flow frame difference maps.

14. The method for detecting a display screen according to claim 13, characterized in that, The process of using a structural similarity algorithm to process multiple normalized optical flow frame difference maps to obtain optical flow structural similarity includes: The structural similarity algorithm is used to process multiple normalized first optical flow frame difference maps to obtain the first structural similarity. The structural similarity algorithm is used to process multiple normalized second optical flow frame difference maps to obtain the second structural similarity. The structural similarity algorithm is used to process multiple normalized third optical flow frame difference maps to obtain the third structural similarity. The structural similarity algorithm is used to process multiple normalized fourth optical flow frame difference maps to obtain the fourth structural similarity. The average of the first structural similarity, the second structural similarity, the third structural similarity, and the fourth structural similarity is taken as the optical flow structural similarity.

15. The method for detecting a display screen according to claim 12, characterized in that, If the optical flow structure similarity and the difference structure similarity meet the similarity requirements, and the initial classification result is a screen tearing issue, the second recognition result for outputting the display screen tearing includes: The optical flow structure similarity and the differential structure similarity are weighted and summed to obtain the final structure similarity. If the final structural similarity is greater than the set threshold, and the initial classification result is a distorted image, a second recognition result with a stuttering display is output.

16. The method for detecting a display screen according to claim 1, characterized in that, After determining the second recognition result of the image based on the initial classification result, the optical flow structure similarity corresponding to the multi-frame optical flow feature maps, and the difference structure similarity corresponding to the multi-frame difference feature maps, the following steps are also included: The initial classification result, the first identification result, and the second identification result are summarized to determine the final identification result; If the final identification result is abnormal, the final identification result and its corresponding system log will be fed back to the controller.

17. A detection device for displaying images, characterized in that, It includes an analysis unit, an extraction unit, a first determination unit, and a second determination unit; The analysis unit is used to analyze multiple frames of images using an image classification model to determine the initial classification result; wherein, the initial classification result includes screen distortion, black screen, and normal display; The extraction unit is used to extract edge feature maps, diagonal gradient feature maps, optical flow feature maps, and difference feature maps from multiple frames of images, respectively. The first determining unit is used to determine a first recognition result of the image based on the relative edge displacement between edge feature maps of adjacent frames and the relative gradient displacement between diagonal gradient feature maps of adjacent frames; wherein, the first recognition result includes screen flickering. The second determining unit is used to determine the second recognition result of the image based on the initial classification result, the optical flow structure similarity corresponding to the multi-frame optical flow feature map, and the difference structure similarity corresponding to the multi-frame difference feature map; wherein, the second recognition result includes display screen stuttering.

18. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, configured to execute the computer program to implement the steps of the detection method for the display screen as described in any one of claims 1 to 16.

19. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, it implements the steps of the detection method for the display screen as described in any one of claims 1 to 16.

20. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the display screen detection method as described in any one of claims 1 to 16.

Images