Display screen anomaly detection method and device, electronic equipment, medium and product

Abstract

The present application relates to the technical field of image processing, and discloses a display screen abnormality detection method and device, electronic equipment, medium and product, the present application is based on the video frame feature map of the monitoring video of the display screen, extracts the time sequence change feature between the continuous video frames, the spatial structure feature corresponding to the different display areas of the display screen, so as to analyze the spatial structure of different display areas separately. Then, the time sequence change intensity of the time sequence change feature, the structure abnormality intensity of each display area is analyzed, so as to measure the influence degree of the time sequence change feature and the spatial structure feature on the screen abnormality detection. Further, the spatial structure feature of the target display area with higher time sequence change feature and structure abnormality intensity is fused, so as to improve the recognition rate of the screen corner area and the micro display abnormality, and obtain the abnormality detection result of the display screen.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to a display screen abnormality detection method and device, electronic equipment, medium and product. BACKGROUND

[0002] With the rapid penetration of intelligent networked vehicles, the complexity of intelligent cockpit systems is further intensified. The frequent scheduling of hardware resources during screen rendering and the exponential increase in inter-screen interaction information can easily lead to the collapse of the vehicle-mounted screen system and screen failure. These failures not only affect the user's experience of the intelligent cockpit, but also pose a potential driving safety hazard due to display failure.

[0003] Traditional detection methods locate screen abnormalities through hardware diagnosis, but rely on manual operation. Existing technologies use automated scripts for abnormality detection, but require separate development of corresponding scripts for different vehicle systems, resulting in poor compatibility. Some detection methods based on virtual machines monitor vehicle-mounted screen failures through monitors, but require high hardware resources. To address the above issues, existing technologies provide detection methods based on image recognition, which use image recognition models to identify screen picture abnormalities. However, this method often focuses on overall picture abnormalities and has a high rate of missed detection for minor abnormalities such as screen corner flickering and minor display misalignment. SUMMARY

[0004] The present application provides a display screen abnormality detection method, device, electronic equipment, medium and product to solve the problem of missing detection of display screen corner or minor display area failure abnormalities in existing technologies.

[0005] In a first aspect, the present application provides a display screen abnormality detection method, which comprises:

[0006] Obtaining a monitoring video of the display screen and extracting a video frame feature map of the monitoring video;

[0007] Extracting time sequence change features between consecutive video frames according to the video frame feature map;

[0008] Dividing the video frame feature map into multiple region feature maps and extracting spatial structure features of each region feature map; wherein the region feature maps correspond one-to-one to different display regions of the display screen;

[0009] Determining the time sequence change intensity of the time sequence change features, obtaining the structure abnormality intensity of each display region according to the spatial structure features, and determining a target display region with a structure abnormality intensity greater than an abnormality intensity threshold;

[0010] The time sequence change feature and the spatial structure feature corresponding to the target display area are fused based on the time sequence change intensity and the structure abnormality intensity of the target display area, and a fusion feature is obtained.

[0011] The video frame feature of the monitoring video of the display screen is used to extract the time sequence change feature between continuous video frames and the spatial structure feature corresponding to different display areas of the display screen, so that the spatial structure of different display areas is analyzed separately. Then, the time sequence change intensity of the time sequence change feature and the structure abnormality intensity of each display area are analyzed to measure the influence degree of the time sequence change feature and the spatial structure feature on the screen abnormality detection. Furthermore, the time sequence change feature and the spatial structure feature of the target display area with high structure abnormality intensity are fused to improve the recognition rate of the screen corner area and the small display abnormality, and the abnormality detection result of the display screen is obtained.

[0012] In an optional implementation, the time sequence change feature and the spatial structure feature corresponding to the target display area are fused based on the time sequence change intensity and the structure abnormality intensity of the target display area, and the fusion feature is obtained.

[0013] The time sequence change intensity and the structure abnormality intensity of the target display area are input into a pre-trained nonlinear model to obtain a fusion coefficient; wherein the fusion coefficient is used to represent the importance degree of the time sequence change feature.

[0014] The time sequence change feature and the spatial structure feature corresponding to the target display area are fused according to the fusion coefficient to obtain a fusion feature.

[0015] The time sequence change intensity and the structure abnormality intensity of the target display area are input into a pre-trained nonlinear model, so that the nonlinear model determines the influence degree of the time sequence change feature and the spatial structure feature of the target display area on the screen abnormality detection, and then outputs the fusion coefficient. In this way, the proportion of the time sequence change feature and the spatial structure feature of the target display area in the feature fusion is adjusted by using the fusion coefficient, and the adaptation degree of the screen abnormality detection in various scenes is improved.

[0016] In an optional implementation, the structure abnormality intensity of each display area is obtained according to the spatial structure feature, and the structure abnormality intensity of each display area is obtained according to the spatial structure feature.

[0017] The luminance variance feature and the texture variance feature of each display area are obtained according to the luminance feature and the texture feature in the spatial structure feature.

[0018] The luminance difference feature between the luminance variance feature of each display area and the preset luminance variance is determined, and the texture difference feature between the texture variance feature of each display area and the preset texture variance is determined.

[0019] According to the luminance difference feature and the texture difference feature, the structural anomaly strength of each display area is calculated.

[0020] The present application can cover the core fault type based on the luminance feature for anomaly detection, and the texture feature is a key dimension for describing the structure of the screen content. The variance is calculated using the luminance feature and the texture feature, the luminance difference feature and the texture difference feature are obtained, the structural anomaly strength of each display area is measured, the full scene coverage is realized while supporting mutual verification, the calculation cost is lower, and it is more suitable for the lightweight demand of screen anomaly detection.

[0021] In an optional implementation, the time sequence change strength of the time sequence change feature is determined, including:

[0022] The mean square error feature of the time sequence change feature is determined.

[0023] According to the comparison relationship between the mean square error feature and the preset mean square error range, the time sequence change strength of the time sequence change feature is calculated.

[0024] The present application captures the fluctuation characteristics of the time sequence change feature by calculating the mean square error feature of the time sequence change feature, suppresses the isolated noise of individual video frames, avoids local misjudgment, and has very low calculation overhead. Therefore, the time sequence change strength of the time sequence change feature is measured by using the comparison relationship of the mean square error feature relative to the preset mean square error range, so as to improve the recognition accuracy of the screen anomaly in combination with the time sequence change strength.

[0025] In an optional implementation, the time sequence change feature between the continuous video frames is extracted according to the video frame feature map, including:

[0026] According to the video frame feature map, the difference feature between each two continuous video frames is calculated;

[0027] According to the difference feature, the time sequence change feature between the continuous video frames is obtained.

[0028] The present application provides the input basis by calculating the difference feature between each two continuous video frames and taking the difference feature as the time sequence change feature, filters out a large amount of static redundant information existing in the continuous video frames, reduces the input of irrelevant static redundant information, and improves the robustness of screen anomaly detection.

[0029] In an optional implementation, according to the obtained fusion feature, the anomaly detection result of the display screen is determined, including:

[0030] The fusion feature is compressed to obtain a fixed-length global feature vector;

[0031] Input the global feature vector into the pre-trained anomaly classification model to obtain an anomaly detection result of the display screen; wherein the anomaly detection result includes a flashing screen, a black screen, a blooming screen, a white screen, and a normal screen.

[0032] The global feature vector is used to determine whether the display screen is abnormal, such as flashing, black screen, blooming screen, white screen, etc., and the global feature vector can effectively reduce the feature dimension, avoiding overfitting of the pre-trained anomaly classification model during anomaly detection.

[0033] In an optional implementation, the training process of the anomaly classification model includes:

[0034] Construct a screen image dataset and a preset model structure;

[0035] Perform a data enhancement operation on the screen image dataset to obtain an enhanced image dataset; the data enhancement operation includes image occlusion, geometric transformation, light adjustment, screen anomaly simulation, and resolution adjustment;

[0036] Add a class label to each image in the enhanced image dataset; the class label corresponds to the abnormal type of the display screen;

[0037] Train the preset model structure using the enhanced image dataset to obtain the pre-trained anomaly classification model.

[0038] Based on the constructed screen image dataset, the enhanced image dataset is constructed and the class label is added through the data enhancement operations such as image occlusion, geometric transformation, light adjustment, screen anomaly simulation, and resolution adjustment, so as to train the model using the enhanced image dataset, and improve the robustness, generalization ability, scene adaptability, and detection accuracy of the anomaly classification model.

[0039] In an optional implementation, the pre-trained anomaly classification model is deployed in the cloud after pruning and quantization processing; the global feature vector is input into the pre-trained anomaly classification model to obtain the anomaly detection result of the display screen, including:

[0040] Encapsulate the global feature vector into a classification request;

[0041] Send the classification request to the cloud through a preset calling interface, so that the cloud calls the pre-trained anomaly classification model based on the classification request to obtain an anomaly classification result;

[0042] Receive the anomaly classification result returned by the cloud to obtain the anomaly detection result of the display screen.

[0043] The application realizes light deployment by deploying the abnormal classification model after pruning and quantization in the cloud, so that the car machine can obtain the abnormal classification result returned by the cloud by calling the abnormal classification model deployed in the cloud, and the delay of screen abnormality detection is reduced.

[0044] In a second aspect, the application provides a display screen abnormality detection device, which comprises:

[0045] A first processing module is configured to acquire a monitoring video of a display screen and extract a video frame feature map of the monitoring video.

[0046] A second processing module is configured to extract a time sequence change feature between continuous video frames according to the video frame feature map.

[0047] A third processing module is configured to divide the video frame feature map into a plurality of region feature maps and extract a spatial structure feature of each region feature map, wherein the region feature maps correspond to different display regions of the display screen one by one.

[0048] A fourth processing module is configured to determine a time sequence change intensity of the time sequence change feature, obtain a structure abnormality intensity of each display region according to the spatial structure feature, and determine a target display region with a structure abnormality intensity greater than an abnormality intensity threshold.

[0049] A fifth processing module is configured to fuse the time sequence change feature and the spatial structure feature corresponding to the target display region based on the time sequence change intensity and the structure abnormality intensity of the target display region, and determine an abnormality detection result of the display screen according to the obtained fused feature.

[0050] In a third aspect, the application provides an electronic device, which comprises a memory and a processor, the memory and the processor are communicatively connected, the memory stores computer instructions, and the processor executes the computer instructions to perform the display screen abnormality detection method of the first aspect or any of the corresponding embodiments thereof.

[0051] In a fourth aspect, the application provides a computer readable storage medium, which stores computer instructions, and the computer instructions are used to make a computer execute the display screen abnormality detection method of the first aspect or any of the corresponding embodiments thereof.

[0052] In a fifth aspect, the application provides a computer program product, which comprises computer instructions, and the computer instructions are used to make a computer execute the display screen abnormality detection method of the first aspect or any of the corresponding embodiments thereof.

[0053] The application has the following beneficial effects:

[0054] The application is based on a video frame feature map of a monitoring video of a display screen, extracts a time sequence change feature between continuous video frames, and a spatial structure feature corresponding to different display regions of the display screen, so as to separately analyze the spatial structure of different display regions. Then, the time sequence change intensity of the time sequence change feature and the structure abnormality intensity of each display region are analyzed, so as to measure the influence degree of the time sequence change feature and the spatial structure feature on screen abnormality detection. Furthermore, the spatial structure feature of a target display region with high time sequence change feature and structure abnormality intensity is fused, so as to improve the recognition rate of screen corner regions and small display abnormalities, and obtain an abnormality detection result of the display screen. BRIEF DESCRIPTION OF DRAWINGS

[0055] In order to more clearly illustrate the specific embodiments of the present application or the technical solutions in the prior art, the drawings required to be used in the specific embodiments or prior art description will be briefly introduced as follows. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0056] Figure 1 is a schematic diagram of an application scenario according to an embodiment of the present application;

[0057] Figure 2 is a first flowchart of a display screen abnormality detection method according to an embodiment of the present application;

[0058] Figure 3 is a second flowchart of a display screen abnormality detection method according to an embodiment of the present application;

[0059] Figure 4 is a flowchart of constructing an enhanced image data set according to an embodiment of the present application;

[0060] Figure 5 is a training flowchart of an abnormality classification model according to an embodiment of the present application;

[0061] Figure 6 is a model framework diagram of a display screen abnormality detection system according to an embodiment of the present application;

[0062] Figure 7 is a structural block diagram of a display screen abnormality detection device according to an embodiment of the present application;

[0063] Figure 8 is a hardware structure schematic diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0064] In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the following will clearly and completely describe the technical solutions in the embodiments of the present application with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some but not all of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of protection of the present application.

[0065] It can be understood that, before using the technical solutions disclosed in the embodiments of the present application, the type, use range, use scenario and the like of the personal information involved in the present application should be informed to the user and the authorization of the user should be obtained through appropriate means according to relevant laws and regulations.

[0066] The terms "first", "second" are only for descriptive purpose, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features limited by "first", "second" can explicitly or implicitly include one or more of the features. In the description of the present application, the meaning of "multiple" is two or more, unless otherwise specifically limited.

[0067] As an optional application scenario of the embodiments of the present application, as shown in Figure 1 The display screen abnormality detection system can include at least one terminal device and at least one server, Figure 1 As shown in the example in the figure, the system includes a computer 101, a mobile terminal 102 and a server 103, and the terminal devices such as the computer 101 and the mobile terminal 102 are connected to the server 103 through a network 110.

[0068] The terminal device can be specifically a smart phone, a tablet computer, a notebook computer, a palm computer, and can also be a desktop computer, a game console, a smart television, a smart wearable device, a vehicle-mounted terminal, a VR (Virtual Reality) device, an AR (Augmented Reality) device, etc. The server 103 can be an independent physical server, or a server cluster or a distributed system, or a cloud server providing cloud services. The network 110 can be a wired network or a wireless network, and its examples include but are not limited to the Internet, an intranet, a local area network, a wide area network, a mobile communication network and a combination thereof.

[0069] With the rapid penetration of intelligent connected vehicles, the complexity of intelligent cockpit systems has further intensified. Industry data shows that car machine problems have become the focus of user complaints. In recent years, car machine related failures (including black screen, screen sticking, blur, etc.) account for 9.56%~15.89% of total automotive complaints. The reason is that high-performance chips are prone to system crashes when processing 3D rendering, frequent scheduling of central processing units (CPU) and graphics processing units (GPU), and exponential increase in inter-screen interaction information, and even some car models have a black screen failure rate of up to 24%. Such abnormalities not only damage the user's experience of the intelligent cockpit, but also pose a potential driving safety hazard due to display failure.

[0070] In view of the above screen abnormality hidden danger and the investment demand of automobile enterprises for car machine reliability construction, the current common methods for detecting car machine screen abnormalities include the following:

[0071] (1) Traditional detection method: This method covers hardware inspection, power / signal line testing, system and software diagnosis, etc. Through tools to measure voltage, check signals, read fault codes to locate abnormal problems. However, this method relies on manual operation, has low efficiency, is difficult to troubleshoot complex implicit software or compatibility faults, and has insufficient robustness to complex lighting, screen resolution changes, display content dynamic updates, etc., resulting in high miss rate and false detection rate.

[0072] (2) Detection method based on automatic script: The core principle of this method is to interact with the car machine native service based on the C layer interface (i.e. the interface connecting the application layer and the car machine native service), draw a specified color by running a script, and then determine the fault according to whether the actual display effect of the screen meets the expectations. However, the developed automatic script needs to be individually adapted for different car machine systems, has poor compatibility, and can only detect display picture abnormalities, making it difficult to identify occasional software defects.

[0073] (3) Detection method based on virtual machine: This method uses a dual virtual machine division of labor mode, in which the monitor driver monitors the abnormal pin signals of the screen, and then transmits the monitoring information to the screen monitor for display. However, this method has specific requirements for the environment and hardware. On the one hand, it relies on virtual machine environment setup, which has a high operation threshold and high hardware resource requirements. On the other hand, its detection range is narrow, focusing only on motor control unit abnormalities, and is suitable for a single fixed detection scenario.

[0074] (4) Image recognition-based detection method: this method first collects the real-time picture of the screen of the vehicle machine through the camera, then inputs the picture into a pre-trained image recognition model such as a convolutional neural network (CNN), YOLO, etc., extracts and analyzes the picture features by the image recognition model to identify abnormal problems such as black screen, flashing screen, flower screen, color distortion, display misplacement, etc. However, the general image recognition model is often difficult to balance between accuracy and speed, and has a high missed judgment rate for small abnormalities such as screen corner flickering and small display misplacement, and has the problems of weak overall anti-interference and being easily affected by the external environment.

[0075] The embodiment of the present application provides a display screen abnormality detection method, based on the video frame feature map of the monitoring video of the display screen, extracts the time sequence change feature between the continuous video frames, the spatial structure feature corresponding to the different display areas of the display screen, so as to analyze the spatial structure of the different display areas separately. Then, analyze the time sequence change intensity of the time sequence change feature, the structure abnormality intensity of each display area, and determine the target display area with high structure abnormality intensity. Thus, the time sequence change intensity and the structure abnormality intensity of the target display area are used for feature fusion of the time sequence change feature and the spatial structure feature of the target display area, so as to improve the recognition rate of the screen corner area and the small display abnormality, and obtain the abnormality detection result of the display screen.

[0076] According to the embodiment of the present application, a display screen abnormality detection method embodiment is provided. It should be noted that the steps shown in the flowchart can be executed in a computer system such as a set of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a different order from here.

[0077] In this embodiment, a display screen abnormality detection method is provided, which can be used in the above-mentioned mobile terminal, such as vehicle terminal, mobile phone, tablet computer, etc. Figure 2 is a flowchart of the display screen abnormality detection method according to the embodiment of the present application, as shown in Figure 2 The flowchart includes the following steps:

[0078] Step S201, acquiring a monitoring video of a display screen and extracting a video frame feature map of the monitoring video.

[0079] Specifically, taking the display screen as a vehicle screen as an example, the display screen is monitored by a vehicle camera to obtain a monitoring video of the display screen, and the monitoring video includes a series of continuous video frames. Then, a pre-set neural network model is used to extract features of each video frame of the monitoring video, extract the RGB feature map of each video frame, and obtain the video frame feature map corresponding to each video frame.

[0080] In this embodiment, when feature extraction is performed on the monitoring video, the preset neural network model used can be an efficient convolutional neural network (Efficient Convolutional Neural Network, EfficientNet), such as an EfficientNet-B3 model. The efficient convolutional neural network has the advantages of small parameter quantity, small calculation quantity, high feature extraction precision, and can effectively avoid hardware resource overload, which is consistent with the precision and efficiency balance requirements of the display screen anomaly detection.

[0081] In step S202, the time sequence change features between the continuous video frames are extracted according to the video frame feature map.

[0082] Specifically, the efficient convolutional neural network model includes multiple channels, each of which is used to represent different dimensions of human eye perception color or brightness, such as a 3-channel RGB feature map (corresponding to a red R channel, a green G channel, and a blue B channel, respectively). The video frame feature map includes a video frame feature value corresponding to each spatial position (i.e., pixel coordinates) and each channel. The efficient convolutional neural network model usually uses a squeeze-and-excitation (SE) attention mechanism to solve the problem of uneven importance of different channel features, but the SE attention is a single-frame independent attention mechanism that only processes the static features of a frame and cannot distinguish the feature importance of different space-time regions in the same channel. Moreover, the SE attention has high noise sensitivity and is easily affected by noise (irrelevant interference information) when processing data.

[0083] The embodiment calculates the time sequence change of the continuous video frame feature map based on the video frame feature value corresponding to each spatial position (i.e., pixel coordinates) and each channel of the continuous video frames, extracts the time sequence change features between the continuous video frames, and uses the dynamic correlation between the continuous frames to better identify dynamic abnormalities of the display screen and improve the detection rate of periodic screen flashing (e.g., 3 times per second of brightness sudden change, normal brightness in a single frame, and fluctuation found by comparing continuous frames).

[0084] In step S203, the video frame feature map is divided into multiple region feature maps, and the spatial structure features of each region feature map are extracted; wherein the region feature map corresponds to one display region of the display screen.

[0085] Specifically, the display area of the display screen includes a frame area (i.e., the peripheral area of the display screen), a middle area (i.e., the main display area in the center of the display screen), and a partition display area (i.e., the functional display area of the display screen, such as a sidebar, a notification bar, a navigation bar, a split-screen window area, etc.). When the vehicle-mounted camera monitors the display screen, in addition to the display screen, the physical area outside the display screen (such as the steering wheel and instrument panel around the car screen) is inevitably included, resulting in that the video frame feature map is easy to contain invalid redundant space features, and the abnormal types of the display screen in different display areas are significantly different, such as more light leakage / broken in the frame area, more screen flicker / defects in the middle area, and more display misplacement in the partition display area.

[0086] Therefore, according to the display area of the display screen, the embodiment divides the video frames of the monitoring video into a plurality of non-overlapping pixel areas, each of which can correspond to the physical area outside the display screen, the frame area, the middle area, and the partition display area in the display screen one by one, and generates a region mask corresponding to each pixel area. The region mask is used to mark and distinguish different pixel areas of the video frame. For example, the region mask corresponding to the physical area outside the display screen is "00", the region mask corresponding to the frame area is "01", the region mask corresponding to the middle area is "10", and the region mask corresponding to the partition display area is "11", etc. The generation process of the region mask can refer to the description of related technologies, which will not be described here.

[0087] Further, based on the region mask, the extracted video feature map is divided into a plurality of non-overlapping region feature maps, for example, the feature map part with the region mask "00" is filtered (i.e., the weight is 0), so that only the region feature map corresponding to different display areas (frame area, middle area, and partition display area) of the display screen is extracted, and the background pixel area (such as instrument panel, steering wheel, etc.) in the video feature map is filtered out.

[0088] In step S203, for each region feature map, the pixel brightness value and the texture gradient value at each spatial position (i.e., pixel coordinates) in the region feature map are extracted to obtain the brightness feature and the texture feature of the region feature map, and the spatial structure feature of the region feature map is obtained.

[0089] In some embodiments, the weighted average method can be used to calculate the average brightness of the pixel at each spatial position in the region feature map at the RGB three channels to obtain the pixel brightness value, and the texture gradient value can be extracted based on the image edge detection algorithm (such as the Sobel operator). For details, please refer to the description of related technologies, which will not be described here.

[0090] The embodiment performs spatial structure feature extraction on the region feature map corresponding to each display region, respectively, to strengthen the detection adaptability of different display regions, and uses the weight distribution of the region mask to further perform targeted processing on different pixel regions of the video frame feature map. For example, using the region mask can directly shield background pixels, increase the attention of the model to the pixel region corresponding to the display screen, and perform calculation, thereby reducing the processing amount of more than 50% of invalid pixels, and increasing the inference speed by about 30%. Thus, through region division and region mask structure, the abnormality detection accuracy of different display regions is improved, and the computing power is allocated differently to avoid high computing power consumption of the whole picture.

[0091] In step S204, the time sequence change strength of the time sequence change feature is determined, the structure abnormality strength of each display region is obtained according to the spatial structure feature, and a target display region with a structure abnormality strength greater than an abnormality strength threshold is determined.

[0092] Specifically, the time sequence change strength of the time sequence change feature and the structure abnormality strength of each display region are obtained by performing abnormality analysis on the time sequence change feature and the spatial structure feature.

[0093] Specifically, since the related art focuses on abnormality detection of the whole picture, the misjudgment rate of small abnormalities such as screen corner flickering and small display misalignment is high. In the embodiment, the structure abnormality strength of each display region is compared with an abnormality strength threshold (which can be set according to the actual detection accuracy requirement, and the higher the accuracy requirement, the smaller the abnormality strength threshold), and a target display region with a high structure abnormality strength is selected, so as to perform abnormality detection using the spatial structure feature of the target display region, to ignore feature data with less influence on the detection target, and to improve the detection accuracy of small display regions, avoid screen abnormality detection errors and omissions, and timely detect small screen abnormalities to improve the product use experience of users.

[0094] In step S205, the time sequence change feature and the spatial structure feature corresponding to the target display region are fused based on the time sequence change strength and the structure abnormality strength of the target display region, and the abnormality detection result of the display screen is determined according to the obtained fusion feature.

[0095] Specifically, the time sequence change strength and the structure abnormality strength of the target display region are used to measure the influence of the time sequence change feature and the spatial structure feature corresponding to the target display region on screen abnormality detection, so as to dynamically adjust the fusion ratio of the time sequence change feature and the spatial structure feature corresponding to the target display region during feature fusion, to obtain fusion features more suitable for the actual scene, and to perform abnormality detection of the display screen using the fusion features to obtain the abnormality detection result.

[0096] The display screen abnormality detection method provided in the embodiment is based on the video frame feature map of the monitoring video of the display screen, extracts the time sequence change feature between continuous video frames and the spatial structure feature corresponding to different display regions of the display screen, so as to separately analyze the spatial structure of different display regions. Then, the time sequence change intensity of the time sequence change feature and the structure abnormality intensity of each display region are analyzed, so as to measure the influence degree of the time sequence change feature and the spatial structure feature on the screen abnormality detection. Furthermore, the spatial structure feature of the target display region with high time sequence change feature and structure abnormality intensity is fused, so as to improve the recognition rate of the screen corner region and the micro display abnormality, and obtain the abnormality detection result of the display screen.

[0097] In the embodiment, a display screen abnormality detection method is provided, which can be applied to the mobile terminal such as the vehicle-mounted terminal, the mobile phone, the tablet computer and the like. Figure 3 The flowchart of the display screen abnormality detection method according to the embodiment of the present application is shown in FIG. 2, which includes the following steps: Figure 3

[0098] In step S301, the monitoring video of the display screen is acquired, and the video frame feature map of the monitoring video is extracted. For details, refer to step S201 of the embodiment shown in FIG. 1, which will not be described herein again. Figure 2

[0099] In step S302, the time sequence change feature between continuous video frames is extracted according to the video frame feature map.

[0100] In step S302, the difference feature between every two continuous video frames is calculated according to the video frame feature map, and the time sequence change feature between continuous video frames is obtained according to the difference feature.

[0101] Specifically, the difference feature between every two continuous video frames can be extracted according to the following formula by determining every two continuous corresponding video frame feature maps:

[0102]

[0103] wherein, represents the video frame feature value of the kth video frame feature map at the spatial position and the channel c, represents the video frame feature value of the kth video frame feature map at the spatial position and the channel c, represents the number of video frame feature maps, represents the difference feature. The time sequence change feature between the kth video frame feature map and the th video frame feature map is ​​​​​​​​

[0104] The SE attention adopted by the related technology only calculates the weight for the spatial domain channel of a single frame, cannot perceive the change characteristics of the channel features over time, and is not conducive to capturing the dynamic information between video frames. The embodiment provides input basis in terms of time attention by calculating the differential features between consecutive video frames and collecting the core time sequence information of the inter-frame changes. Through differential feature calculation, a large amount of static redundant information existing in consecutive video frames is filtered out, for example, the differential feature result of the fixed background in screen anomaly recognition is close to 0, thereby effectively reducing the input of static redundant information invalid for screen anomaly detection, making the model accurately focus on the abnormal problem area in the screen, and greatly reducing the robustness problem caused by environmental interference.

[0105] The embodiment provides input basis by calculating the differential features between every two consecutive video frames and filtering out a large amount of static redundant information existing in consecutive video frames, thereby reducing the input of irrelevant static redundant information and improving the robustness of screen anomaly detection.

[0106] In step S303, the video frame feature map is divided into a plurality of region feature maps, and the spatial structure features of each region feature map are extracted; wherein the region feature map corresponds to a different display region of the display screen in one-to-one correspondence. For details, please refer to step S203 of the embodiment shown in Figure 2 The step S203 of the embodiment shown in the above is not repeated here.

[0107] In step S304, the time sequence change strength of the time sequence change feature is determined, the structure anomaly strength of each display region is obtained according to the spatial structure feature, and the target display region with the structure anomaly strength greater than the anomaly strength threshold is determined.

[0108] Specifically, the above step S304 includes:

[0109] In step S3041, the mean square error feature of the time sequence change feature is determined, and the time sequence change strength of the time sequence change feature is calculated according to the comparison relationship between the mean square error feature and the preset mean square error range.

[0110] Specifically, the mean square error feature of the time sequence change feature can be calculated according to the following formula:

[0111]

[0112] wherein, is the mean square error feature at the channel c, is the mean value of all differential features of the channel c, is the height of the video frame feature map, is the width of the video frame feature map.

[0113] Further, the time sequence change intensity of the time sequence change feature at each channel is calculated according to the mean square deviation feature, and the calculation formula is:

[0114]

[0115] wherein, the time sequence change intensity of the time sequence change feature at channel c is represented, and the time sequence change intensity , the minimum value of the preset mean square deviation range is represented, the maximum value of the preset mean square deviation range is represented. The preset mean square deviation range can be set by obtaining the mean square deviation range corresponding to the time sequence change feature of various screens.

[0116] In this embodiment, the mean square deviation feature is used to measure the deviation degree of data from the mean value. By using the mean square deviation feature for the time sequence change intensity calculation of the difference feature of the continuous video frames, the fluctuation characteristics of the difference feature (the dynamic fluctuation degree of data changing over time) can be captured, and the isolated noise of individual video frames (interference signals that exist alone in data, are irrelevant to the surrounding effective information, have a small range, and affect a local part) can be suppressed. In the process of accurately quantifying the time sequence change continuity, local misjudgment can be avoided. Moreover, the calculation of the mean square deviation only involves basic operations such as the difference feature and the mean value of the difference feature, without using complex convolution, matrix multiplication or activation function. The calculation complexity of the single-channel MSE is (N is the number of video frames), and the overall calculation overhead is extremely low, which is completely suitable for scenes with limited computing power such as embedded devices and real-time detection.

[0117] In this embodiment, the mean square deviation feature of the time sequence change feature is calculated to capture the fluctuation characteristics of the time sequence change feature, suppress the isolated noise of individual video frames, avoid local misjudgment, and have extremely low calculation overhead. Therefore, the time sequence change intensity of the time sequence change feature is measured by comparing the mean square deviation feature with the preset mean square deviation range, so as to improve the recognition accuracy of the screen anomaly in combination with the time sequence change intensity.

[0118] In step S3042, the brightness variance feature and the texture variance feature of each display area are obtained according to the brightness feature and the texture feature in the spatial structure feature.

[0119] In some embodiments, for each region feature map, the brightness variance feature of the region feature map can be extracted according to the following formula:

[0120]

[0121] wherein, the brightness variance feature is represented, is the pixel brightness value at the spatial position , and channel c. is the mean value of all pixel luminance values of the region feature map in channel c, is the height of the region feature map, is the width of the region feature map.

[0122] In some embodiments, for each region feature map, a texture variance feature of the region feature map can be extracted according to the following formula:

[0123]

[0124] wherein, denotes the texture variance feature, is the texture gradient value at the spatial position , channel c, is the mean value of all texture gradient values of the region feature map in channel c.

[0125] Step S3043, determining a luminance difference feature between the luminance variance feature of each display region and a preset luminance variance, and determining a texture difference feature between the texture variance feature of each display region and a preset texture variance.

[0126] In some embodiments, for each display region, a luminance difference feature corresponding to the display region can be determined according to the following formula:

[0127]

[0128] wherein, denotes the luminance difference feature, is the preset luminance variance. It should be noted that the preset luminance variance can be set according to a luminance variance benchmark value of a normal screen.

[0129] In some embodiments, for each display region, a texture difference feature corresponding to the display region can be determined according to the following formula:

[0130]

[0131] wherein, denotes the texture difference feature, is the preset texture variance. It should be noted that the preset texture variance can be set according to a texture variance benchmark value of a normal screen.

[0132] Step S3044, calculating a structural anomaly strength of each display region according to the luminance difference feature and the texture difference feature.

[0133]

[0134] wherein, denotes the structural anomaly strength, a weight corresponding to the luminance difference feature, The weight can be set according to the importance of the luminance difference feature.

[0135] In this embodiment, since the luminance feature is a direct indicator of the existence of screen display, the abnormality detection based on the luminance feature can cover the core fault type, and the texture feature is a key dimension describing the structure regularity of the screen content. The variance calculation is performed using the luminance feature and the texture feature, the luminance difference feature and the texture difference feature obtained are used to measure the structure abnormality strength of each display area, the full-scene coverage is realized while supporting mutual verification, the calculation cost is low, and it is more suitable for the lightweight demand of screen abnormality detection.

[0136] Specifically, the display screen will appear a continuous image changing over time when responding to user operations and real-time information updates, and the display information and content elements in the interface fixed area are arranged in order, thus having a significant spatiotemporal distribution regularity. Since the original data features have complex correlations, especially nonlinear correlations and multi-factor interaction relationships, they are often limited by spatial dimensions, and it is generally difficult to effectively depict the relationships in complex spaces through low-dimensional processing. For example, two types of data may completely overlap on a two-dimensional plane and cannot be separated by a straight line, but when mapped to a three-dimensional space, they may exhibit separable spatial distribution patterns. In high-dimensional space, originally mixed key feature information, such as dynamic dependencies in spatiotemporal data and abstract textures in images, will be amplified and decoupled. Complex relationships that are difficult to fit with linear methods, such as nonlinear dependencies between data and dynamic interaction rules, will also be transformed into more manageable forms in high-dimensional space.

[0137] Therefore, the present embodiment maps low-dimensional data to high-dimensional space through luminance feature and texture feature extraction, and mines rules in low-dimensional space that cannot be discovered from a high-dimensional perspective. After extracting these features, the core part of the high-dimensional feature data is compressed by calculating the time series change strength and the structure abnormality strength, so as to capture the most important regularity characteristics in the data, while ignoring the feature data that has less impact on the target, and then through layer-by-layer processing, the original data is converted into structured and generalized abstract relationships, so that the extracted data has spatiotemporal characteristics.

[0138] In step S305, the time series change feature and the spatial structure feature corresponding to the target display area are fused based on the time series change strength and the structure abnormality strength of the target display area, and the abnormality detection result of the display screen is determined according to the obtained fused feature.

[0139] Specifically, the above step S305 includes:

[0140] In step S3051, the time sequence change intensity and the structural abnormality intensity of the target display area are input into the pre-trained nonlinear model to obtain a fusion coefficient; the fusion coefficient is used to represent the importance degree corresponding to the time sequence change feature.

[0141] Specifically, since the time sequence change intensity reflects the time sequence dynamics (such as screen flashing and freezing) of continuous video frames, and the structural abnormality intensity reflects the spatial structural abnormality (such as black screen and screen flashing) of a single frame, the two are not in a simple linear relationship, and therefore the splicing feature is input into the pre-trained nonlinear model to mine the nonlinear relationship between the time sequence abnormality and the spatial structural abnormality, determine the time sequence change intensity and the target association mode of the structural abnormality intensity , and obtain the fusion coefficient used to represent the importance degree corresponding to the time sequence change feature.

[0142] It should be noted that the nonlinear model can be constructed based on a multi-layer perception (MLP), and the nonlinear model is trained to enable it to autonomously learn the optimal association mode of the time sequence change intensity and the structural abnormality intensity under different abnormal scenarios, thereby improving the adaptation ability of the model. The nonlinear model is activated by a function to obtain the fusion coefficient :

[0143]

[0144] wherein, denotes a weight matrix of the function, is a bias term of the function, denotes a linear transformation, and the function can be a nonlinear function used to filter irrelevant features or redundant features. denotes a weight matrix of the activation function, is a bias term of the activation function. , and can be adjusted in the training process of the nonlinear model.

[0145] ​This embodiment further explores the higher-order correlation between the intensity of temporal variation and the intensity of structural anomalies by nesting linear and nonlinear transformations, demonstrating their impact on feature fusion and achieving quantitative output of the dynamic strategy. Compared to traditional fusion methods such as fixed weights and linear weighting, this embodiment generates fusion coefficients through nonlinear models and activation functions. It can dynamically adapt to different scenarios such as time-driven screen anomalies (e.g., flickering, lag), space-driven screen anomalies (e.g., black screen, partial screen distortion), and mixed anomalies (e.g., flickering + screen distortion), achieving non-linear correlation of spatiotemporal features and matching the model's expressive power. Experimental data shows that the detection accuracy of the above method for flickering, black screen, and mixed anomalies is significantly higher than that of a fixed weight (fusion coefficient). =0.5) Increased by 18%, 12%, and 23%.

[0146] Step S3052: Based on the fusion coefficient, the temporal variation features and the spatial structure features corresponding to the target display area are fused to obtain the fused features.

[0147] Specifically, the temporal variation features and the spatial structure features corresponding to the target display area are fused according to the following formula:

[0148]

[0149] in, Indicates spatial location ,aisle Fusion characteristics at the location, This represents the temporal change characteristics corresponding to the current time t (i.e., the temporal change characteristics between the video frame feature map extracted at time t-1 and the video frame feature map extracted at time t). It captured the dynamic changes between consecutive frames; It represents the spatial structural features (including brightness and texture features) of the target display area, capturing the spatial layout, texture and other structural information within a single frame.

[0150] It should be noted that before fusing the temporal variation features and the spatial structure features corresponding to the target display area, the temporal variation features need to be cropped according to the pixel region corresponding to the target display area to ensure... , as well as The corresponding pixel regions are consistent, that is, the spatial structural features and temporal change features corresponding to the target display area are fused to obtain the fused features.

[0151] It should be noted that the fusion coefficient of the time-series variation feature part Coefficients of spatial structural features The sum is 1, which meets the basic mathematical rules, and can reflect the complementary relationship between the two. When the fusion coefficient approaches 1, it indicates that the time sequence change feature has a very obvious influence on the current anomaly detection, and therefore a higher value is assigned to represent the degree of such influence; on the contrary, when the fusion coefficient approaches 0, it indicates that the spatial structure feature (such as the spatial structure of a black screen or a flower screen) is more important, and therefore has a higher weight. As can be seen, the fusion coefficient learned by the nonlinear model can be used for weighted fusion processing according to the characteristics of the input data, and the attention resources can be dynamically allocated according to the real-time scene, so as to achieve the effect of adaptively adjusting the fusion proportion of the time sequence change feature and the spatial structure feature.

[0152] Compared with a model using only a single channel dimension feature, the embodiment can more comprehensively and accurately express the characteristics of the input data, has a more accurate recognition rate, and improves the performance of the model in anomaly detection, and adapts to different scene and task requirements. On the other hand, the value of the fusion coefficient directly reflects the proportion of the time sequence change feature and the spatial structure feature in the final fusion feature, improves the interpretability of the model, and helps the model to understand which type of information is more dependent on when processing different inputs, providing a basis for analysis and optimization of the model.

[0153] The embodiment inputs the time sequence change intensity and the structural anomaly intensity of the target display area into the pre-trained nonlinear model, so that the nonlinear model determines the influence of the time sequence change feature and the spatial structure feature of the target display area on the screen anomaly detection, and outputs the fusion coefficient. In order to adjust the proportion of the time sequence change feature and the spatial structure feature of the target display area in feature fusion by using the fusion coefficient, and improve the adaptability of screen anomaly detection in various scenes.

[0154] Step S3053, according to the obtained fusion feature, determining the anomaly detection result of the display screen.

[0155] In some optional embodiments, the above step S3053 comprises:

[0156] Step a1, compressing the fusion feature to obtain a fixed-length global feature vector.

[0157] Specifically, after obtaining the fusion feature, a global average pooling (GAP) layer is used to compress the fusion feature into a fixed-length global feature vector. The global average pooling layer can perform a global average operation on the fusion feature, compressing the two-dimensional fusion feature into a fixed-length vector, aggregating the global information of the feature map, eliminating the difference in spatial dimensions, greatly reducing the feature dimension, reducing the parameter quantity of the subsequent anomaly classification model, and avoiding overfitting.

[0158] Step a2, inputting the global feature vector into the pre-trained anomaly classification model to obtain an abnormality detection result of the display screen; wherein the abnormality detection result includes a flash screen, a black screen, a garbled screen, a white screen, and a normal screen.

[0159] The embodiment compresses the fusion feature into a fixed-length global feature vector, aggregates the global information of the fusion feature, and uses the global feature vector to determine whether the display screen has abnormality such as flash screen, black screen, garbled screen, and white screen. The global feature vector can effectively reduce the feature dimension and avoid overfitting of the pre-trained anomaly classification model during abnormality detection.

[0160] In some embodiments, the training process of the anomaly classification model includes:

[0161] Step c1, constructing a screen image dataset and a preset model structure.

[0162] Specifically, as shown in Figure 4 , a vehicle-mounted camera or a screen recording device is used to collect screen images containing various states such as normal, garbled screen, black screen, white screen, and flash screen, and cover scenes such as strong light, dim light, and dynamic interface. The size of the screen image is adjusted to a standard image of 512*512 pixels, thereby constructing a screen image dataset. Furthermore, a preset model architecture is constructed based on a fully connected layer (Fully Connected Layer) and a Softmax classifier.

[0163] Step c2, performing a data enhancement operation on the screen image dataset to obtain an enhanced image dataset; the data enhancement operation includes image occlusion, geometric transformation, light adjustment, screen abnormality simulation, and resolution adjustment.

[0164] Exemplarily, again referring to Figure 4The screen image in the screen image data set is subjected to local image occlusion; the screen image is subjected to random rotation (±15°), horizontal flip, scaling (0.8-1.2 times) and other geometric transformations; the screen image is subjected to brightness increase / decrease (±20%), contrast change (±15%), Gaussian noise addition (Gaussian noise is 0.1) and other illumination adjustment; the local area of the screen image is subjected to color jittering or Gaussian blur to simulate screen abnormal states such as screen flashing / white screen transition; the size of the screen image is adjusted to 224x224, 380x380, 456x456 and the like, and the model is trained with screen images of various resolutions.

[0165] The embodiment enhances the robustness of the abnormal classification model by combining one or more of the above data enhancement operations on the screen image data set.

[0166] Step c3, adding a class label to each image in the enhanced image data set; the class label corresponds to the abnormal type of the display screen.

[0167] Specifically, referring again to Figure 4 A class label is labeled for each image in the enhanced image data set, and the class label includes flashing, screen flashing, black screen, white screen and normal screen, corresponding to the abnormal type of the display screen. All image samples are labeled with a class label by an artificial labeling system to form a high-quality multi-modal training data set, thereby effectively improving the generalization ability of the abnormal classification model.

[0168] Step c4, training the preset model structure using the enhanced image data set to obtain a pre-trained abnormal classification model.

[0169] Specifically, as Figure 5 indicated, an iterative optimization training strategy is used to improve the performance of the preset model structure. After initializing the model using the pre-trained model weight, the loss of the predicted class and the class label is calculated by using a cross-entropy loss function, and the parameters are updated by using an AdamW optimizer for backward propagation. The model weight of the preset model structure is saved regularly during the training process, and model pruning and quantization operations are performed after convergence, and finally a pre-trained abnormal classification model is output. The compression ratio of the abnormal classification model is improved by more than 30% and the accuracy loss is less than 1%, which is conducive to realizing lightweight deployment.

[0170] In some embodiments, during the training process of the anomaly classification model, the training framework can use PyTorch, the optimizer selects the AdamW optimizer, and the initial learning rate can be 1e-4. And use the cross-entropy loss function to optimize the classification task, use the official pre-training model weight of EfficientNet for transfer learning, and accelerate the convergence of the model. Through model pruning and INT8 quantization, the model volume is compressed, realizing lightweight deployment and improving the model inference speed (the inference speed reaches more than 30FPS).

[0171] In some embodiments, during the training process of the anomaly classification model, the display screen image of the car machine is collected in real time, the time sequence change feature and the spatial structure feature of different display areas are extracted, and the time sequence change intensity and the structure abnormal intensity are calculated, so as to fuse the time sequence change feature and the spatial structure feature corresponding to the target display area, obtain the fusion feature, input the fusion feature into the anomaly classification model, obtain the probability distribution of the abnormal type of the display screen output by the anomaly classification model, obtain the abnormal detection result, and trigger the alarm according to the target abnormal type corresponding to the maximum probability, prompt the user to repair in time.

[0172] In some embodiments, the pre-trained anomaly classification model is deployed in the cloud after pruning and quantization. The car machine encapsulates the global feature vector as a classification request, sends the classification request to the cloud through a preset calling interface, so that the cloud calls the pre-trained anomaly classification model based on the classification request to obtain the anomaly classification result. The car machine receives the anomaly classification result returned by the cloud to obtain the abnormal detection result of the display screen.

[0173] In some embodiments, taking the car machine as an example, after the training of the anomaly classification model is completed, the pre-trained anomaly classification model is pruned and quantized to construct a lightweight feature extraction network. Subsequently, an inference engineering framework (including a preset neural network model such as EfficientNet-B3 model used for video frame feature extraction, a model framework of time sequence change feature and spatial structure feature, a calculation framework of time sequence change intensity and structure abnormal intensity, and a feature fusion framework) containing a preset calling interface (such as API interface) is built and deployed on the car machine. The pruned and quantized anomaly classification model is deployed to the cloud using PyTorch, so that the car machine realizes remote calling through the preset calling interface, ensuring the real-time inference service capability of low delay and high concurrency.

[0174] Further, the vehicle machine collects a monitoring video of the display screen through a vehicle-mounted camera, calls a deployed inference service for real-time analysis and processing to obtain a global feature vector, the vehicle machine system encapsulates the global feature vector as a classification request, sends the classification request to the cloud through an API interface, and obtains an abnormal classification result (normal screen / flower screen / black screen / white screen / flickering screen corresponding to a probability) returned by the cloud. According to the abnormal type with the largest probability in the abnormal classification result, the vehicle machine triggers a corresponding alarm to remind the user to repair.

[0175] The display screen abnormality detection scheme of the present application is described in detail below with the display screen as an example of a vehicle-mounted screen and in combination with a specific application example.

[0176] The related art relies on manual work and specific adaptation or focuses on a single scene, and the overall detection is based on single-frame static features, which is difficult to effectively identify dynamic display problems. Even the relatively optimal detection method based on image recognition model is difficult to balance accuracy and speed. The attention mechanism used in the related art adopts fixed weight distribution, lacks adaptive processing for time sequence changes (such as dynamic display content updates) and spatial structure characteristics (such as screen corner details) in the vehicle-mounted scene, resulting in a high miss rate for small size abnormalities and weak anti-interference.

[0177] As shown in Figure 6 The present embodiment provides a model architecture diagram for display screen abnormality detection, which adopts a hierarchical image classification architecture to process the screen detection task. After the input image is extracted by the EfficientNet-B3 backbone network to obtain the basic video frame feature map, it is compressed in the channel dimension through the fusion attention module, and then the channel and spatial dual attention enhancement is performed in combination with the fusion attention module to obtain the fusion feature. The fusion feature is compressed into a fixed-length global feature vector by global average pooling, and the global feature vector is mapped by a fully connected layer and then output by a Softmax classifier to obtain the abnormality detection result of the display screen.

[0178] The present application proposes a display screen abnormality detection method based on time sequence channel and spatial structure coupled attention, introduces a dual modeling mechanism of time sequence attention and structure attention, and realizes adaptive fusion of the two attention features through a fusion coefficient, finally realizes accurate detection of vehicle-mounted screen abnormalities. It has been verified that this method can make the accuracy of vehicle-mounted screen abnormality detection reach about 99%, and the average single-frame detection time is ≤30ms, achieving the effect of balancing detection accuracy and efficiency.

[0179] Specifically, the dynamic correlation between consecutive frames is captured by calculating the difference features of adjacent video frames, and the data fluctuation is stabilized by using the mean square error feature, so as to build an efficient time attention structure. The model processed by the time attention module can filter static noise interference, reduce redundant information, reduce the order of magnitude of feature data processing, and significantly improve the robustness and efficiency of dynamic anomaly identification, thereby solving the problem of SE attention failure in time dimension anomaly detection.

[0180] And, according to the abnormal characteristics of different display areas of the car machine screen, feature division and region mask processing are performed, the core fault scene is covered through brightness variance feature and texture variance feature calculation, and a stable spatial attention structure is established to capture spatial level abnormal features. The feature priority of different display areas is distinguished by using this structure, and the problem of missing judgment of local anomalies in related technologies is solved.

[0181] The fusion attention mechanism of the application firstly introduces the time sequence attention mechanism by calculating the difference features of consecutive frames and the time sequence change intensity, and then introduces the structure attention mechanism through spatial feature division and structure abnormality intensity calculation. Then, the adaptive fusion of the two is realized by using the fusion coefficient, while the model interpretability is enhanced, and finally it is inserted into the EfficientNet model. Through the fusion attention module, the spatio-temporal feature extraction ability of the abnormal area of the screen is enhanced, and the diversified adaptation ability of the model under different abnormal scenes is improved.

[0182] Specifically, the nonlinear relationship between time sequence attention and structure attention is mined through a nonlinear model to calculate the fusion coefficient and dynamically realize the adaptive fusion of the two. The abnormal core problem of the display screen is locked through spatio-temporal feature cooperation, avoiding the missed detection of continuous frame abnormal changes and spatial differences by single channel dimension attention, and effectively dealing with the car machine screen detection under actual multi-dimensional mixed abnormal scenes.

[0183] The application realizes the spatio-temporal feature adaptive fusion of screen anomaly detection under different scenes by establishing the spatio-temporal attention and dynamic fusion mechanism, has excellent adaptability and interpretability. Moreover, the accuracy of anomaly detection under each scene is improved, the calculation overhead is reduced by combining the lightweight classification architecture (global average pooling and full connection layer design), and the generalization ability of the model to abnormal types such as screen flashing, black screen and white screen is enhanced by using multi-scene data enhancement strategy, which can not only extract features more accurately and comprehensively, but also support lightweight and efficient completion of multi-classification detection tasks, and has obvious advantages in vehicle-mounted screen anomaly detection.

[0184] Compared with the related art, the average accuracy of the vehicle-mounted screen anomaly detection is improved by 14.9%, and the average single-frame detection time is still maintained within 30 ms, realizing the construction of a vehicle screen anomaly detection model with high precision, high efficiency, and interpretability, which can realize 24-hour uninterrupted detection of the vehicle screen abnormal state, and improve the accuracy of vehicle screen anomaly detection and the intelligent level of automobile cabin testing.

[0185] In the embodiment, a display screen anomaly detection device is also provided, which is used to realize the above-mentioned embodiments and preferred embodiments, and will not be described again. As used below, the term "module" can be a combination of software and / or hardware that realizes a predetermined function. Although the device described in the following embodiments is preferably realized in software, hardware, or a combination of software and hardware is also possible and is conceived.

[0186] The embodiment provides a display screen anomaly detection device, as shown in Figure 7 , comprising:

[0187] The first processing module 701 is configured to acquire a monitoring video of a display screen and extract a video frame feature map of the monitoring video;

[0188] The second processing module 702 is configured to extract a time sequence change feature between continuous video frames according to the video frame feature map;

[0189] The third processing module 703 is configured to divide the video frame feature map into a plurality of region feature maps and extract a spatial structure feature of each region feature map; wherein the region feature map corresponds to a different display region of the display screen one by one;

[0190] The fourth processing module 704 is configured to determine a time sequence change intensity of the time sequence change feature, obtain a structure anomaly intensity of each display region according to the spatial structure feature, and determine a target display region with a structure anomaly intensity greater than an anomaly intensity threshold;

[0191] The fifth processing module 705 is configured to fuse the time sequence change feature and the spatial structure feature corresponding to the target display region based on the time sequence change intensity and the structure anomaly intensity of the target display region, and determine an anomaly detection result of the display screen according to the obtained fusion feature.

[0192] In some optional embodiments, the second processing module 702 is further configured to:

[0193] According to the video frame feature map, a difference feature between every two continuous video frames is calculated;

[0194] According to the difference feature, the time sequence change feature between the continuous video frames is obtained.

[0195] In some optional embodiments, the fourth processing module 704 is further configured to:

[0196] determine a mean square error feature of the timing change feature;

[0197] calculate a timing change strength of the timing change feature according to a comparison relationship between the mean square error feature and a preset mean square error range.

[0198] In some optional embodiments, the fourth processing module 704 is further configured to:

[0199] obtain a luminance variance feature and a texture variance feature of each display area according to the luminance feature and the texture feature in the spatial structure feature;

[0200] determine a luminance difference feature between the luminance variance feature of each display area and a preset luminance variance, and determine a texture difference feature between the texture variance feature of each display area and a preset texture variance;

[0201] calculate a structure abnormality strength of each display area according to the luminance difference feature and the texture difference feature.

[0202] In some optional embodiments, the fifth processing module 705 is further configured to:

[0203] input the timing change strength and the structure abnormality strength of the target display area into a pre-trained nonlinear model to obtain a fusion coefficient; wherein the fusion coefficient is used to represent the importance degree corresponding to the timing change feature;

[0204] fuse the timing change feature and the spatial structure feature corresponding to the target display area according to the fusion coefficient to obtain a fusion feature.

[0205] In some optional embodiments, the fifth processing module 705 is further configured to:

[0206] compress the fusion feature to obtain a fixed-length global feature vector;

[0207] input the global feature vector into a pre-trained anomaly classification model to obtain an abnormality detection result of the display screen; wherein the abnormality detection result includes a flashing screen, a black screen, a blooming screen, a white screen, and a normal screen.

[0208] In some optional embodiments, the training process of the anomaly classification model includes:

[0209] constructing a screen image dataset and a preset model structure;

[0210] performing a data enhancement operation on the screen image dataset to obtain an enhanced image dataset; the data enhancement operation includes image occlusion, geometric transformation, light adjustment, screen abnormality simulation, and resolution adjustment;

[0211] adding a category label to each image in the enhanced image dataset; the category label corresponds to an abnormal type of the display screen;

[0212] training the preset model structure by using the enhanced image dataset to obtain a pre-trained abnormal classification model.

[0213] In some optional embodiments, the pre-trained abnormal classification model is deployed in the cloud after pruning and quantization processing; the fifth processing module 705 is further configured to:

[0214] encapsulating the global feature vector as a classification request;

[0215] sending the classification request to the cloud through a preset calling interface, so that the cloud calls the pre-trained abnormal classification model based on the classification request to obtain an abnormal classification result;

[0216] receiving the abnormal classification result returned by the cloud to obtain an abnormal detection result of the display screen.

[0217] The display screen abnormality detection apparatus provided in the embodiments of the present application can perform the display screen abnormality detection method provided in any of the embodiments of the present application, and has the corresponding function modules and beneficial effects of the execution method. The further function description of the above-mentioned various modules and units is the same as that of the corresponding embodiments, which will not be described here again.

[0218] Figure 8 A structural schematic diagram of an electronic device provided in the embodiments of the present application.

[0219] The following will be specifically described with reference to Figure 8 which shows a structural schematic diagram of an electronic device suitable for implementing the electronic device in the embodiments of the present application. The electronic device can include a processor (such as a central processor, a graphics processor, etc.) 801, which can perform various appropriate actions and processes according to programs stored in a read-only memory (ROM) 802 or programs loaded from a storage 808 into a random access memory (RAM) 803. In the RAM 803, various programs and data required for the operation of the electronic device are also stored. The processor 801, the ROM 802, and the RAM 803 are connected to each other through a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0220] Generally, the following apparatuses can be connected to the I / O interface 805: an input apparatus 806 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output apparatus 807 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage 808 including, for example, a magnetic tape, a hard disk, etc.; and a communication apparatus 809. The communication apparatus 809 can allow the electronic device to communicate with other devices wirelessly or by wire to exchange data. Although not shown, the electronic device can further include a power supply apparatus (such as a battery, a solar cell, etc.) that supplies power to the above-mentioned components.Figure 8 The electronic device is shown with various devices, but it should be understood that all of the shown devices are not required, and that more or fewer devices can alternatively be implemented.

[0221] In particular, the processes described above with reference to the flowcharts can be implemented as a computer software program according to embodiments of the present application. For example, embodiments of the present application include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods illustrated by the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via the communication device 809, or installed from the memory 808, or installed from the ROM 802. When the computer program is executed by the processor 801, the above-described functions defined in the display screen anomaly detection method of embodiments of the present application are performed.

[0222] Figure 8 The electronic device shown is merely an example and should not limit the function and scope of use of embodiments of the present application.

[0223] Embodiments of the present application also provide a computer-readable storage medium, the above-mentioned method according to embodiments of the present application can be implemented in hardware, firmware, or as computer code recordable on a storage medium, or as computer code originally stored in a remote storage medium or non-transitory machine-readable storage medium downloaded through a network and to be stored in a local storage medium, so that the method described herein can be processed by such software stored on a storage medium using a general-purpose computer, a special-purpose processor, or programmable or special-purpose hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only memory, a random access memory, a flash memory, a hard disk or a solid state disk, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that the computer, processor, microprocessor controller or programmable hardware includes a storage component that can store or receive software or computer code, when the software or computer code is accessed and executed by the computer, processor or hardware, the display screen anomaly detection method shown in the above embodiments is implemented.

[0224] Part of the present application can be applied as a computer program product, for example, computer program instructions, when executed by a computer, through the operation of the computer, can invoke or provide the method and / or technical solutions according to the present application. Those skilled in the art should understand that the form of computer program instructions in computer readable medium includes but is not limited to source files, executable files, installation package files and the like, and accordingly, the way of computer program instructions executed by computer includes but is not limited to: the computer directly executes the instructions, or the computer compiles the instructions and then executes the corresponding compiled program, or the computer reads and executes the instructions, or the computer reads and installs the instructions and then executes the corresponding installed program. Here, the computer readable medium can be any available computer readable storage medium or communication medium accessible to the computer.

[0225] Although the embodiments of the present application are described in conjunction with the drawings, various modifications and changes can be made by those skilled in the art without departing from the spirit and scope of the present application, and such modifications and changes fall within the scope defined by the appended claims.

Claims

1. A display screen abnormality detection method characterized by comprising: The method comprises: obtaining a monitoring video of a display screen and extracting a video frame feature map of the monitoring video; calculating a difference feature between each two continuous video frames according to the video frame feature map; obtaining a time sequence change feature between continuous video frames according to the difference feature; the calculation formula of the difference feature is: wherein, represents the video frame feature value of the kth video frame feature map at the spatial position , channel c, represents the video frame feature value of the kth video frame feature map at the spatial position , channel c, is the number of video frame feature maps, is the differential feature; the temporal change feature between the kth video frame feature map and the mth video frame feature map, i.e. ;​​​ dividing the video frame feature map into a plurality of region feature maps, and extracting a spatial structure feature of each region feature map; wherein the region feature map corresponds to a different display region of the display screen one by one, and the display region of the display screen includes a frame area, a middle area and a partition display area; determining a mean square difference feature of the time sequence change feature; calculating a time sequence change intensity of the time sequence change feature according to a comparison relationship between the mean square difference feature and a preset mean square difference range; the calculation formula of the mean square difference feature is: wherein, is the mean square difference feature at channel c, is the mean of all difference features of channel c, is the height of the video frame feature map, is the width of the video frame feature map; the calculation formula of the time sequence change intensity is: wherein, represents a timing variation strength indicating a timing variation characteristic at the channel c, then the timing variation strength , represents a minimum value of the preset mean square error range, represents a maximum value of the preset mean square error range; obtaining a brightness variance feature and a texture variance feature of each display region according to the brightness feature and the texture feature in the spatial structure feature; determining a brightness difference feature between the brightness variance feature of each display region and a preset brightness variance, and determining a texture difference feature between the texture variance feature of each display region and a preset texture variance; calculating a structure abnormality intensity of each display region according to the brightness difference feature and the texture difference feature; the calculation formula of the brightness variance feature is: wherein, represents a luminance variance feature, is the spatial position , the pixel luminance value at channel c, is the mean of all pixel luminance values in channel c of the region feature map, is the height of the region feature map, is the width of the region feature map; the calculation formula of the texture variance feature is: wherein, denotes a texture variance feature, is the mean of all texture gradient values in channel c of the region feature map, is the mean of all texture gradient values in channel c of the region feature map, is the mean of all texture gradient values in channel c of the region feature map, the calculation formula of the brightness difference feature is: wherein, represents a luminance difference feature, is a preset luminance variance; the calculation formula of the texture difference feature is: wherein, represents a texture difference feature, is a preset texture variance; the calculation formula of the structure abnormality intensity is: wherein, represents the structural abnormal intensity, is the weight corresponding to the brightness difference feature; determining a target display region with a structure abnormality intensity greater than an abnormality intensity threshold; fusing the time sequence change feature and the spatial structure feature corresponding to the target display region based on the time sequence change intensity and the structure abnormality intensity of the target display region, and determining an abnormality detection result of the display screen according to the obtained fusion feature; the fusion of the time sequence change feature and the spatial structure feature corresponding to the target display region based on the time sequence change intensity and the structure abnormality intensity of the target display region comprises: inputting the time sequence change intensity and the structure abnormality intensity of the target display region into a pre-trained nonlinear model to obtain a fusion coefficient; wherein the fusion coefficient is used to represent the importance degree corresponding to the time sequence change feature; fusing the time sequence change feature and the spatial structure feature corresponding to the target display region according to the fusion coefficient to obtain a fusion feature.

2. The display screen abnormality detection method according to claim 1, characterized by, determining the abnormality detection result of the display screen according to the obtained fusion feature comprises: compressing the fusion feature to obtain a fixed-length global feature vector; inputting the global feature vector into a pre-trained abnormality classification model to obtain the abnormality detection result of the display screen; wherein the abnormality detection result includes flashing screen, black screen, screen, white screen and normal screen.

3. The display screen abnormality detection method according to claim 2, characterized by, the training process of the abnormality classification model comprises: constructing a screen image data set and a preset model structure; The screen image dataset is subjected to a data enhancement operation to obtain an enhanced image dataset; the data enhancement operation includes image occlusion, geometric transformation, light adjustment, screen anomaly simulation, and resolution adjustment; A category label corresponding to an abnormal type of the display screen is added to each image in the enhanced image dataset; The pre-trained anomaly classification model is trained using the enhanced image dataset.

4. The display screen abnormality detection method according to claim 3, characterized by, The pre-trained anomaly classification model is deployed in the cloud after pruning and quantization processing; the global feature vector is input into the pre-trained anomaly classification model to obtain an abnormal detection result of the display screen, including: The global feature vector is encapsulated as a classification request; The cloud is sent the classification request through a pre-set calling interface, so that the cloud calls the pre-trained anomaly classification model based on the classification request to obtain an anomaly classification result; The cloud returns the anomaly classification result, and the abnormal detection result of the display screen is obtained.

5. A display screen abnormality detection apparatus characterized by comprising: The device includes: A first processing module configured to obtain a monitoring video of a display screen and extract a video frame feature map of the monitoring video; A second processing module configured to calculate a difference feature between every two consecutive video frames based on the video frame feature map, and obtain a time sequence change feature between the consecutive video frames based on the difference feature; The calculation formula of the difference feature is: wherein, represents the video frame feature value of the m-th video frame feature map at the spatial position , channel c, represents the video frame feature value of the m-th video frame feature map at the spatial position , channel c, represents the video frame feature value of the m-th video frame feature map at the spatial position , channel c, is the number of video frame feature maps, is the differential feature; the temporal change feature between the k-th video frame feature map and the m-th video frame feature map, i.e. , and . A third processing module configured to divide the video frame feature map into a plurality of region feature maps and extract a spatial structure feature of each region feature map; the region feature map corresponds to a different display region of the display screen one by one, and the display region of the display screen includes a frame area, a middle area, and a partition display area; A fourth processing module configured to determine a mean square error feature of the time sequence change feature, and calculate a time sequence change intensity of the time sequence change feature based on a comparison relationship between the mean square error feature and a pre-set mean square error range; The calculation formula of the mean square error feature is: wherein, is the mean of all difference features at channel c, is the mean of all difference features at channel c, is the height of the video frame feature map, is the width of the video frame feature map; The calculation formula of the time sequence change intensity is: wherein, represents a timing variation strength indicating a timing variation characteristic at the channel c, then the timing variation strength , represents a minimum value of the preset mean square error range, represents a maximum value of the preset mean square error range; The fourth processing module is further configured to obtain a brightness variance feature and a texture variance feature of each display region based on a brightness feature and a texture feature in the spatial structure feature, determine a brightness difference feature between the brightness variance feature of each display region and a pre-set brightness variance, and determine a texture difference feature between the texture variance feature of each display region and a pre-set texture variance; and calculate a structure abnormal intensity of each display region based on the brightness difference feature and the texture difference feature; The calculation formula of the brightness variance feature is: wherein, represents a luminance variance feature, is the mean of all pixel luminance values in the channel c, is the mean of all pixel luminance values in the channel c, is the mean of all pixel luminance values in the channel c, is the height of the region feature map, is the width of the region feature map; The calculation formula of the texture variance feature is: wherein, denotes a texture variance feature, is the mean of all texture gradient values in channel c of the region feature map, is the mean of all texture gradient values in channel c of the region feature map, is the mean of all texture gradient values in channel c of the region feature map, The calculation formula of the brightness difference feature is: wherein, represents a luminance difference feature, is a preset luminance variance; The calculation formula of the texture difference feature is: wherein, represents a texture difference feature, is a preset texture variance; The calculation formula of the structure abnormal intensity is: wherein, represents the structural abnormal intensity, is the weight corresponding to the brightness difference feature; The fourth processing module is further configured to determine a target display region with a structure abnormal intensity greater than an abnormal intensity threshold; A fifth processing module configured to fuse the time sequence change feature and the spatial structure feature corresponding to the target display region based on the time sequence change intensity and the structure abnormal intensity of the target display region, and determine an abnormal detection result of the display screen based on the obtained fused feature; The fifth processing module is further configured to: inputting the time-series change intensity and a structure anomaly intensity of the target display area into a pre-trained nonlinear model to obtain a fusion coefficient; wherein the fusion coefficient is used to represent an importance degree corresponding to the time-series change feature; fusing the time-series change feature and a spatial structure feature corresponding to the target display area according to the fusion coefficient to obtain a fusion feature.

6. An electronic device, comprising: comprising: a memory and a processor, which are in communication connection with each other, and the memory stores computer instructions; the processor executes the computer instructions to perform the display screen anomaly detection method in any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer instructions, and the computer instructions are used to make the computer execute the display screen anomaly detection method in any one of claims 1 to 4.

8. A computer program product, characterised in that, comprising computer instructions, and the computer instructions are used to make the computer execute the display screen anomaly detection method in any one of claims 1 to 4.

Images