Display exception processing method, edge collaboration system, electronic equipment and storage medium

Abstract

The invention provides a display exception processing method, an edge collaboration system, electronic equipment and a storage medium, and relates to the technical field of computers. Comprising the steps that image data of a current image frame in a frame buffer area are collected in real time, and the image data comprise pixel information of a plurality of pixel points; performing detection processing on the image data based on a pulse neural network to obtain image abnormal data; determining a first abnormal index and a second abnormal index according to the image abnormal data; and according to the first exception index and the second exception index, determining an exception repair mode, and executing exception repair based on the exception repair mode. The image data of the obtained image frame is subjected to anomaly detection processing through the pulse neural network, a retina neural mechanism can be simulated to accurately recognize display anomaly, and the anomaly detection accuracy and efficiency are improved; and the exception repair mode is determined by combining the two indexes of the exception pixel ratio and the exception duration, so that the exception repair accuracy can be improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and more specifically, to a display exception handling method, an edge collaboration system, an electronic device, and a storage medium. Background Art

[0002] In today's digital age, with the rapid development of various graphics-intensive applications such as artificial intelligence, virtual reality, big data visualization, and high-end games, graphics card driver management and display exception handling technologies are facing the dual challenges of surging computing power demand and system reliability requirements.

[0003] Currently, graphics drivers are typically statically loaded. When the Xorg service (an essential component of the X Window System) starts, the specified driver is loaded through the xorg.conf configuration file. Dynamic monitoring of the graphics card's display status is not performed during operation. If display anomalies occur, the xorg.conf configuration file must be modified and the Xorg service manually restarted to load a new graphics driver.

[0004] The above method has a delayed response to graphics card anomalies and has poor compatibility in multi-GPU scenarios. Summary of the Invention

[0005] The purpose of this application is to address the deficiencies in the above-mentioned prior art and provide a display anomaly processing method, an edge collaboration system, an electronic device and a storage medium, so as to improve the recognition accuracy and efficiency of image display anomalies, and at the same time improve the accuracy of anomaly repair.

[0006] To achieve the above objectives, the technical solutions adopted in the embodiments of the present application are as follows:

[0007] In a first aspect, an embodiment of the present application provides a method for handling display exceptions, comprising:

[0008] Acquire image data of a current image frame in a frame buffer in real time, wherein the image data includes pixel information of a plurality of pixels;

[0009] Performing detection processing on the image data based on a pulse neural network to obtain image abnormality data, wherein the image abnormality data includes: an image abnormality type and abnormal pixel information under the image abnormality type;

[0010] Determining a first abnormality indicator and a second abnormality indicator based on the image abnormality data, wherein the first abnormality indicator is used to characterize the proportion of abnormal pixels in a spatial dimension, and the second abnormality indicator is used to characterize the duration of the abnormality;

[0011] An abnormality repair method is determined according to the first abnormality indicator and the second abnormality indicator, and abnormality repair is performed based on the abnormality repair method.

[0012] Optionally, the real-time acquisition of image data of the current image frame in the frame buffer includes:

[0013] During the process of starting and running the system desktop program, the image data of the current image frame in the frame buffer is captured in real time through the library interface.

[0014] Optionally, the detecting and processing the image data based on the spiking neural network to obtain image abnormality data includes:

[0015] Performing data encoding on pixel information of each pixel in the image data to obtain an input pulse sequence corresponding to each pixel;

[0016] The input pulse sequence corresponding to each pixel point is input into the pulse neural network for pulse signal propagation and hierarchical analysis to obtain the image abnormality data.

[0017] Optionally, the performing data encoding on pixel information of each pixel in the image data to obtain an input pulse sequence corresponding to each pixel includes:

[0018] Determine the pulse emission rate corresponding to each pixel based on the brightness and color information of each pixel;

[0019] Generate multiple pulse generation times according to the pulse emission rate corresponding to each pixel point, where each pulse generation time is used to indicate the specific moment when a pulse of the pixel point in the pulse sequence of the current image frame occurs;

[0020] According to the pulses emitted by each pixel at each pulse generation time, an input pulse sequence corresponding to each pixel is generated.

[0021] Optionally, inputting the input pulse sequence corresponding to each pixel point into the pulse neural network for pulse signal propagation and hierarchical analysis to obtain the image abnormality data includes:

[0022] The input pulse sequence corresponding to each pixel point is input into the pulse neural network, processed and propagated in sequence through each layer of neurons in the pulse neural network, and the image abnormality data is determined according to the output pulse sequence of the terminal neuron.

[0023] Optionally, inputting the input pulse sequence corresponding to each pixel point into the spiking neural network, sequentially processing and propagating the pulse sequence through each layer of neurons in the spiking neural network, and determining the image abnormality data based on the output pulse sequence of the terminal neuron includes:

[0024] Determine the membrane potential change information of each neuron in the current layer based on the input pulse sequence corresponding to each pixel point and the synaptic weight between the pixel point and each neuron in the current layer to which the signal is transmitted;

[0025] According to the membrane potential change information of each neuron in the current layer, the output pulse sequence of each neuron in the current layer is generated, and the output pulse sequence of each neuron in the current layer is propagated to the corresponding neurons in the next layer for processing until the output pulse sequence of each neuron at the end is obtained;

[0026] The image abnormality data is determined according to the output pulse sequence of each terminal neuron.

[0027] Optionally, determining the image abnormality data according to the output pulse sequence of each terminal neuron includes:

[0028] According to the output pulse sequence of each terminal neuron, the frequency information corresponding to each terminal neuron is determined;

[0029] According to the frequency information corresponding to each terminal neuron, the position information of each terminal neuron and the membrane potential information of each terminal neuron, the image abnormality type and the abnormal pixel information under the image abnormality type are determined. The image abnormality type includes: flower screen, black screen or static abnormality.

[0030] Optionally, determining a first abnormality indicator and a second abnormality indicator according to the image abnormality data includes:

[0031] Determining the number of abnormal pixels according to the abnormal pixel information under the abnormal type of the image;

[0032] Determining the first abnormality indicator according to the number of abnormal pixels and the total number of pixels in the current image frame;

[0033] The second abnormality indicator is determined according to the current time corresponding to the current image frame and the abnormality start time, wherein the abnormality start time is the time when the image abnormality first occurs, and the image abnormality continues to occur between the abnormality start time and the current time.

[0034] Optionally, determining an abnormality repair method according to the first abnormality indicator and the second abnormality indicator includes:

[0035] If the proportion represented by the first abnormality indicator is greater than or equal to the proportion threshold and the abnormality duration represented by the second abnormality indicator is greater than or equal to the duration threshold, determining that the abnormality repair method is to switch the graphics card device;

[0036] Otherwise, it is determined that the repair method is local abnormal pixel repair.

[0037] Optionally, performing abnormality repair based on the abnormality repair method includes:

[0038] If the abnormality repair method is to switch the graphics card device, then suspend the rendering task of the current graphics card device;

[0039] Determine the target standby graphics card device according to the priority information of each standby graphics card device;

[0040] According to the information of the target standby graphics card device, modify the configuration information of the graphics card associated service and modify the dynamic linker, and load the target standby graphics card device from a specified directory of the system;

[0041] Reinitialize the rendering task, and call the atomic submit function to submit the status information of the target standby graphics card device to the graphics processor.

[0042] Optionally, performing abnormality repair based on the abnormality repair method includes:

[0043] If the abnormality repair method is local abnormal pixel repair, then the first interpolation pixel point and the second interpolation pixel point corresponding to each abnormal pixel point are determined according to the pixel coordinate information and pixel offset information of each abnormal pixel point;

[0044] Determine restored pixel information corresponding to each abnormal pixel point based on pixel information of a first interpolated pixel point corresponding to each abnormal pixel point and pixel information of a second interpolated pixel point corresponding to each abnormal pixel point;

[0045] Re-render the current image frame based on the repaired pixel information corresponding to each abnormal pixel point.

[0046] Optionally, the current graphics card device is determined by:

[0047] Monitor the currently running rendering thread and determine the current graphics card device according to the mapping relationship between the rendering thread and the graphics card device.

[0048] Optionally, before determining the current graphics card device according to the mapping relationship between the rendering thread and the graphics card device, the method further includes:

[0049] During the process of each rendering thread calling the context association function to associate the open image library with the frame buffer, each rendering thread is monitored, and a mapping relationship between the identifier of each rendering thread and the identifier of the graphics card device used by each rendering thread is recorded.

[0050] In a second aspect, an embodiment of the present application further provides an edge collaboration system, comprising: a master node device and at least one edge node device; a pulse neural network is deployed on each edge node device;

[0051] The edge node device is used to perform the method steps related to abnormality detection and local abnormal pixel repair in the display abnormality processing method provided in the first aspect;

[0052] The master node device is used for global coordination.

[0053] In a third aspect, an embodiment of the present application provides an electronic device comprising: a processor, a storage medium and a bus, wherein the storage medium stores machine-readable instructions executable by the processor. When the electronic device is running, the processor and the storage medium communicate through the bus, and the processor executes the machine-readable instructions to implement the display exception handling method provided in the first aspect.

[0054] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon. When the computer program is run by a processor, the display exception handling method provided in the first aspect is executed.

[0055] The beneficial effects of this application are:

[0056] The present application provides a display abnormality processing method, an edge collaboration system, an electronic device, and a storage medium, including: real-time acquisition of image data of the current image frame in the frame buffer, the image data including pixel information of multiple pixels; detecting and processing the image data based on a pulse neural network to obtain image abnormality data; determining a first abnormality indicator and a second abnormality indicator based on the image abnormality data; determining an abnormality repair method based on the first abnormality indicator and the second abnormality indicator, and performing abnormality repair based on the abnormality repair method. This method uses a pulse neural network to perform abnormality detection processing on the image data of the image frame obtained from the frame buffer, which can simulate the retinal neural mechanism to accurately identify display abnormalities and realize millisecond-level dynamic analysis, thereby improving the accuracy and efficiency of abnormality detection; and determining the abnormality repair method by combining the dual indicators of abnormal pixel ratio and abnormal duration makes the abnormality repair method more in line with the current abnormal scenario, thereby improving the accuracy of abnormality repair.

[0057] By adopting a distributed edge collaboration architecture, it can effectively support hierarchical repair methods such as local repair and global graphics card device switching in multi-graphics processor scenarios, reducing system load. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without creative work.

[0059] Figure 1 A schematic diagram of the hardware architecture of an exception handling system provided in an embodiment of the present application;

[0060] Figure 2 A flowchart of a display exception handling method provided in an embodiment of the present application;

[0061] Figure 3 A flowchart of another display exception handling method provided in an embodiment of the present application;

[0062] Figure 4 A flowchart of another display exception handling method provided in an embodiment of the present application;

[0063] Figure 5 A flowchart of another display exception handling method provided in an embodiment of the present application;

[0064] Figure 6 A flowchart of another display exception handling method provided in an embodiment of the present application;

[0065] Figure 7 A flowchart of another display exception handling method provided in an embodiment of the present application;

[0066] Figure 8 A flowchart of another display exception handling method provided in an embodiment of the present application;

[0067] Figure 9 A flowchart of another display exception handling method provided in an embodiment of the present application;

[0068] Figure 10 A schematic diagram of an edge collaboration system architecture provided in an embodiment of the present application;

[0069] Figure 11 A schematic diagram of an edge collaborative processing flow provided in an embodiment of the present application;

[0070] Figure 12 A schematic diagram of a display exception handling device provided in an embodiment of the present application;

[0071] Figure 13 A schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0072] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It should be understood that the drawings in the present application only serve the purpose of illustration and description and are not used to limit the scope of protection of the present application. In addition, it should be understood that the schematic drawings are not drawn to scale. The flowcharts used in this application illustrate the operations implemented according to some embodiments of the present application. It should be understood that the operations of the flowcharts can be implemented out of sequence, and steps without logical context can be reversed or implemented simultaneously. In addition, those skilled in the art, under the guidance of the contents of this application, can add one or more other operations to the flowchart, or remove one or more operations from the flowchart.

[0073] In addition, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. The components of the embodiments of the present application generally described and shown in the drawings here can be arranged and designed in various configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the claimed application, but merely represents selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of the present application.

[0074] It should be noted that the term "comprising" will be used in the embodiments of the present application to indicate the existence of the features declared thereafter, but does not exclude the addition of other features.

[0075] With the advancement of computer technology, graphics cards have become increasingly important in computer systems. In the gaming world, players have higher expectations for game graphics quality and frame rates, such as those seeking 4K resolution and a smooth gaming experience at high frame rates. This requires powerful graphics processing capabilities from graphics cards, and graphics card drivers optimize performance for smoother, more detailed graphics. In professional graphics applications, such as animation, architectural design, and film and television post-production, designers need to work with complex 3D models and high-precision images. Graphics card drivers ensure efficient collaboration between professional software and graphics cards, improving work efficiency.

[0076] Display anomalies, such as screen distortion, black screens, freezing, and flickering, can severely impact the user experience. In daily office work, these issues can cause garbled document displays and unclear video conference images, reducing productivity. For gamers, they can disrupt immersion and even prevent gameplay. In professional scenarios requiring extremely high display quality, such as medical imaging and financial trading quotes, display anomalies can have serious consequences.

[0077] The currently implemented graphics card drivers and anomaly detection methods are as follows:

[0078] Existing solution 1: static driver loading mechanism

[0079] Implementation: When the Xorg service (the server-side implementation of the X Window System) is started, the specified driver is loaded through the xorg.conf configuration file. There is no dynamic monitoring during operation.

[0080] Typical defects: delayed fault response: manual restart of the X service is required (average recovery time is 8 minutes and 23 seconds); poor compatibility: the driver conflict rate is as high as 32% in multi-GPU (Graphics Processing Unit) scenarios.

[0081] Existing solution 2: visual detection based on CNN (Convolutional Neural Network).

[0082] Implementation method: Use convolutional neural networks such as YOLOv3 (real-time target detection algorithm) to periodically capture screen frames (sampling interval ≥ 200ms) and use classification models to determine display anomalies.

[0083] Typical defects: High computing latency: single frame processing takes 217±15ms; Excessive power consumption: Continuous operation power consumption >8W, resulting in a 37% decrease in mobile device battery life.

[0084] Existing Solution 3: Multi-GPU Polling Monitoring

[0085] Implementation: Poll each GPU's status register via the PCIe (Peripheral Component Interconnect Express, a high-speed serial computer expansion bus standard) bus (cycle 1s), and use a majority voting mechanism to switch the master device.

[0086] Typical defects: High misjudgment rate: Instantaneous state jitter leads to 23.6% false switching (laboratory stress test data). Poor scalability: The bus load increases by 18% for each additional GPU.

[0087] Based on the defects of the existing technology, this solution provides a display exception handling method, which uses a spiking neural network (SNN) to perform millisecond-level dynamic analysis of the screen frame buffer (framebuffer), simulates the retinal neural mechanism to accurately identify screen abnormalities, and has a detection delay of ≤8ms. The abnormal pixel space ratio and the abnormal duration are combined to make a joint judgment of the abnormal type, and the abnormal repair method corresponding to the abnormal type is used to repair the abnormality, thereby improving the accuracy of abnormal repair. Distributed node deployment is adopted to support hierarchical processing of local repair (interpolation compensation) and global drive hot switching (interruption ≤200ms) in multi-GPU scenarios, reducing system load.

[0088] Figure 1 This is a schematic diagram of the hardware architecture of an exception handling system provided in an embodiment of the present application. Figure 1 The hardware system includes: field programmable gate array, processing chip, switching matrix and graphics card group.

[0089] The spiking neural network processing layer can be deployed in the field programmable gate array. In addition, the front-end acquisition layer can also be deployed above the spiking neural network processing layer.

[0090] Front-end acquisition layer: Real-time capture of the screen frame buffer through the HDMI 2.0 (a version of the High-Definition Multimedia Interface HDMI) interface, supporting 4K@60Hz input (implemented by the ADV7611 (receiver chip) decoding chip).

[0091] Spiking neural network processing layer: It is a 4-layer spiking neural network (input layer → spike convolution layer → temporal integration layer → decision layer) deployed in the above-mentioned field programmable gate array, which completes millisecond-level anomaly detection (single frame processing ≤8ms).

[0092] The processing chip is equipped with a driver management layer, which is used to implement exception recovery and PCIe hot swap control.

[0093] The switching matrix can be a PCIe 4.0 switching matrix, which is used to implement dynamic routing switching of multiple GPUs and supports 8 ports.

[0094] The graphics card group is used to perform rendering tasks and supports OpenGL 3.3 (Open Graphics Library, an application programming interface that is cross-programming language and cross-platform). For example, after the processing chip determines that there is a display anomaly, it can fix the anomaly by switching the graphics card in the graphics card group.

[0095] Figure 2A flow chart of a display exception handling method provided in an embodiment of the present application; can be applied to electronic devices, the system architecture of which can be, for example, Figure 1 As shown in the architecture. Figure 2 As shown, the method may include:

[0096] S101 . Acquire image data of a current image frame in a frame buffer in real time, where the image data includes pixel information of a plurality of pixels.

[0097] The frame buffer temporarily stores the current screen frame's image data; it stores information such as the color and depth of each pixel in a frame. This information is stored in an orderly fashion based on the screen's resolution and pixel arrangement. When a display device displays an image, the screen controller reads this data from the frame buffer and outputs it pixel by pixel to the screen, forming a visible image.

[0098] The frame buffer can be collected in real time to obtain image data of the current image frame in real time. In this embodiment, the image data may include pixel information of all pixels constituting the current image frame. The pixel information may include color information, brightness information, etc. of the pixels.

[0099] S102: Detect and process the image data based on a pulse neural network to obtain image abnormality data.

[0100] Image abnormality data includes: image abnormality type and abnormal pixel information under the image abnormality type.

[0101] Because spiking neural networks mimic the information processing methods of biological neurons, transmitting and processing information in the form of pulse trains, this approach is closer to the image processing mechanism of biological visual systems. They can more naturally process complex information in images and may be more adaptable to detecting abnormal data. Abnormal data in images often exhibit specific patterns or changes in the time dimension. Spiking neural networks can naturally process time series information. By modeling the temporal dynamics of pulse trains, they can capture subtle changes in image data over time, thereby more accurately detecting abnormal data.

[0102] Optionally, the collected image data may be input into a pulse neural network for detection processing to obtain image abnormality data.

[0103] In this embodiment, the acquired image abnormality data may include: image abnormality type and abnormal pixel information associated with the image abnormality type. The image abnormality type may be, for example, a distorted screen or a black screen; and the abnormal pixel information associated with the image abnormality type may be, for example, the number of abnormal pixels.

[0104] S103: Determine a first abnormality indicator and a second abnormality indicator according to the image abnormality data.

[0105] The first anomaly indicator is used to characterize the proportion of abnormal pixels in the spatial dimension, and the second anomaly indicator is used to characterize the duration of the anomaly.

[0106] According to the obtained image abnormality data, the first abnormality index and the second abnormality index can be determined respectively. In this embodiment, the abnormal pixel ratio information and the abnormality duration can be calculated respectively based on the image abnormality data.

[0107] The abnormal pixel ratio information refers to the ratio of abnormal pixels to total pixels in the current frame image, and the abnormal duration refers to the time from the start time of the abnormality to the end time of the abnormality.

[0108] S104: Determine an abnormality repair method according to the first abnormality indicator and the second abnormality indicator, and perform abnormality repair based on the abnormality repair method.

[0109] A dual-indicator judgment can be performed by combining the first and second abnormality indicators to determine the abnormality repair method. This dual-indicator judgment method can improve the accuracy of the judgment results, allowing the selection of an abnormality repair method that is more suitable for the current abnormality scenario. This achieves targeted abnormality repair and improves the accuracy of the abnormality repair results.

[0110] In summary, the display abnormality processing method provided by this embodiment includes: real-time acquisition of image data of the current image frame in the frame buffer, the image data including pixel information of multiple pixels; detecting and processing the image data based on a pulse neural network to obtain image abnormality data; determining a first abnormality indicator and a second abnormality indicator based on the image abnormality data; determining an abnormality repair method based on the first abnormality indicator and the second abnormality indicator, and performing abnormality repair based on the abnormality repair method. This method uses a pulse neural network to perform real-time abnormality detection and processing on the image data of the image frame obtained from the frame buffer, which can simulate the retinal neural mechanism to accurately identify display abnormalities and realize millisecond-level dynamic analysis, thereby improving the accuracy and efficiency of abnormality detection; and determining the abnormality repair method by combining the dual indicators of abnormal pixel ratio and abnormal duration makes the abnormality repair method more in line with the current abnormal scenario, thereby improving the accuracy of abnormality repair.

[0111] Optionally, in step S101 , collecting image data of the current image frame in the frame buffer in real time may include: capturing image data of the current image frame in the frame buffer in real time through a library interface during startup and operation of the system desktop program.

[0112] In one possible implementation, after the system desktop program is started, the image data of the current image frame in the frame buffer can be captured in real time during the running process, because the desktop program uses Opengl rendering, that is, it uses the graphics card driver.

[0113] Optionally, you can use the libdrm interface to capture the frame buffer data of / dev / dri / card0 in real time. libdrm is an open source userspace library that provides a userspace interface for the kernel's DRM (Direct Rendering Manager) subsystem, allowing userspace programs to interact with the kernel's DRM subsystem.

[0114] / dev / dri / card0 is one of the paths of the DRM device file in the Linux system. By accessing the device file / dev / dri / card0, you can directly interact with the graphics card and use the hardware acceleration function of the graphics card to perform graphics rendering, video decoding and other operations.

[0115] Figure 3 A flowchart of another display abnormality processing method provided in an embodiment of the present application; optionally, in step S102, detecting and processing image data based on a spiking neural network to obtain image abnormality data may include:

[0116] S201 , performing data encoding on pixel information of each pixel in the image data to obtain an input pulse sequence corresponding to each pixel.

[0117] The input data of the spiking neural network is usually a pulse sequence. Therefore, the pixel information of each pixel in the image data can be first encoded to obtain the input pulse sequence corresponding to each pixel.

[0118] S202 , inputting the input pulse sequence corresponding to each pixel point into the pulse neural network for pulse signal propagation and hierarchical analysis to obtain image abnormality data.

[0119] The input pulse sequence corresponding to each pixel point can be input into the spiking neural network together. In the spiking neural network, the pulse sequence will be propagated and processed through each layer of neurons in the network to eventually convert the input pulse sequence into a sparse pulse pattern, realize efficient spatiotemporal feature encoding, and obtain image abnormality data.

[0120] Figure 4 A flowchart of another display abnormality processing method provided in an embodiment of the present application is provided. Optionally, in step S201, data encoding is performed on pixel information of each pixel in the image data to obtain an input pulse sequence corresponding to each pixel, which may include:

[0121] S301 : Determine the pulse emission rate corresponding to each pixel point according to the brightness information and color information of each pixel point.

[0122] The process of encoding pixels into pulse sequences is mainly achieved by mapping the brightness and color information of the pixels to the pulse emission rate. First, the pulse emission rate λ corresponding to the i-th pixel can be calculated using the following formula: i :

[0123] λ i =αI i +βR i +γG i +δB i

[0124] Among them, I i Indicates the brightness of the pixel; R i , G i 、B i Represents the color information of the three channels of red, green and blue respectively; λ i represents the pulse emission rate; α, β, γ, and δ are weight coefficients used to adjust the proportion of brightness and color information in the pulse sequence.

[0125] The pulse rate refers to the frequency at which a neuron generates action potentials (i.e., pulses) per unit time, and is usually measured in Hertz (Hz). For example, if a neuron generates 10 action potentials in 1 second, its pulse rate is 10 Hz.

[0126] The calculated pulse firing rate will further affect the generation of the pulse train.

[0127] S302 : Generate multiple pulse generation times according to the pulse emission rate corresponding to each pixel point.

[0128] A pulse generation time is used to indicate the specific moment when a pulse occurs at a pixel point in the pulse sequence of the current image frame.

[0129] Alternatively, the pulse train generation formula can be expressed as Among them, t i,k Indicates the specific time when the kth pulse signal occurs in the pulse sequence generated by the i-th pixel. i,k The pulse firing rate λ i Control generation, i.e., the generated pulse firing rate λ i K times can be randomly generated, each time is a t i,k , each t i,k It can be called a pulse generation time.

[0130] S303 , generating an input pulse sequence corresponding to each pixel point according to the pulses emitted by each pixel point at each pulse generation time.

[0131] δ(tt i,k ) means that at time t i,k The pulses emitted at each pulse generation time can constitute the input pulse sequence corresponding to pixel i.

[0132] For example, the first pulse signal occurs at t i,1 , the second pulse signal is sent at t i,2 ..., the time when the kth pulse signal occurs is t i,k , time t i,1 The pulse emitted is δ(tt i,1 ), time t i,2 The pulse emitted is δ(tt i,2 )…, time t i,k The pulse emitted is δ(tt i,k ), then the input pulse sequence S obtained after pixel i is encoded i (t) is: {δ(tt i,1 ),δ(tt i,2 )…,δ(tt i,k )}.

[0133] In one implementation, the three-channel color information of each pixel can be encoded as a whole according to the above method; in another implementation, the three-channel color information of each pixel can be encoded separately to obtain a pulse sequence corresponding to each channel.

[0134] The above-mentioned method adopts frequency coding to encode the pixels. In practical applications, time coding can also be adopted to map the pixel value of the pixel to the time point when the pulse occurs to encode the pixel.

[0135] Optionally, in step S202, the input pulse sequence corresponding to each pixel point is input into the pulse neural network for pulse signal propagation and hierarchical analysis to obtain image abnormality data, which may include: inputting the input pulse sequence corresponding to each pixel point into the pulse neural network, processing and propagating it in sequence through each layer of neurons in the pulse neural network, and determining the image abnormality data based on the output pulse sequence of the terminal neuron.

[0136] The input pulse sequences corresponding to each pixel are sequentially fed into the spiking neural network. Starting from the network's input layer, the pulse sequences are sequentially received, integrated, processed, and propagated to the next layer of neurons according to pre-set connection weights and neuron firing rules. During signal propagation, neurons in each layer gradually extract image features through mechanisms such as membrane potential accumulation and pulse firing, simulating the information processing process of biological neurons. Finally, the output pulse sequences generated by neurons in the terminal output layer of the spiking neural network are compared with the pulse firing pattern characteristics corresponding to different display anomalies to identify image anomalies.

[0137] Figure 5 A flowchart of another display anomaly processing method provided in an embodiment of the present application is provided. Optionally, in the above steps, the input pulse sequence corresponding to each pixel point is input into the spiking neural network, processed and propagated sequentially through each layer of neurons in the spiking neural network, and image anomaly data is determined based on the output pulse sequence of the terminal neuron. This may include:

[0138] S401. Determine the membrane potential change information of each neuron in the current layer according to the input pulse sequence corresponding to each pixel point and the synaptic weight between the pixel point and each neuron in the current layer to which the pulse is transmitted.

[0139] Spiking neural networks primarily consist of an input layer, hidden layers, and an output layer. Each layer can be composed of multiple neurons, and neurons in each layer are connected by synapses. Input layer neurons typically do not perform complex computations; they simply pass the input spike train to the hidden layer. Hidden layers can have one or more layers, and neurons in these layers process and transform the input spike train. Neurons are connected through synapses, each with different weights that determine the strength of the spike train. Based on the input spike train received, neurons in the hidden layer integrate the membrane potential and make spike decisions according to their own neuronal models, passing the processed spike train to the next layer. The output spike train of neurons in the output layer represents the network's processing of the input information. For example, in image classification tasks, the spike pattern of output layer neurons can indicate the image's category.

[0140] Alternatively, the state of each neuron can be represented by its membrane potential, and the change in membrane potential can be expressed by the following formula:

[0141]

[0142] Among them, S i (t) is the input pulse sequence mentioned above, V j (t) is the membrane potential of the jth neuron at time t, τ is the time constant, w jiis the synaptic weight between the i-th input and the j-th neuron.

[0143] By calculating the dynamic changes of the neuron membrane potential, it is ultimately decided whether to trigger the output pulse.

[0144] S402. Generate an output pulse sequence for each neuron in the current layer based on the membrane potential change information of each neuron in the current layer, and propagate the output pulse sequence for each neuron in the current layer to the corresponding neurons in the next layer for processing until the output pulse sequence for each neuron at the end is obtained.

[0145] For a neuron, when the membrane potential V j (t) exceeds the preset threshold V th When the neuron fires a pulse, the membrane potential is reset and immediately reset to its resting value V reset After that, the membrane potential starts to accumulate again, and when V j (t) accumulates again to more than V th , another pulse is emitted.

[0146] For a neuron, after processing the input pulse sequence, the obtained output pulse sequence can be composed of multiple pulses emitted by the neuron.

[0147] That is, when the membrane potential V j (t) accumulates to more than V th When the neuron generates a pulse to send to the next layer of neurons, the membrane potential is reset to the resting value V reset For example, after a period of weighted pulse accumulation, V j (t) reaches the threshold V th , the neuron emits a pulse, which will be transmitted along the connection between it and the output layer neurons. If the membrane potential does not reach the threshold, the neuron will not emit a pulse and continue to accumulate the input pulse sequence.

[0148] It is worth noting that the input pulse sequence and the output pulse sequence are not one-to-one corresponding. After processing by each layer of neurons, the output pulse sequence of each terminal neuron is not an output pulse sequence corresponding to each pixel point, but an overall output pulse sequence is obtained for the input pulse sequence of each pixel point.

[0149] This is because the information in the input spike train undergoes nonlinear transformations in neurons, rather than a simple one-to-one correspondence. For example, even if there are multiple spikes in the input spike train, if their intensities and temporal distribution do not cause the neuron's membrane potential to reach the threshold, the neuron will not fire, resulting in a mismatch between the number and timing of the output spike train and the input spike train.

[0150] Furthermore, spiking neural networks typically contain multiple layers and a large number of neurons, and information undergoes multiple processing and transformations as it passes through the network. After being processed by intermediate layers such as hidden layers, the input spike train is gradually abstracted and integrated. The final output spike train reflects the comprehensive processing results of the entire network on the input information, rather than directly corresponding to the input spike train.

[0151] For example, in this scenario, that is, in the image recognition task, the input is a pulse sequence encoded by image pixels. After being processed by a multi-layer pulse neural network, the output is a pulse sequence about abstract information such as the category to which the image belongs.

[0152] S403: Determine abnormal image data based on the output pulse sequence of each terminal neuron.

[0153] Optionally, the output pulse sequence of each terminal neuron can be used to determine the pulse emission pattern, and the pulse emission pattern is matched with the pulse emission pattern features corresponding to different abnormalities to determine the image abnormality data.

[0154] Figure 6 A flowchart of another display abnormality processing method provided in an embodiment of the present application; optionally, in step S403, determining image abnormality data based on the output pulse sequence of each terminal neuron may include:

[0155] S501. Determine frequency information corresponding to each terminal neuron according to the output pulse sequence of each terminal neuron.

[0156] Optionally, the output pulse train emission pattern of the terminal neuron can be analyzed from multiple aspects such as pulse emission frequency, pulse interval, pulse train characteristics, etc.

[0157] In some embodiments, the pulse emission frequency of the terminal neuron in each time window can be counted based on a preset time window and the output pulse sequence of the terminal neuron.

[0158] S502 : Determine the image abnormality type and abnormal pixel information under the image abnormality type according to the frequency information corresponding to each terminal neuron, the position information of each terminal neuron, and the membrane potential information of each terminal neuron.

[0159] Image abnormality types include: distorted screen, black screen or static screen.

[0160] The position information of each terminal neuron refers to the specific coordinates or relative position of each neuron in the spatial structure of its location in the terminal layer of the neural network. Based on the position information of each neuron, the area in the image corresponding to each neuron can be determined.

[0161] In this embodiment, the image abnormality types can be divided into flower screen, black screen or static abnormality. The pulse emission pattern characteristics corresponding to the flower screen may include but are not limited to: high-frequency bursts of local neuronal clusters (>200Hz), -RGB channel pulse phase difference>5ms (chromaticity asynchrony), and sudden changes in spatial activation density (Δ>60%). The pulse emission pattern characteristics corresponding to the black screen may include but are not limited to: a sudden drop in the pulse emission rate of the entire network (<5Hz), the membrane potential is continuously lower than the threshold (no pulse triggering), and the loss of cross-layer pulse synchronization. The pulse emission pattern characteristics corresponding to the static abnormality may include but are not limited to: time-series periodic pulses (fixed interval ±2ms), no event-driven pulses (normal dynamic scenes should have random intervals), and cross-frame pulse correlation>0.9.

[0162] High-frequency bursts in localized neuronal clusters (>200Hz): When a spiking neural network processes an image, if a population of neurons in a local area emits high-frequency pulses exceeding 200Hz, it means that the area may be receiving abnormally strong stimulation or experiencing data processing disturbances. For example, in image recognition, if a high-frequency burst of neurons in a localized area occurs, it could be because of drastic changes in the image data in that area (such as excessive brightness or darkness, sharp edges, or noise interference), causing overexcitement of neurons. Alternatively, it could be because of abnormal network parameters, leading to incorrect activation of neurons in that area.

[0163] RGB channel pulse phase difference > 5ms (color asynchrony): For color images, the data of the three RGB channels need to be processed collaboratively to present the correct color information. When the time difference of the neuronal pulses corresponding to the three channels exceeds 5ms, the color information cannot be synchronously integrated in the time dimension. This phenomenon is common when there is a failure in the color transmission or processing of the image, such as the time calibration error of the RGB channels when the sensor collects data, or the inconsistent processing speed of the neural network for different channel data during the processing process, which ultimately manifests as abnormalities such as image color misalignment and color cast.

[0164] Sudden changes in spatial activation density (Δ>60%): Spatial activation density reflects the proportion of neurons in the neuronal space corresponding to the image that are in an activated state. When this ratio suddenly changes by more than 60%, it indicates that there has been an abnormal change in the local or overall information distribution of the image. For example, the sudden appearance of a large area of ​​noise in the image will cause a large number of neurons in the corresponding area to activate, resulting in a sharp increase in spatial activation density. Conversely, if information is lost in part of the image, the activation density of the corresponding neurons may drop sharply. This sudden change is contrary to the smooth information distribution characteristics of a normal image.

[0165] A sudden drop in the network-wide spike rate (<5Hz): The network-wide spike rate represents the activity of the entire spiking neural network. A drop below 5Hz indicates a significant decrease in the network's overall processing power, potentially preventing effective image feature extraction and analysis. This could be due to significant loss or corruption of the input image data, or network parameter malfunctions that prevent neurons from properly receiving and processing signals, resulting in an inability to generate sufficient spike outputs and distorting image analysis results.

[0166] Membrane potential remains below threshold (no pulse triggering): Neurons generate pulses by accumulating membrane potential above the threshold. If the membrane potential remains below the threshold, the neuron cannot activate normally and cannot effectively respond to the input image information. This may be due to insufficient input signal strength received by the neuron, or it may be due to abnormal neuronal function, such as ion channel failure or synaptic transmission impairment. Ultimately, the image information cannot be properly transmitted and processed at the neuron, resulting in the loss of local or global image features.

[0167] Loss of cross-layer spike synchronization: Spiking neural networks achieve deep image processing through the orderly transmission and synchronous collaboration of spike signals between layers. Loss of cross-layer spike synchronization means that information transmission between layers becomes disrupted. For example, neurons in the lower layer cannot transmit processed information to neurons in the upper layer at the correct time, resulting in incomplete or erroneous information received by the upper layer. This, in turn, affects the integration and recognition of image features and causes deviations in image analysis results.

[0168] Periodic pulses (fixed interval ±2ms): Under normal circumstances, the pulse sequence of neurons responding to image information should have a certain degree of randomness to accommodate the diversity and complexity of image content. The presence of periodic pulses with a fixed interval of ±2ms indicates that neuronal activity is exhibiting an unusual pattern. This is likely due to some interference or malfunction in the network, which has fixed the neuronal firing pattern and prevented it from truly reflecting the actual image information, leading to image analysis failure.

[0169] No event-driven pulses (normal dynamic scenes should have random intervals): When processing dynamic scene images, normal neurons should generate pulse responses with random intervals based on events such as the movement of objects in the image and changes in light. The absence of event-driven pulses means that the neurons are insensitive to dynamic changes in the image and cannot capture key information. This may be due to the network's insufficient ability to extract dynamic features, or the input image data failing to effectively trigger the neurons' dynamic response mechanism, resulting in errors in image analysis in dynamic scenes.

[0170] Cross-frame spike correlation > 0.9: When processing videos or sequential image frames, adjacent frames exhibit a certain degree of correlation. However, excessively high cross-frame spike correlation (> 0.9) indicates that the neuronal spike trains corresponding to adjacent frames are too similar, lacking the expected variation. This may be due to repetitive or static content in the input frames, or the network's inability to effectively distinguish between frames during processing, failing to capture the dynamic changes in the image sequence. Consequently, the image sequence analysis results may not reflect the actual scene.

[0171] Based on the frequency information corresponding to each terminal neuron, the burst frequency of the local neuron cluster, the pulse firing rate of the entire network, cross-layer pulse synchronization, temporal periodic pulses, non-event-driven pulses, and cross-frame pulse correlation can be determined. Based on the position information of each terminal neuron, the spatial activation density can be determined. Based on the membrane potential information of each terminal neuron, the characteristic of whether the membrane potential is continuously below the threshold can be determined. The characteristics determined based on the frequency information corresponding to each terminal neuron, the position information of each terminal neuron, and the membrane potential information of each terminal neuron are then compared with the pulse firing pattern characteristics corresponding to each abnormality type, and the successfully matched abnormality type is used as the image abnormality type of the current frame image.

[0172] Based on the correspondence between the positions of the terminal neurons and the regions in the image, the image regions where abnormalities occur can be determined and the abnormal pixel information can be counted, for example, the number of abnormal pixels in the abnormal image regions can be counted.

[0173] It is worth noting that in addition to using pulse neural networks as in this solution to detect image display anomalies, methods such as convolutional neural networks (CNN), auto-encoders (AE), and generative adversarial networks (GAN) can also be used.

[0174] Figure 7 A flowchart of another display abnormality processing method provided in an embodiment of the present application; optionally, in step S103, determining the first abnormality indicator and the second abnormality indicator based on the image abnormality data may include:

[0175] S601: Determine the number of abnormal pixels according to abnormal pixel information under an abnormal image type.

[0176] The number of abnormal pixels in the abnormal pixel area may be determined according to the abnormal pixel area.

[0177] S602: Determine a first abnormality indicator according to the number of abnormal pixels and the total number of pixels in the current image frame.

[0178] The first abnormality indicator may be the abnormal pixel ratio, which may be obtained by dividing the number of abnormal pixels by the total number of pixels, and may also be referred to as the abnormal range ratio.

[0179] S603: Determine a second abnormality indicator according to the current time corresponding to the current image frame and the abnormality start time.

[0180] The abnormality start time is the time when the image abnormality first occurs, and the image abnormality continues to occur between the abnormality start time and the current time.

[0181] The second abnormality indicator may be the duration of the abnormality. The abnormality may be continuously monitored and displayed, and the abnormality start time and abnormality end time may be determined. The abnormality end time minus the abnormality start time may be used to obtain the abnormality duration.

[0182] It is worth noting that anomalies usually occur intermittently, and there may be a time difference between two anomalies. The duration of the anomaly here refers to the duration of the current anomaly, not the duration from the first anomaly to the current anomaly. Therefore, the start time of the anomaly here is the time when the anomaly occurred, and anomalies continue to occur from the start time of the anomaly to the current time.

[0183] For example, if the anomaly occurs continuously from 12:00 to 13:00, from 15:00 to 16:00, and from 17:00 to the current time, the start time of the anomaly can be 17:00, and the end time of the anomaly can be the current time, so the duration of the anomaly from the beginning to the target can be determined.

[0184] Of course, in some embodiments, statistically calculating the pixel mean and standard deviation of each pixel in the abnormal pixel area, analyzing the histogram distribution of pixel values, or analyzing the texture information of pixels can also be used as analysis indicators.

[0185] Optionally, in step S104, determining the abnormality repair method based on the first abnormality indicator and the second abnormality indicator may include: if the proportion represented by the first abnormality indicator is greater than or equal to the proportion threshold and the abnormality duration represented by the second abnormality indicator is greater than or equal to the duration threshold, then determining that the abnormality repair method is to switch the graphics card device.

[0186] In one possible implementation, a dual-threshold judgment can be performed based on the abnormal pixel ratio and the abnormal duration to determine an abnormality repair method that is more suitable for the current scenario.

[0187] Optionally, when the percentage of abnormal pixels is greater than or equal to a percentage threshold, and the duration of the abnormality is greater than or equal to a duration threshold, it can be determined that the abnormality repair method is to switch graphics card devices. In this scenario, the abnormality is more serious, and the abnormality can be repaired by switching graphics card devices.

[0188] Otherwise, the restoration method is determined to be local abnormal pixel restoration.

[0189] If the proportion of abnormal pixels is greater than or equal to the proportion threshold and the duration of the abnormality is less than the duration threshold; or if the proportion of abnormal pixels is less than the proportion threshold and the duration of the abnormality is greater than or equal to the duration threshold; or if the proportion of abnormal pixels is less than the proportion threshold and the duration of the abnormality is less than the duration threshold, in all three cases, the repair method is determined to be local abnormal pixel repair. In these scenarios, the abnormality is relatively less serious and can be repaired through local repair.

[0190] In another feasible manner, the repair method may be determined based on the first abnormality indicator, the second abnormality indicator and the operating status of the graphics card.

[0191] In some cases, even if the first abnormality indicator and the second abnormality indicator both exceed the corresponding thresholds, the graphics card operation status indicates that the graphics card has no abnormality. In this case, local repair can be used to repair it first.

[0192] In other cases, even if the first abnormality indicator or the second abnormality indicator does not exceed the corresponding threshold, the graphics card operation status indicates that the graphics card is abnormal. At this time, it can still be determined to be repaired by switching the graphics card device.

[0193] Figure 8 A flowchart of another display exception handling method provided in an embodiment of the present application is provided; optionally, in step S104, performing exception repair based on the exception repair method may include:

[0194] S701: If the abnormality repair method is to switch the graphics card device, suspend the rendering task of the current graphics card device.

[0195] If the system determines that switching graphics devices is the correct way to fix the issue, it will automatically switch graphics devices. First, it will temporarily pause the rendering task of the currently running graphics device to prevent further display anomalies.

[0196] The DRM_IOCTL_SET_MASTER command can be used to suspend the currently running graphics card device. DRM_IOCTL_SET_MASTER is an ioctl command in DRM that is used to set a process as the master process of the DRM device.

[0197] S702: Determine a target standby graphics card device according to priority information of each standby graphics card device.

[0198] The system may determine a target spare graphics card device from among the spare graphics card devices according to the priorities of the spare graphics card devices.

[0199] S703: Modify the configuration information of the graphics card associated service and the dynamic linker according to the information of the target standby graphics card device, and load the target standby graphics card device from the designated directory of the system.

[0200] Based on the determined target backup graphics device information, the graphics device can be switched by modifying the graphics card-related service, namely the Xorg service configuration information, and the Glibc (GNU C Library) dynamic loader. Xorg is an open source implementation of the X Window System (often referred to as X), and the Xorg service is a key service based on the Xorg software.

[0201] Optionally, the target backup graphics card device can be loaded from a specified directory / lib / modules. / lib / modules is an important directory in the Linux system, mainly used to store kernel modules.

[0202] S704: Reinitialize the rendering task and call the atomic submit function to submit the status information of the target standby graphics card device to the graphics processor.

[0203] After the target standby graphics device is loaded, the system will reinitialize the rendering task, restore normal display, and call the drmModeAtomicCommit atomic commit function to submit the status information of the target standby graphics device to the graphics processor to resume the rendering task.

[0204] Figure 9 A flowchart of another display exception handling method provided in an embodiment of the present application is provided; optionally, in step S104, performing exception repair based on the exception repair method may include:

[0205] S801: If the abnormality repair method is local abnormal pixel repair, determine the first interpolation pixel point and the second interpolation pixel point corresponding to each abnormal pixel point according to the pixel coordinate information and pixel offset information of each abnormal pixel point.

[0206] If the problem is a local abnormal pixel repair, the new pixel information of the abnormal pixel can be calculated through interpolation algorithms and re-rendered. Among them, methods such as bilinear interpolation or high-order interpolation can be used to achieve this.

[0207] Optionally, the following interpolation calculation formula can be used to calculate the repaired pixel information of the abnormal pixel point:

[0208] I fixed (x,y)=Interpolation(I(x-Δx,y-Δy),I(x+Δx,y+Δy))

[0209] Among them, I(x,y) is the original pixel information of the abnormal pixel, I fixed (x, y) is the repaired pixel information of the abnormal pixel, and Δx and Δy are the offset information of adjacent pixels.

[0210] The first interpolated pixel point and the second interpolated pixel point corresponding to the abnormal pixel point can be determined based on the pixel offset information. The first interpolated pixel point is the pixel point corresponding to (x-Δx, y-Δy), and the second interpolated pixel point is the pixel point corresponding to (x+Δx, y+Δy).

[0211] S802: Determine restored pixel information corresponding to each abnormal pixel point based on pixel information of a first interpolated pixel point corresponding to each abnormal pixel point and pixel information of a second interpolated pixel point corresponding to each abnormal pixel point.

[0212] Then, an interpolation operation is performed based on the pixel information of the first interpolated pixel point and the pixel information of the second interpolated pixel point corresponding to each abnormal pixel point, so that the repaired pixel information corresponding to the abnormal pixel point can be obtained.

[0213] S803: Re-render the current image frame according to the repaired pixel information corresponding to each abnormal pixel point.

[0214] According to the repaired pixel information corresponding to each abnormal pixel point, the current frame image can be re-rendered to repair the display screen.

[0215] In some embodiments, an algorithm based on image content perception may also be used to analyze texture features around abnormal image regions, extract similar texture blocks from other normal regions of the image, and fill the abnormal regions by splicing and adjusting these texture blocks.

[0216] Alternatively, a generative model based on deep learning, such as a generative adversarial network, can be used to train a generator network to learn the distribution of normal images. For abnormal areas, the generator can generate image content that matches the surrounding environment based on the surrounding contextual information to fill in the abnormal areas.

[0217] Optionally, in step S701 , the current graphics card device is determined by monitoring a currently running rendering thread and determining the current graphics card device according to a mapping relationship between the rendering thread and the graphics card device.

[0218] In some embodiments, the graphics card device currently running on the system can be determined by monitoring the currently running rendering thread and determining the graphics card device called by the currently running rendering thread based on the mapping relationship between the rendering thread and the graphics card device.

[0219] The rendering thread usually communicates with the graphics card device through the graphics card driver. The rendering thread sends rendering tasks to the graphics card device and monitors the execution status of the tasks. If the graphics card device has an abnormality, the rendering thread may detect an error.

[0220] Optionally, in the above steps, before determining the current graphics card device based on the mapping relationship between the rendering thread and the graphics card device, it may also include: in the process of each rendering thread calling the context association function to associate the open image library with the frame buffer, monitoring each rendering thread and recording the mapping relationship between the identifier of each rendering thread and the identifier of the graphics card device used by each rendering thread.

[0221] In a multi-threaded rendering environment, each rendering thread may be responsible for different rendering tasks and may use different graphics devices. Since the glXMakeCurrent() function is the key interface for establishing an association between a rendering thread and a graphics device, the system can add monitoring logic to this interface to obtain the working status of the graphics device in real time.

[0222] When the rendering thread calls the glXMakeCurrent() function, the system can record the time of the call, the identifier of the associated graphics device, and other information. By analyzing this information, the usage frequency and load status of the graphics device can be understood.

[0223] Among them, glXMakeCurrent() is an important function for OpenGL rendering in the X Window System. Its function is to associate an OpenGL context with a specific X window or pixel buffer, and set the current rendering thread to the thread operating the context.

[0224] By monitoring the rendering thread's call to glXMakeCurrent(), the mapping relationship between the rendering thread and the graphics card device used can be recorded, so that the currently running graphics card device can be determined by subsequently monitoring the rendering thread's operation.

[0225] In summary, the display abnormality processing method provided by this embodiment includes: real-time acquisition of image data of the current image frame in the frame buffer, the image data including pixel information of multiple pixels; detecting and processing the image data based on a pulse neural network to obtain image abnormality data; determining a first abnormality indicator and a second abnormality indicator based on the image abnormality data; determining an abnormality repair method based on the first abnormality indicator and the second abnormality indicator, and performing abnormality repair based on the abnormality repair method. This method uses a pulse neural network to perform abnormality detection and processing on the image data of the image frame obtained from the frame buffer, which can simulate the retinal neural mechanism to accurately identify display abnormalities and realize millisecond-level dynamic analysis, thereby improving the accuracy and efficiency of abnormality detection; and determining the abnormality repair method by combining the dual indicators of abnormal pixel ratio and abnormal duration makes the abnormality repair method more in line with the current abnormal scenario, thereby improving the accuracy of abnormality repair.

[0226] Figure 10 This is a schematic diagram of an edge collaboration system architecture provided by an embodiment of the present application. Figure 10 As shown, the edge collaboration system can be composed of a main node device and at least one edge node device.

[0227] Among them, a pulse neural network is deployed on each edge node device; the pulse neural network is used to detect anomalies of the input pulse sequence according to the above implementation method.

[0228] The above-mentioned processing chip can be deployed on the master node device to perform global coordination.

[0229] When running, the edge node device is used to perform the method steps related to anomaly detection and local abnormal pixel repair in the above embodiment.

[0230] Optionally, the edge node device focuses on local repair, and the master node device focuses on global coordination, including global resource scheduling, dual-indicator abnormality repair judgment, and switching of graphics card devices.

[0231] When the abnormal range is small, that is, it is judged to be a local abnormal repair method, the edge node device can perform the repair locally and autonomously; when the abnormal range is large, that is, it is judged to be switching the graphics card device, the main node device triggers the switching of the graphics card device.

[0232] Figure 11A schematic diagram of an edge collaborative processing flow provided by an embodiment of the present application. Each edge node device can process different frame images in parallel and, in accordance with the above-described processing flow, perform real-time analysis and processing of the image data of each collected frame image and synchronize abnormal data. Each edge node device synchronizes the abnormal data analyzed by the device to the master node device, which then determines the abnormal repair method according to the above-described judgment logic. If the local repair method is used, the edge node device is controlled to perform the repair autonomously. If the graphics card device is switched, the switching of the graphics card device is triggered.

[0233] By adopting a distributed edge collaboration architecture, it can effectively support hierarchical repair methods such as local repair and global graphics card device switching in multi-graphics processor scenarios, reducing system load.

[0234] The following describes the apparatus, device, storage medium, etc. used to execute the display exception handling method provided in this application. The specific implementation process and technical effects are described above and will not be repeated below.

[0235] Figure 12 This is a schematic diagram of a display exception handling device provided in an embodiment of the present application. The functions implemented by the display exception handling device correspond to the steps performed by the above method. The device can be understood as the above-mentioned computer device or server, or the processor of the server, or can also be understood as a component independent of the above-mentioned server or processor that implements the functions of the present application under the control of the server, such as Figure 12 As shown, the apparatus may include: a collection module 200, a processing module 210, a determination module 220, and a repair module 230;

[0236] An acquisition module 200 is used to acquire image data of a current image frame in a frame buffer in real time, where the image data includes pixel information of a plurality of pixels;

[0237] The processing module 210 is configured to detect and process the image data based on a spiking neural network to obtain image abnormality data, where the image abnormality data includes: an image abnormality type and abnormal pixel information associated with the image abnormality type;

[0238] A determination module 220 is configured to determine a first abnormality indicator and a second abnormality indicator based on the image abnormality data, wherein the first abnormality indicator is used to characterize the proportion of abnormal pixels in the spatial dimension, and the second abnormality indicator is used to characterize the duration of the abnormality;

[0239] The repair module 230 is configured to determine an abnormality repair method according to the first abnormality indicator and the second abnormality indicator, and perform abnormality repair based on the abnormality repair method.

[0240] Optionally, the acquisition module 200 is specifically configured to capture image data of a current image frame in a frame buffer in real time through a library interface during startup and operation of a system desktop program.

[0241] Optionally, the processing module 210 is specifically configured to perform data encoding on pixel information of each pixel in the image data to obtain an input pulse sequence corresponding to each pixel;

[0242] The input pulse sequence corresponding to each pixel point is input into the pulse neural network for pulse signal propagation and hierarchical analysis to obtain image abnormality data.

[0243] Optionally, the processing module 210 is specifically configured to determine a pulse emission rate corresponding to each pixel point based on brightness information and color information of each pixel point;

[0244] Generate multiple pulse generation times according to the pulse emission rate corresponding to each pixel point, where each pulse generation time is used to indicate the specific moment when a pulse of the pixel point in the pulse sequence of the current image frame occurs;

[0245] According to the pulses emitted by each pixel at each pulse generation time, an input pulse sequence corresponding to each pixel is generated.

[0246] Optionally, the processing module 210 is specifically configured to input the input pulse sequence corresponding to each pixel point into the pulse neural network, process and propagate it in sequence through each layer of neurons in the pulse neural network, and determine the image abnormality data based on the output pulse sequence of the terminal neuron.

[0247] Optionally, the processing module 210 is specifically configured to determine the membrane potential change information of each neuron in the current layer according to the input pulse sequence corresponding to each pixel point and the synaptic weights between the pixel point and each neuron in the current layer to which the pulse is transmitted;

[0248] According to the membrane potential change information of each neuron in the current layer, the output pulse sequence of each neuron in the current layer is generated, and the output pulse sequence of each neuron in the current layer is propagated to the corresponding neurons in the next layer for processing until the output pulse sequence of each neuron at the end is obtained;

[0249] According to the output pulse sequence of each terminal neuron, the abnormal image data is determined.

[0250] Optionally, the processing module 210 is specifically configured to determine frequency information corresponding to each terminal neuron according to the output pulse sequence of each terminal neuron;

[0251] Based on the frequency information corresponding to each terminal neuron, the position information of each terminal neuron, and the membrane potential information of each terminal neuron, the image abnormality type and the abnormal pixel information under the image abnormality type are determined. The image abnormality type includes: flower screen, black screen or static abnormality.

[0252] Optionally, the determination module 220 is specifically configured to determine the number of abnormal pixels according to abnormal pixel information under the abnormal type of the image;

[0253] Determining a first abnormality indicator based on the number of abnormal pixels and the total number of pixels in the current image frame;

[0254] A second abnormality indicator is determined based on the current time corresponding to the current image frame and the abnormality start time, wherein the abnormality start time is the time when the image abnormality first occurs, and the image abnormality continues to occur between the abnormality start time and the current time.

[0255] Optionally, the determination module 220 is specifically configured to determine that the abnormality repair method is to switch the graphics card device if the proportion represented by the first abnormality indicator is greater than or equal to the proportion threshold and the abnormality duration represented by the second abnormality indicator is greater than or equal to the duration threshold;

[0256] Otherwise, the restoration method is determined to be local abnormal pixel restoration.

[0257] Optionally, the repair module 230 is specifically configured to suspend the rendering task of the current graphics card device if the abnormality repair method is to switch the graphics card device;

[0258] Determine the target standby graphics card device according to the priority information of each standby graphics card device;

[0259] According to the information of the target backup graphics card device, the configuration information of the graphics card associated service and the dynamic linker are modified, and the target backup graphics card device is loaded from the specified directory of the system;

[0260] Reinitialize the rendering task and call the atomic submit function to submit the status information of the target backup graphics device to the graphics processor.

[0261] Optionally, the repair module 230 is specifically configured to determine a first interpolation pixel point and a second interpolation pixel point corresponding to each abnormal pixel point according to pixel coordinate information and pixel offset information of each abnormal pixel point if the abnormality repair method is local abnormal pixel repair;

[0262] Determine restored pixel information corresponding to each abnormal pixel point based on pixel information of a first interpolated pixel point corresponding to each abnormal pixel point and pixel information of a second interpolated pixel point corresponding to each abnormal pixel point;

[0263] Re-render the current image frame based on the repaired pixel information corresponding to each abnormal pixel point.

[0264] Optionally, the determination module 220 is further configured to monitor the currently running rendering thread and determine the current graphics card device according to a mapping relationship between the rendering thread and the graphics card device.

[0265] Optionally, the processing module 210 is further used to monitor each rendering thread and record the mapping relationship between the identifier of each rendering thread and the identifier of the graphics card device used by each rendering thread during the process of each rendering thread calling the context association function to associate the open image library with the frame buffer.

[0266] The above-mentioned device is used to execute the method provided in the above-mentioned embodiment. Its implementation principle and technical effect are similar and will not be repeated here.

[0267] The above modules can be one or more integrated circuits configured to implement the above methods, such as one or more application specific integrated circuits (ASICs), one or more digital singnal processors (DSPs), or one or more field programmable gate arrays (FPGAs). For another example, when a module is implemented by scheduling program code through a processing element, the processing element can be a general-purpose processor, such as a central processing unit (CPU) or other processor that can call program code. For another example, these modules can be integrated together and implemented in the form of a system-on-a-chip (SOC).

[0268] The above modules can be connected or communicate with each other via a wired connection or a wireless connection. The wired connection may include a metal cable, an optical cable, a hybrid cable, etc., or any combination thereof. The wireless connection may include a connection in the form of a LAN, a WAN, Bluetooth, ZigBee, or NFC, or any combination thereof. Two or more modules can be combined into a single module, and any module can be divided into two or more units. Those skilled in the art will clearly understand that for the convenience and brevity of description, the specific working process of the system and device described above can refer to the corresponding process in the method embodiment, and will not be repeated in this application.

[0269] Figure 13 This is a schematic diagram of an electronic device provided in an embodiment of the present application. The electronic device includes: a processor 801 and a storage medium 802.

[0270] The storage medium 802 is used to store programs, and the processor 801 calls the programs stored in the storage medium 802 to execute the above method embodiment. The specific implementation methods and technical effects are similar and will not be repeated here.

[0271] Among them, the storage medium 802 stores program code, and when the program code is executed by the processor 801, the processor 801 executes the various steps of the display exception handling method according to various exemplary embodiments of the present application described in the above "Exemplary Method" section of this specification.

[0272] The processor 801 can be a general-purpose processor, such as a central processing unit (CPU), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, and can implement or execute the various methods, steps and logic block diagrams disclosed in the embodiments of the present application. The general-purpose processor can be a microprocessor or any conventional processor, etc. The steps of the method disclosed in conjunction with the embodiments of the present application can be directly embodied as being executed by a hardware processor, or can be executed by a combination of hardware and software modules in the processor.

[0273] Storage medium 802 is a kind of non-volatile computer readable storage medium, which can be used for storing non-volatile software programs, non-volatile computer executable programs and modules. Storage medium can include at least one type of storage medium, for example, can include flash memory, hard disk, multimedia card, card type storage medium, random access storage medium (Random Access Memory, RAM), static random access storage medium (Static Random Access Memory, SRAM), programmable read-only storage medium (Programmable Read Only Memory, PROM), read-only storage medium (Read Only Memory, ROM), electrically erasable programmable read-only storage medium (Electrically Erasable Programmable Read-Only Memory, EEPROM), magnetic storage medium, disk, optical disk, etc. Storage medium is any other medium that can be used to carry or store desired program code with instruction or data structure form and can be accessed by computer, but is not limited to this. The storage medium 802 in the embodiment of the present application can also be a circuit or other arbitrarily capable of realizing storage function, for storing program instructions and / or data.

[0274] Optionally, the present application also provides a program product, such as a computer-readable storage medium, comprising a program, which is used to perform the above method embodiment when executed by a processor.

[0275] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0276] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0277] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0278] The above-mentioned integrated unit implemented in the form of a software functional unit can be stored in a computer-readable storage medium. The above-mentioned software functional unit is stored in a storage medium and includes a number of instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) or a processor (English: processor) to execute some steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: a USB flash drive, a mobile hard disk, a read-only storage medium (English: Read-Only Memory, abbreviated: ROM), a random access storage medium (English: Random Access Memory, abbreviated: RAM), a disk or an optical disk, and other media that can store program code.

Claims

1. A display exception processing method, characterized in that: include: Acquire image data of a current image frame in a frame buffer in real time, wherein the image data includes pixel information of a plurality of pixels; Performing detection processing on the image data based on a pulse neural network to obtain image abnormality data, wherein the image abnormality data includes: an image abnormality type and abnormal pixel information under the image abnormality type; Determining a first abnormality indicator and a second abnormality indicator based on the image abnormality data, wherein the first abnormality indicator is used to characterize the proportion of abnormal pixels in a spatial dimension, and the second abnormality indicator is used to characterize the duration of the abnormality; An abnormality repair method is determined according to the first abnormality indicator and the second abnormality indicator, and abnormality repair is performed based on the abnormality repair method.

2. The method according to claim 1, characterized in that The real-time acquisition of image data of the current image frame in the frame buffer includes: During the process of starting and running the system desktop program, the image data of the current image frame in the frame buffer is captured in real time through the library interface.

3. The method according to claim 1, characterized in that The detecting and processing the image data based on the pulse neural network to obtain image abnormality data includes: Performing data encoding on pixel information of each pixel in the image data to obtain an input pulse sequence corresponding to each pixel; The input pulse sequence corresponding to each pixel point is input into the pulse neural network for pulse signal propagation and hierarchical analysis to obtain the image abnormality data.

4. The method according to claim 3, characterized in that The step of encoding pixel information of each pixel in the image data to obtain an input pulse sequence corresponding to each pixel includes: Determine the pulse emission rate corresponding to each pixel based on the brightness and color information of each pixel; Generate multiple pulse generation times according to the pulse emission rate corresponding to each pixel point, where each pulse generation time is used to indicate the specific moment when a pulse of the pixel point in the pulse sequence of the current image frame occurs; According to the pulses emitted by each pixel at each pulse generation time, an input pulse sequence corresponding to each pixel is generated.

5. The method according to claim 3, characterized in that The input pulse sequence corresponding to each pixel point is input into the pulse neural network for pulse signal propagation and hierarchical analysis to obtain the image abnormality data, including: The input pulse sequence corresponding to each pixel point is input into the pulse neural network, processed and propagated in sequence through each layer of neurons in the pulse neural network, and the image abnormality data is determined according to the output pulse sequence of the terminal neuron.

6. The method according to claim 5, characterized in that The input pulse sequence corresponding to each pixel point is input into the spiking neural network, processed and propagated in sequence through neurons in each layer of the spiking neural network, and the image abnormality data is determined according to the output pulse sequence of the terminal neuron, including: Determine the membrane potential change information of each neuron in the current layer based on the input pulse sequence corresponding to each pixel point and the synaptic weight between the pixel point and each neuron in the current layer to which the signal is transmitted; According to the membrane potential change information of each neuron in the current layer, the output pulse sequence of each neuron in the current layer is generated, and the output pulse sequence of each neuron in the current layer is propagated to the corresponding neurons in the next layer for processing until the output pulse sequence of each neuron at the end is obtained; The image abnormality data is determined according to the output pulse sequence of each terminal neuron.

7. The method according to claim 6, characterized in that The determining of the image abnormality data according to the output pulse sequence of each terminal neuron includes: According to the output pulse sequence of each terminal neuron, the frequency information corresponding to each terminal neuron is determined; According to the frequency information corresponding to each terminal neuron, the position information of each terminal neuron and the membrane potential information of each terminal neuron, the image abnormality type and the abnormal pixel information under the image abnormality type are determined. The image abnormality type includes: flower screen, black screen or static abnormality.

8. The method according to claim 1, characterized in that Determining a first abnormality indicator and a second abnormality indicator according to the image abnormality data includes: Determining the number of abnormal pixels according to the abnormal pixel information under the abnormal type of the image; Determining the first abnormality indicator according to the number of abnormal pixels and the total number of pixels in the current image frame; The second abnormality indicator is determined according to the current time corresponding to the current image frame and the abnormality start time, wherein the abnormality start time is the time when the image abnormality first occurs, and the image abnormality continues to occur between the abnormality start time and the current time.

9. The method according to claim 1, characterized in that The determining of an abnormality repair method according to the first abnormality indicator and the second abnormality indicator includes: If the proportion represented by the first abnormality indicator is greater than or equal to the proportion threshold and the abnormality duration represented by the second abnormality indicator is greater than or equal to the duration threshold, determining that the abnormality repair method is to switch the graphics card device; Otherwise, it is determined that the repair method is local abnormal pixel repair.

10. The method according to claim 9, characterized in that The performing of abnormality repair based on the abnormality repair method includes: If the abnormality repair method is to switch the graphics card device, then suspend the rendering task of the current graphics card device; Determine the target standby graphics card device according to the priority information of each standby graphics card device; According to the information of the target standby graphics card device, modify the configuration information of the graphics card associated service and modify the dynamic linker, and load the target standby graphics card device from a specified directory of the system; Reinitialize the rendering task, and call the atomic submit function to submit the status information of the target standby graphics card device to the graphics processor.

11. The method according to claim 9, characterized in that The performing of abnormality repair based on the abnormality repair method includes: If the abnormality repair method is local abnormal pixel repair, then the first interpolation pixel point and the second interpolation pixel point corresponding to each abnormal pixel point are determined according to the pixel coordinate information and pixel offset information of each abnormal pixel point; Determine restored pixel information corresponding to each abnormal pixel point based on pixel information of a first interpolated pixel point corresponding to each abnormal pixel point and pixel information of a second interpolated pixel point corresponding to each abnormal pixel point; Re-render the current image frame based on the repaired pixel information corresponding to each abnormal pixel point.

12. The method according to claim 10, characterized in that The current graphics card device is determined in the following manner: Monitor the currently running rendering thread and determine the current graphics card device according to the mapping relationship between the rendering thread and the graphics card device.

13. The method according to claim 12, characterized in that Before determining the current graphics card device according to the mapping relationship between the rendering thread and the graphics card device, the method further includes: During the process of each rendering thread calling the context association function to associate the open image library with the frame buffer, each rendering thread is monitored, and a mapping relationship between the identifier of each rendering thread and the identifier of the graphics card device used by each rendering thread is recorded.

14. An edge collaboration system, characterized in that: include: A master node device and at least one edge node device; A pulse neural network is deployed on each edge node device; The edge node device is used to perform method steps related to abnormality detection and local abnormal pixel repair in the display abnormality processing method according to any one of claims 1 to 13; The master node device is used for global coordination.

15. An electronic device, characterized in that: include: A processor, a storage medium and a bus, wherein the storage medium stores program instructions executable by the processor. When the electronic device is running, the processor and the storage medium communicate via the bus, and the processor executes the program instructions to implement the display exception handling method according to any one of claims 1 to 13.

16. A computer-readable storage medium, characterized in that The storage medium stores a computer program, and when the computer program is executed by the processor, the display exception processing method according to any one of claims 1 to 13 is implemented.

Images