DLP defective pixel calibration and compensation method and system based on Transform attention mechanism

Abstract

The invention relates to the technical field of dead pixel detection and compensation of a digital light processing projection system, in particular to a DLP dead pixel calibration and compensation method and system based on a Transform attention mechanism. According to the DLP dead pixel calibration and compensation method based on the Transform attention mechanism, a multi-scale feature pyramid is constructed, global features are extracted by using a layered multi-mode Transform encoder, and a dead pixel confidence coefficient heat map is generated in combination with a dynamic sparse mask multi-head attention mechanism; dynamically selecting compensation strategies for different types of dead pixels by adopting a dead pixel type-scene demand dual-drive time-space domain compensation pipeline; ambient light and partition temperature sensor data are integrated, and the influence of optical crosstalk and thermal drift is eliminated through a brightness adaptive calibration function; and finally, carrying out closed-loop verification and iterative optimization by adopting a loss function fused with HVS perception loss. According to the method, the defective pixel detection precision is improved from 82.3% to 99.2%, the PSNR after compensation is improved from 32.6 dB to 42.5 dB, the processing delay is reduced to 7ms / frame, and the visual quality and reliability of a DLP system are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of bad pixel detection and compensation of digital light processing projection systems, and belongs to an intelligent bad pixel processing method combining global perception and dynamic compensation mechanism, in particular to a DLP bad pixel calibration and compensation method, system and computer readable storage medium based on a Transformer attention mechanism. BACKGROUND

[0002] Digital light processing (DLP) projection technology is widely used in commercial, educational and home theater fields due to its high brightness, high contrast and fast response. The core imaging component of DLP system is digital micro-mirror device (DMD), which is composed of millions of micro-mirrors, and the on-off state of each micro-mirror controls the brightness of a pixel. However, during manufacturing and long-term use, some micro-mirrors may fail due to physical defects or aging, resulting in constant bright, constant dark or flickering bad pixels on the projected image, which seriously affects the visual experience.

[0003] In the prior art, the processing scheme for DLP bad pixels mainly relies on image processing and local statistical methods. Typical methods include: setting adaptive threshold based on sub-pixel brightness sorting to identify bad pixels, using sparse storage strategy to record bad pixel distribution, using minimum gradient direction for interpolation compensation, and response compensation based on linear fitting. Although these methods improve the efficiency of processing to some extent, the core logic of which is heavily dependent on local neighborhood pixel statistics and pre-set artificial rules, there are the following fundamental technical limitations: (1) The detection logic relying on local statistical features is prone to misjudgment when facing complex texture background or dense bad pixel area, and it is difficult to distinguish between real bad pixels and complex image details; (2) The interpolation algorithm used lacks flexibility and is difficult to adapt to cross textures and fine structures in the image, and after compensation, it is easy to introduce blur or artifacts, which destroys the integrity of the picture; (3) The existing methods completely lack the ability to perceive the global context and semantic information of the image, resulting in an isolated and shortsighted decision-making process that cannot make intelligent judgments consistent with human visual perception; (4) For the inherent thermal drift phenomenon of DLP system and environmental light crosstalk and other physical disturbances, traditional schemes lack effective adaptive suppression mechanisms, resulting in rapid invalidation of compensation parameters after system state changes, poor robustness; (5) For two-dimensional irregular bad pixel clusters, traditional storage and calculation strategies are difficult to achieve efficient processing, often compromising between processing accuracy and real-time performance.

[0004] Therefore, there is an urgent need for an intelligent DLP bad pixel calibration and compensation technology that can break through the limitations of local vision, has global perception ability, and is self-adaptive to system and environmental changes, to fundamentally solve the above technical bottlenecks. SUMMARY

[0005] The technical problem to be solved by the present application is that the existing DLP bad pixel processing technology in the background art relies on local statistics and fixed rules, and has problems such as complex texture misjudgment, compensation rigidity, lack of global perception, heat drift sensitivity, and low efficiency of bad pixel cluster processing. The present application provides a DLP bad pixel calibration and compensation method based on a Transformer attention mechanism, which realizes global context perception through multi-scale feature extraction and hierarchical modal encoding, high-precision bad pixel detection through a dynamic sparse attention mechanism, adaptive compensation strategy selection through a dual-driven compensation pipeline, environment interference suppression through multi-sensor fusion calibration, and finally ensures optimal visual perception through perception loss closed-loop verification, thereby improving the accuracy of bad pixel processing.

[0006] The technical solution adopted by the present application to solve the technical problem is a DLP bad pixel calibration and compensation method based on a Transformer attention mechanism, comprising the following steps: S1, performing adaptive histogram equalization, wavelet domain noise reduction, and texture complexity and optical crosstalk preprocessing on the original image stream output by the DLP optical engine, and constructing a multi-scale feature pyramid; S2, inputting the multi-scale feature pyramid into a hierarchical multi-modal Transformer encoder for global feature extraction, and inputting the output feature vector into a dynamic sparse mask multi-head attention calculation core to generate a bad pixel confidence heat map; S3, determining the bad pixel type according to the bad pixel confidence heat map, and starting a bad pixel type-scene demand dual-driven spatio-temporal compensation pipeline to dynamically select and execute corresponding spatial compensation, temporal compensation, or hybrid compensation strategies for different types of bad pixels; S4, integrating the data of an ambient light sensor and a micromirror array partition temperature sensor, and dynamically correcting the pixel brightness value processed by the compensation strategy through a brightness adaptive calibration function; S5, verifying the calibrated image using a loss function that fuses the human visual system (HVS) perception loss, if the verification fails, generating an error map to feed back to S2 for iterative optimization, if the verification passes, outputting the final image.

[0007] By introducing the Transformer attention mechanism, the system has the ability to perceive global context; by multi-sensor fusion calibration, the robustness of the system in different environments and working conditions is improved; through the closed-loop verification mechanism, the reliability and accuracy of the compensation results are ensured, and finally the efficient, accurate and adaptive bad pixel repair is realized.

[0008] Further, the texture complexity and light crosstalk preprocessing in S1 specifically include: Calculate the texture entropy of the image, which integrates edge density and gradient variance, and perform light crosstalk suppression on high brightness areas in the image based on ambient light sensor data; output the calculated texture entropy map together with the multi-scale feature pyramid for subsequent bad pixel detection.

[0009] The texture entropy provides a key input for the subsequent dynamic sparse attention, enabling the model to intelligently distinguish between complex textures and real bad pixels, reducing the false detection rate; the light crosstalk preprocessing suppresses the interference in high brightness areas, improving the signal-to-noise ratio of subsequent bad pixel detection and ensuring high-precision detection.

[0010] Further, the process of global feature extraction by the hierarchical multi-modal Transformer encoder in S2 is as follows: Load the bad pixel type modal library containing dark point, bright point and flicker bad point specific feature embedding vectors, embed pixel-level bad pixel modal at the low layer of the encoder, embed local structure bad pixel modal at the middle layer, and embed global semantic bad pixel modal at the high layer. Through the cross-scale feature fusion gate, features from different scales are fused.

[0011] Multi-level and multi-modal feature embedding enables the model to accurately learn and distinguish the features of different types of bad pixels such as dark points, bright points and flicker bad points; cross-scale feature fusion ensures the effective combination of global information and local details, thereby improving the recall rate and accuracy of bad pixel feature extraction.

[0012] Further, the process of generating a bad pixel confidence heat map by the dynamic sparse mask multi-head attention calculation core in S2 is as follows: Dynamically generate a sparse mask based on texture entropy, where when the texture entropy is higher than a set threshold, set the first sparsity, and when the texture entropy is lower than the set threshold, set the second sparsity higher than the first sparsity; introduce a light crosstalk compensation factor obtained from ambient light data, and calculate the attention weight through the following formula: , where Q, K, and V are query, key, and value vectors, d is the vector dimension, is the dynamic sparse mask, is the light crosstalk compensation factor; Generate a bad pixel confidence heat map with the same resolution as the input image, where the pixel value represents the probability that the point is a bad pixel.

[0013] By adjusting the sparsity adaptively through texture entropy, the balance between calculation efficiency and detection accuracy is achieved; by introducing a light crosstalk factor to modify the weight, the detection model is immune to the inherent physical interference of the DLP system, which is the core of high-precision and high-efficiency bad pixel detection, and a reliable bad pixel confidence heat map is generated.

[0014] Further, in S3, the bad pixel type is determined according to the bad pixel confidence heat map, specifically: When the confidence of a pixel is greater than 0.85, it is determined to be a permanent bad pixel; When the confidence of a pixel is between 0.4 and 0.85, it is determined to be a suspected dynamic bad pixel; When the confidence of a pixel is less than 0.4, it is determined to be a normal pixel.

[0015] By setting two key thresholds of 0.85 and 0.4, the classification of bad pixel types is refined, which provides a basis for subsequent differentiated compensation, ensures the pertinence of compensation, and avoids overcompensation for transient bad pixels or insufficient compensation for permanent bad pixels.

[0016] Further, in S3, the bad pixel type- scene demand dual-driven spatio-temporal compensation pipeline, specifically: By automatically identifying the conference projection, cinema projection or AR glasses projection scene through image resolution and brightness parameters; for permanent bad pixels, radial basis interpolation guided by attention weight, attention weighted bilinear interpolation or lightweight radial basis interpolation is selected for spatial compensation according to the scene type; for suspected dynamic bad pixels, after excluding false bad pixels through multi-frame motion consistency analysis, temporal interpolation is performed, wherein the number of reference frames for analysis is dynamically adjusted by the attention weight; for bad pixel clusters, a spatio-temporal joint optimization algorithm is used for hybrid compensation; It also includes potential bad pixel pre-compensation, specifically: collecting micro-mirror flip response time sensor data; predicting potential bad pixels in a preset time period based on flip response time through an LSTM network; performing pre-compensation on the identified potential bad pixels, including adjusting their brightness compensation coefficients.

[0017] For different bad pixel types and performance requirements of different application scenarios, the optimal algorithm is dynamically selected to balance performance and effect; the newly added potential bad pixel pre-compensation function greatly improves the reliability of the DLP system by monitoring the micro-mirror health status and intervening in advance.

[0018] Further, in S4, the brightness adaptive calibration function dynamically corrects the compensation value through the following formula: , Wherein, C is the calibrated brightness value, B is the compensation estimated value, E is the ambient light intensity, and γ is the light crosstalk suppression coefficient, whose value is between 0.8 and 1.0, and decreases with the increase of ambient light intensity E; is the temperature drift value; is the first dynamic adjustment coefficient, whose value is a function based on attention weight w, pixel coordinates ; is the second dynamic adjustment coefficient, whose value is a function based on temperature gradient , historical drift curve H and partition temperature data ; The first dynamic adjustment coefficient , when identified as a cinema projection scene, its output value is increased by 20% based on the reference; The second dynamic adjustment coefficient , when the partition temperature data indicates that the pixel is located in the high temperature area of the optical engine center, its output value is increased by 20% based on the reference.

[0019] The formula deeply integrates image semantics, environmental state and compensation algorithm, and α and β are dynamic adaptive functions, which can quantitatively and accurately compensate for the brightness erosion caused by ambient light and the brightness drift caused by system heating in real time, and ensure the accuracy and stability of the compensation brightness under various working conditions.

[0020] Further, the loss function S5 that fuses the human visual system (HVS) perception loss is : , Wherein, is the reconstruction error between the compensation area and the normal area; is the attention entropy; is the HVS perception loss, which gives a first weight to the error of the dark field area and a second weight to the error of the texture edge area, and the first weight is greater than the second weight; is the edge consistency error; is the predefined weight coefficient of the attention entropy, the HVS perception loss and the edge consistency error; During iterative optimization, the verification threshold of the edge consistency error is dynamically adjusted according to the scene type.

[0021] By introducing the HVS perception loss, it is ensured that the repaired picture is natural and comfortable under human eye observation; the dynamic verification threshold mechanism enables the system to flexibly adjust the acceptance standard according to the scene characteristics, and realizes the balance between processing precision and speed.

[0022] Also provided is a system for implementing the DLP bad pixel calibration and compensation method based on the Transformer attention mechanism described in the above scheme, comprising: An image preprocessing module for performing adaptive histogram equalization, wavelet domain noise reduction, and texture complexity and light crosstalk preprocessing on the original image stream output by the DLP optical engine, and constructing a multi-scale feature pyramid; A hierarchical multi-modal Transformer encoder connected to the image preprocessing module for receiving the multi-scale feature pyramid, the hierarchical multi-modal Transformer encoder loaded with a bad pixel type modal library containing dark spot, bright spot, and flicker bad pixel exclusive feature embedding vectors, and extracting feature vectors containing global pixel dependencies by embedding pixel-level bad pixel modal at the low layer, embedding local structure bad pixel modal at the middle layer, embedding global semantic bad pixel modal at the high layer, and performing feature fusion via a cross-scale feature fusion gate; A dynamic sparse mask multi-head attention calculation core connected to the hierarchical multi-modal Transformer encoder for receiving the feature vectors, texture entropy data, and ambient light data; the calculation core is: According to the texture entropy, a sparse mask is dynamically generated, a light crosstalk compensation factor obtained from the ambient light data is introduced, and the attention weight is calculated by the formula: to generate a bad pixel confidence heat map with the same resolution as the input image; A spatio-temporal compensation pipeline connected to the dynamic sparse mask multi-head attention calculation core for determining the bad pixel type according to the bad pixel confidence heat map, and dynamically selecting and executing corresponding spatial compensation, temporal compensation, or hybrid compensation strategies for permanent bad pixels, suspected dynamic bad pixels, or bad pixel clusters based on the identified conference projection, cinema projection, or AR glasses projection scene; A brightness adaptive calibrator connected to the spatio-temporal compensation pipeline for integrating data from the ambient light sensor and the micro-mirror array partition temperature sensor, and dynamically correcting the pixel brightness value after compensation strategy processing by the calibration formula: A compensation verification unit connected to the brightness adaptive calibrator for verifying the calibrated image using a loss function that fuses human visual system (HVS) perception loss and feeding back error information to the hierarchical multi-modal Transformer encoder for iterative optimization when the verification fails.

[0023] Also provided is a computer-readable storage medium having a computer program stored thereon, the program being executed by a processor to implement the DLP bad pixel calibration and compensation method based on the Transformer attention mechanism described in the above scheme.

[0024] ​The beneficial effects of this invention are: By using a hierarchical multimodal Transformer encoder and a dynamic sparse mask attention mechanism, the limitations of traditional local statistical methods are overcome, enabling global context awareness of the entire frame image. This allows for accurate differentiation between complex textures and real defects, improving the detection accuracy of atypical extreme points, clustered defects, and flickering defects from 82.3% to 99.2%, while reducing the false positive rate by 76% in complex textures and areas with dense defects. A dual-driven spatiotemporal compensation pipeline based on defect type and scene requirements is constructed. The optimal compensation strategy is dynamically selected according to the defect type and projection scene to solve the texture destruction problem caused by fixed interpolation direction. After compensation, the PSNR of the image is improved from 32.6dB to 42.5dB, and the edge retention rate exceeds 99%. At the same time, accurate performance adaptation is achieved in different scenarios such as conferences, cinemas, and AR. By using the micromirror health prediction module to pre-compensate for potential bad pixels, the system is upgraded from passive repair to active prevention, increasing the fault-free working time of the DLP system by 40%. Combined with brightness adaptive calibration through multi-sensor fusion, the effects of thermal drift and ambient light crosstalk are effectively suppressed, enabling the system to maintain stable performance under different working conditions. The dynamic sparse masking mechanism adaptively adjusts the computation density based on texture complexity, which significantly improves processing efficiency while ensuring accuracy. The processing latency is reduced from 17ms / frame to 7ms / frame, power consumption is reduced from 3.2W to 1.4W, the number of parameters is reduced by 37%, and the inference speed is increased by 2.3 times. By introducing a closed-loop verification mechanism that integrates HVS perceptual loss, the compensation results are optimally aligned with human subjective visual experience, solving the problem of high objective indicators but poor subjective experience. The text projection edge clarity rate reaches 99%, and the subjective invisibility rate of bad pixels in dark scenes of cinemas is improved to 98%. Attached Figure Description

[0025] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0026] Figure 1 This is a structural diagram of the system of the present invention.

[0027] Figure 2 This is a flowchart of the method of the present invention.

[0028] Figure 3 This is a feature map of the hierarchical multimodal Transformer encoder in the method of this invention.

[0029] Figure 4 This is a flowchart of the verification process in the method of the present invention. Detailed Implementation

[0030] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.

[0031] Example 1 like Figure 1 As shown, a DLP bad pixel labeling and compensation system based on Transformer attention mechanism includes an image preprocessing module, a hierarchical multimodal Transformer encoder, a dynamic sparse mask multi-head attention calculation core, a spatiotemporal compensation pipeline, a compensation verification unit, and a brightness adaptive calibrator. The image preprocessing module is used to perform adaptive histogram equalization, wavelet domain noise reduction, and texture complexity and optical crosstalk preprocessing on the raw image stream output by the DLP optical engine, and to construct a multi-scale feature pyramid. The hierarchical multimodal Transformer encoder is connected to the image preprocessing module to receive multi-scale feature pyramids. The hierarchical multimodal Transformer encoder is loaded with a bad point type modality library containing feature embedding vectors specific to dark spots, bright spots and flickering bad points. It extracts feature vectors containing global pixel dependencies by embedding pixel-level bad point modalities at the low level, local structural bad point modalities at the middle level, and global semantic bad point modalities at the high level, and performs feature fusion through cross-scale feature fusion gate. The dynamic sparse mask multi-head attention computation core is connected to a hierarchical multimodal Transformer encoder to receive feature vectors, texture entropy data, and ambient light data; the computation core is: A sparse mask is dynamically generated based on texture entropy, and an optical crosstalk compensation factor obtained from ambient light data is introduced, using the formula: Calculate attention weights to generate a bad pixel confidence heatmap with the same resolution as the input image; The spatiotemporal compensation pipeline is connected to the dynamic sparse mask multi-head attention calculation core, which is used to determine the type of bad point based on the bad point confidence heatmap, and dynamically select and execute the corresponding spatial compensation, temporal compensation or hybrid compensation strategy for permanent bad points, suspected dynamic bad points or bad point clusters based on the identified conference projection, cinema projection or AR glasses projection scene. The brightness adaptive calibrator is connected to the spatiotemporal compensation pipeline to integrate data from the ambient light sensor and the micromirror array zoned temperature sensor, and is calibrated using the following formula: Dynamically correct the pixel brightness values ​​after the compensation strategy is applied. The compensation verification unit is connected to the brightness adaptive calibrator and is used to utilize the loss function of the human visual system HVS perceptual loss. The calibrated image is verified, and error information is fed back to the hierarchical multi-modal Transformer encoder for iterative optimization when the verification fails.

[0032] In addition, the system of the embodiment further comprises a control processing unit, which controls the dynamic sparse mask multi-head attention calculation core to perform calculation through coordination signals.

[0033] The DLP optical engine generates an original projection image stream, integrates a bad pixel monitoring circuit and a micro-mirror flip response time sensor, and outputs a micro-mirror array state signal and flip delay data in real time, with a precision of ±0.1 μs; a micro-mirror flip response time acquisition circuit is added to provide hardware data support for potential bad pixel prediction; the layout of the micro-mirror array circuit is optimized to support partition temperature acquisition.

[0034] The original image of the image preprocessing module and the ambient light sensor data are subjected to serialization processing. First, adaptive histogram equalization is performed to enhance the contrast of the image. Then, a wavelet domain noise reduction algorithm is used to suppress noise while preserving high-frequency edge information in the image. Subsequently, the texture entropy of the image is calculated, which integrates edge density and gradient variance to quantify the texture complexity of the region. At the same time, the image is subjected to light crosstalk preprocessing based on ambient light data, focusing on suppressing the crosstalk effect in high brightness areas. Finally, the processed image is used to construct a multi-scale feature pyramid (scale factors are 1, 0.5, and 0.25), and the calculated texture entropy Figure 1 and output, providing multi-granularity input features for subsequent bad pixel detection and compensation, solving the edge blurring problem caused by traditional bilateral filtering, and providing input for subsequent dynamic sparse masks through texture entropy and light crosstalk preprocessing, improving the anti-interference ability of bad pixel detection.

[0035] The hierarchical multi-modal Transformer encoder inputs the preprocessed multi-scale features and the bad pixel type modal library. Through position encoding and bad pixel type modal embedding, low-level pixel-level, middle-level local structure, and high-level global semantics are extracted through 12 encoder layers to extract global pixel dependency relationships, including multi-head attention, local receptive field enhancement, and cross-scale feature fusion gate. Finally, a high-dimensional feature vector is outputted, which integrates micro-mirror structure features and bad pixel type features, with a parameter quantity reduction of 37%, a speed improvement of 2.3 times, and a type feature adaptability improvement of 40%. Compared with the traditional Transformer encoder, the recall rate of different type bad pixel feature extraction is improved by 7-12%, among which the recall rate of flicker bad pixels is improved from 92.3% to 99.5%.

[0036] The dynamic sparse mask multi-head attention calculation core inputs the feature vector output by the hierarchical multi-modal Transformer encoder, the texture entropy data, and the ambient light data. The input feature vector is linearly transformed to generate a query vector (Q), a key vector (K), and a value vector (V). The query vector is used to focus on the features of the current pixel, the key vector is used to encode the context information, and the value vector carries the reference information for repair. Then, the dynamic sparse mask calculation is performed. The sparsity is dynamically adjusted according to the texture entropy of the image. When the texture entropy is greater than the threshold 0.6, the sparsity is set to 0.3 to retain more context information for high-texture complexity regions such as text edges. When the texture entropy is lower than the threshold, the sparsity is set to 0.7 to reduce invalid calculations and improve efficiency for low-texture complexity regions such as solid color backgrounds.

[0037] The calculation of the light crosstalk factor is associated with the ambient light data, and its value is the product of 0.1 and the ambient light intensity. It is mainly used to correct the weight of high brightness regions. After scaling the dot product of the query vector and the key vector, the dynamic sparse mask and the light crosstalk factor are added, and then the Softmax normalization processing is performed to generate the final attention weight.

[0038] The dynamic sparse mask multi-head attention calculation core outputs a bad pixel confidence heat map with the same resolution as the input and a value range of [0, 1]. The dynamic sparse mask and the light crosstalk factor are introduced to balance the detection accuracy and computational efficiency, and to eliminate the false high confidence caused by light crosstalk.

[0039] The bad pixel confidence heat map output has the same resolution as the input image and a value range of [0, 1]. 0 is a normal pixel, and 1 is a confirmed bad pixel. If the confidence is greater than 0.85, it is determined as a permanent bad pixel and compensation is started. If the confidence is between 0.4 and 0.85, it is a suspected dynamic bad pixel and time domain verification is started. If the confidence is less than 0.4, it is a normal pixel and is directly output.

[0040] The spatiotemporal compensation pipeline is a three-level compensation architecture, as shown in the following table:

[0041] The brightness adaptive calibrator performs multi-sensor fusion calculation, and the formula is: where C is the calibrated brightness value, B is the compensation estimate value, E is the ambient light intensity, and γ is the light crosstalk suppression coefficient, which is between 0.8 and 1.0 and decreases as the ambient light intensity E increases. is the temperature drift value; is the first dynamic adjustment coefficient, which is a function based on the attention weight w, the pixel coordinates , and the ambient light intensity E; is the second dynamic adjustment coefficient, which is a function based on the temperature gradient , a historical drift curve H, and partition temperature data .

[0042] A multi-scale feature extraction network is constructed by a Transformer encoder, combined with position encoding and learnable bad pixel pattern embedding, to realize global dependency modeling between the entire frame of pixels. This mechanism breaks through the traditional local neighborhood restriction and improves the perception ability of complex textures and bad pixel clusters. A spatial sparse mask is introduced in attention calculation to suppress the interference of irrelevant areas, generate a high-resolution bad pixel confidence heat map, and realize accurate detection of atypical extreme points and cluster-shaped bad pixels. A spatial-temporal-hybrid three-level compensation pipeline is constructed to dynamically guide the interpolation direction, reference frame number and brightness compensation strength through attention weight, realizing multi-scene adaptive compensation. Environmental light and temperature sensors are integrated to construct a dynamic calibration function based on attention weight, realizing real-time suppression of thermal drift and environmental interference. Verification targets based on reconstruction loss and attention entropy are introduced to realize real-time evaluation of compensation results and iterative optimization of system parameters, forming a self-improving intelligent processing closed loop.

[0043] Embodiment Two As shown in Figure 2 , a DLP bad pixel calibration and compensation method based on the Transformer attention mechanism includes the following steps: Step One, the original image stream output by the DLP optical engine is subjected to adaptive histogram equalization, wavelet domain noise reduction, and texture complexity and light crosstalk preprocessing, and a multi-scale feature pyramid is constructed. Through adaptive histogram equalization, the contrast of the image is enhanced. Wavelet domain noise reduction is used to preserve high-frequency edges. Texture complexity and light crosstalk preprocessing specifically includes: calculating the texture entropy of the image, which integrates edge density and gradient variance, and based on the ambient light sensor data, light crosstalk suppression is performed on the high brightness areas in the image; the calculated texture entropy map and the multi-scale feature pyramid are output together for subsequent bad pixel detection. The constructed multi-scale feature pyramid scales = [1, 0.5, 0.25] provides image features of original size, half size and quarter size for subsequent intelligent algorithms.

[0044] Step Two, input the multi-scale feature pyramid into the layered multi-modal Transformer encoder for global feature extraction, and input the output feature vector into the dynamic sparse mask multi-head attention calculation core to generate a bad pixel confidence heat map.

[0045] The hierarchical multi-modal Transformer encoder first loads a bad point type modal library containing dark point, bright point and flicker bad point specific feature embedding vectors. The encoder embeds pixel-level bad point modalities at the low layer, local structure bad point modalities at the middle layer, and global semantic bad point modalities at the high layer. Cross-scale feature fusion gates are used to fuse features from different scales.

[0046] As shown in FIG. 1, feature extraction is achieved by combining position encoding with modal embedding to implement a 12-layer encoder layer: 1-4 layers of low-layer embedding of pixel-level bad point modalities, such as dark point brightness attenuation features; 5-8 layers of middle-layer embedding of local structure modalities, such as cluster-shaped bad point distribution features; and 9-12 layers of high-layer embedding of global semantic modalities, such as text area and image area scene semantics. The encoder layer integrates multi-head attention, local receptive field enhancement and cross-scale feature fusion gates to achieve global pixel dependency modeling and micro-lens array structure feature capture. Figure 3

[0047] The feature vector output by the hierarchical multi-modal Transformer encoder enters the dynamic sparse mask multi-head attention calculation core to generate a Query vector focusing on the current pixel, a Key vector encoding the context environment, and a Value vector carrying the repair reference. The attention weight calculation formula is as follows: where d is the vector dimension, is a dynamic sparse mask, is a light crosstalk compensation factor; =f(texture entropy), i.e., a sparse mask is dynamically generated according to the texture entropy, where when the texture entropy is higher than a set threshold (0.6), a first sparsity (0.3) is set, and when the texture entropy is lower than the set threshold, a second sparsity (0.7) higher than the first sparsity (0.3) is set; =0.1x ambient light intensity, i.e., a light crosstalk compensation factor obtained from ambient light data is introduced, and a bad point confidence heat map with the same resolution as the input image is finally generated; where the pixel value represents the probability that the point is a bad point. When the confidence of a pixel is greater than 0.85, it is determined to be a permanent bad point, such as a dark point or a bright point; when the confidence of a pixel is between 0.4 and 0.85, it is determined to be a suspected dynamic bad point, such as a flicker bad point; and when the confidence of a pixel is less than 0.4, it is determined to be a normal pixel.

[0048] Step three, determine the bad point type according to the bad point confidence heat map, and start the bad point type-scene demand dual-driven spatio-temporal compensation pipeline to dynamically select and execute the corresponding spatial compensation, temporal compensation or mixed compensation strategy for different types of bad points.

[0049] ​​Bad pixel type-scene requirement dual-driven spatio-temporal compensation pipeline, automatically identify conference projection, cinema projection or AR glasses projection scene through image resolution and brightness parameters; For permanent bad pixels, according to the scene type, the spatial compensation is carried out by radial basis interpolation guided by attention weight, attention weighted bilinear interpolation or lightweight radial basis interpolation, the conference scene adopts radial basis interpolation guided by attention weight, the processing delay is less than 5ms, the cinema scene adopts attention weighted bilinear interpolation, the edge retention rate is greater than 99%, and the AR scene adopts lightweight radial basis interpolation, and the power consumption is reduced by 30%; For suspected dynamic bad pixels, after excluding false bad pixels through multi-frame motion consistency analysis, time domain interpolation is performed, wherein the number of reference frames for analysis is dynamically adjusted by attention weight, generally 3-5 frames; For bad pixel clusters, a spatio-temporal joint optimization algorithm is used for hybrid compensation, such as spatial difference and multi-frame iterative compensation, wherein the cinema scene additionally performs 3-frame optimization to improve dark field accuracy; It also includes potential bad pixel pre-compensation, collects micro-mirror flip response time sensor data, and then predicts potential bad pixels in a preset time period based on flip response time through LSTM network; pre-compensation is performed on the identified potential bad pixels, including adjusting the brightness compensation coefficient; wherein the normal mirror flip response time is less than 5us, and the degradation is greater than 10us, the prediction preset time is 100 hours, and the brightness compensation coefficient is adjusted from 1.0 to 1.2-1.5 in advance for the micro-mirror with flip delay of 8-10us.

[0050] Step four, integrate the data of ambient light sensor and micro-mirror array partition temperature sensor, and dynamically correct the pixel brightness value processed by compensation strategy through brightness self-adaptive calibration function; wherein the formula is: , wherein C is the calibrated brightness value, B is the compensation estimated value, E is the ambient light intensity, and γ is the light crosstalk suppression coefficient, whose value is between 0.8 and 1.0, and decreases with the increase of ambient light intensity E; is the temperature drift value; is the first dynamic adjustment coefficient, whose value is a function based on attention weight w, pixel coordinates ; is the second dynamic adjustment coefficient, whose value is a function based on temperature gradient , historical drift curve H and partition temperature data ; and the first dynamic adjustment coefficient , when identified as a cinema projection scene, its output value is increased by 20% based on the benchmark; the second dynamic adjustment coefficient , when the partition temperature data indicates that the pixel is located in the high temperature area of the optical engine center, its output value is increased by 20% based on the benchmark.

[0051] Step five, as shown in Figure 4 , the calibrated image is verified by a loss function fusing human visual system (HVS) perceptual loss, if the verification fails, an error map is generated and fed back to S2 for iterative optimization, if the verification passes, the final image is output; wherein the loss function fusing human visual system (HVS) perceptual loss is : , wherein, is the reconstruction error between the compensation area and the normal area; is the attention entropy; is the HVS perceptual loss, which gives a first weight to the error of the dark field area and a second weight to the error of the texture edge area, and the first weight is greater than the second weight; is the edge consistency error; is the predefined weight coefficient of the attention entropy, the HVS perceptual loss and the edge consistency error; and during iterative optimization, the verification threshold of the edge consistency error is dynamically adjusted according to the scene type.

[0052] The hierarchical multi-modal Transformer breaks through the traditional neighborhood restriction and single feature embedding bottleneck, realizes the whole frame pixel correlation analysis and accurate matching of different types of bad point features, and reduces the misjudgment rate of complex texture area by 76% and the misjudgment rate of light crosstalk scene by 68%. Through dynamic adjustment of compensation strategy by attention weight and scene recognition, the speed demand of conference projection, the precision demand of cinema projection and the power demand of AR projection are considered. Integrate the partition temperature sensor and the micro-mirror flip response time sensor to realize accurate compensation of thermal drift and pre-compensation of potential bad points, DLP system MTBF is improved by 40%, and PSNR attenuation caused by thermal drift is reduced from 3.2dB to 0.6dB. Fusing HVS perceptual loss and dynamic verification threshold, solve the technical difficulty of high objective PSNR but poor subjective experience, the edge clarity rate of text projection reaches 99%, and the subjective invisibility rate of cinema dark field bad points is improved to 98%. The specific contents are as follows:

[0053] Embodiment three A computer readable storage medium having stored thereon a computer program, when the program is executed by a processor, realizing the DLP bad point pixel calibration and compensation method based on Transformer attention mechanism of embodiment two.

[0054] With the above ideal embodiments according to the present application as the inspiration, through the above description, relevant staff can make various changes and modifications without deviating from the technical idea of the present application. The technical scope of the present application is not limited to the content of the specification, and must be determined according to the scope of the claims.

Claims

1. A DLP method for identifying and compensating for defective pixels based on Transformer attention mechanism, characterized in that, Includes the following steps: S1. Adaptive histogram equalization, wavelet domain noise reduction, and texture complexity and optical crosstalk preprocessing are performed on the raw image stream output by the DLP optical engine, and a multi-scale feature pyramid is constructed. S2. Input the multi-scale feature pyramid into the hierarchical multimodal Transformer encoder for global feature extraction, and input the output feature vector into the dynamic sparse mask multi-head attention calculation core to generate a bad pixel confidence heatmap. S3. Determine the type of bad pixel based on the confidence heatmap of bad pixels, and start the dual-drive spatiotemporal compensation pipeline of bad pixel type and scenario requirements to dynamically select and execute the corresponding spatial compensation, temporal compensation or hybrid compensation strategy for different types of bad pixels. S4. Data from the integrated ambient light sensor and the micromirror array partitioned temperature sensor are used to dynamically correct the pixel brightness values ​​after the compensation strategy is applied through a brightness adaptive calibration function. S5. Validate the calibrated image using the loss function of the human visual system (HVS) perceptual loss. If the validation fails, generate an error map and feed it back to S2 for iterative optimization. If the validation passes, output the final image.

2. The DLP bad pixel labeling and compensation method based on Transformer attention mechanism according to claim 1, characterized in that, The texture complexity and optical crosstalk preprocessing in S1 specifically include: The texture entropy of the image is calculated, which combines edge density and gradient variance. Based on ambient light sensor data, light crosstalk suppression is performed on high-brightness areas in the image. The calculated texture entropy map and the multi-scale feature pyramid are output together for subsequent bad pixel detection.

3. The DLP bad pixel labeling and compensation method based on Transformer attention mechanism according to claim 1, characterized in that, The process of global feature extraction in the hierarchical multimodal Transformer encoder in S2 is as follows: Load a bad point type modality library containing feature embedding vectors specific to dark spots, bright spots, and flickering bad points. Embed pixel-level bad point modalities in the low layer of the encoder, embed local structural bad point modalities in the middle layer, and embed global semantic bad point modalities in the high layer. Through cross-scale feature fusion gate, features from different scales are fused.

4. The DLP bad pixel identification and compensation method based on Transformer attention mechanism according to claim 2, characterized in that, The process of generating the bad pixel confidence heatmap in the core of dynamic sparse mask multi-head attention computation in S2 is as follows: A sparse mask is dynamically generated based on texture entropy. When the texture entropy is higher than a set threshold, a first sparsity is set; when the texture entropy is lower than the set threshold, a second sparsity higher than the first sparsity is set. An optical crosstalk compensation factor obtained from ambient light data is introduced, and attention weights are calculated using the following formula: ， Where Q, K, and V are the query, key, and value vectors, respectively, and d is the vector dimension. For dynamic sparse masks, This is the optical crosstalk compensation factor; A bad pixel confidence heatmap with the same resolution as the input image is generated based on attention weights, where the pixel value represents the probability that the point is a bad pixel.

5. The DLP bad pixel identification and compensation method based on Transformer attention mechanism according to claim 4, characterized in that, In S3, the type of bad pixel is determined based on the bad pixel confidence heatmap, specifically as follows: A pixel is considered a permanent bad pixel when its confidence level is greater than 0.

85. When the confidence level of a pixel is between 0.4 and 0.85, it is judged as a suspected dynamic bad pixel; A pixel is considered a normal pixel when its confidence level is less than 0.

4.

6. The DLP bad pixel identification and compensation method based on Transformer attention mechanism according to claim 1, characterized in that, In S3, the dual-driven spatiotemporal compensation pipeline for bad pixel types and scenario requirements is implemented as follows: The system automatically identifies conference projection, cinema projection, or AR glasses projection scenarios based on image resolution and brightness parameters. For permanent defects, spatial compensation is performed using radial basis interpolation guided by attention weights, attention-weighted bilinear interpolation, or lightweight radial basis interpolation, depending on the scene type. For suspected dynamic defects, false defects are eliminated through multi-frame motion consistency analysis before temporal interpolation is performed, where the number of reference frames used for analysis is dynamically adjusted by the attention weights. For defect clusters, a spatiotemporal joint optimization algorithm is used for hybrid compensation. It also includes potential defective pixel pre-compensation, specifically: acquiring micromirror flip response time sensor data; predicting potential defective pixels within a preset time period based on the flip response time using an LSTM network; and performing pre-compensation on the identified potential defective pixels, including adjusting their brightness compensation coefficient.

7. The DLP bad pixel identification and compensation method based on Transformer attention mechanism according to claim 1, characterized in that, In S4, the brightness adaptive calibration function dynamically corrects the compensation value using the following formula: ， Where C is the calibrated brightness value, B is the compensated estimated value, E is the ambient light intensity, and γ is the optical crosstalk suppression coefficient, which is between 0.8 and 1.0 and decreases as the ambient light intensity E increases. This represents the temperature drift value. This is the first dynamic adjustment coefficient, whose value is based on the attention weight w and pixel coordinates. The function; This is the second dynamic adjustment factor, whose value is based on the temperature gradient. Historical drift curve H and zone temperature data The function; First dynamic adjustment coefficient When identified as a cinema projection scene, its output value is increased by 20% compared to the baseline; Second dynamic adjustment coefficient When the partition temperature data This indicates that when a pixel is located in the high-temperature region at the center of the optical engine, its output value is increased by 20% compared to the baseline.

8. The DLP bad pixel identification and compensation method based on Transformer attention mechanism according to claim 1, characterized in that, The loss function in S5 that incorporates the perceptual loss of the human visual system (HVS). for: ， in, To compensate for the reconstruction error between the local area and the normal area; For attention entropy; The HVS perceptual loss assigns a first weight to the error in the dark field region and a second weight to the error in the texture edge region, with the first weight being greater than the second weight. This refers to edge consistency error; The weighting coefficients for predefined attention entropy, HVS perceptual loss, and edge consistency error; During iterative optimization, the verification threshold for edge consistency error is dynamically adjusted according to the scenario type.

9. A system for implementing the DLP bad pixel identification and compensation method based on the Transformer attention mechanism as described in any one of claims 1 to 8, characterized in that, include: The image preprocessing module is used to perform adaptive histogram equalization, wavelet domain noise reduction, and texture complexity and optical crosstalk preprocessing on the raw image stream output by the DLP optical engine, and to construct a multi-scale feature pyramid. The hierarchical multimodal Transformer encoder, connected to the image preprocessing module, is used to receive a multi-scale feature pyramid. The hierarchical multimodal Transformer encoder is loaded with a bad point type modality library containing feature embedding vectors specific to dark spots, bright spots, and flickering bad points. It extracts feature vectors containing global pixel dependencies by embedding pixel-level bad point modalities at the low level, local structural bad point modalities at the middle level, and global semantic bad point modalities at the high level, and performs feature fusion through a cross-scale feature fusion gate. The dynamic sparse mask multi-head attention computation core is connected to the hierarchical multimodal Transformer encoder to receive feature vectors, texture entropy data, and ambient light data; the computation core is: A sparse mask is dynamically generated based on texture entropy, and an optical crosstalk compensation factor obtained from ambient light data is introduced, using the formula: Calculate attention weights to generate a bad pixel confidence heatmap with the same resolution as the input image; The spatiotemporal compensation pipeline is connected to the dynamic sparse mask multi-head attention computing core. It is used to determine the type of bad pixel based on the bad pixel confidence heatmap, and dynamically select and execute the corresponding spatial compensation, temporal compensation or hybrid compensation strategy for permanent bad pixels, suspected dynamic bad pixels or bad pixel clusters based on the identified conference projection, cinema projection or AR glasses projection scene. A brightness adaptive calibrator, connected to a spatiotemporal compensation pipeline, integrates data from an ambient light sensor and a micromirror array zoned temperature sensor, and uses a calibration formula: Dynamically correct the pixel brightness values ​​after the compensation strategy is applied. The compensation verification unit, connected to the brightness adaptive calibrator, is used to utilize the loss function of the fused human visual system HVS perceptual loss. The calibrated image is verified, and if the verification fails, the error information is fed back to the hierarchical multimodal Transformer encoder for iterative optimization.

10. A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the DLP bad pixel labeling and compensation method based on the Transformer attention mechanism as described in any one of claims 1 to 8.

Images