A video fault tolerance method, device, equipment and storage medium

Abstract

The application discloses a video fault tolerance method and device, equipment and storage medium, and relates to the technical field of video image processing. The method comprises the following steps: detecting an abnormal event, and determining the confidence of the abnormal event; determining a target working mode according to the abnormal event and the confidence of the abnormal event; if the target working mode is a single-channel straight-through mode, selecting an infrared image or a visible light image as a target output object according to the confidence of the abnormal event; if the target working mode is a visible light dominant mode or an infrared dominant mode, determining the respective weights of the infrared image and the visible light image according to the target working mode, and fusing the infrared image and the visible light image according to the respective weights of the infrared image and the visible light image to obtain the target output object; and outputting the target output object. The method can enable the system to output more usable pictures under different abnormal conditions.

Description

Technical Field

[0001] This application relates to the field of video image processing technology, and in particular to a video fault tolerance method, apparatus, device and storage medium. Background Technology

[0002] Dual-channel (infrared + visible light) fusion systems are prone to various failure modes. Regarding sensor failure, vibration can cause lens loosening or wiper malfunction, while high temperatures can cause the infrared sensor's NUC (Non-Uniformity Correction) to fail, resulting in stripes in the image. Regarding extreme optical interference, direct sunlight can cause momentary blinding (overexposure) of the visible light, and mud splashes can obstruct the lens. Regarding synchronization anomalies, electromagnetic interference can cause misalignment of the timestamps of the two video frames, resulting in motion ghosting after fusion. Regarding environmental degradation, extremely low illumination leads to a very poor visible light signal-to-noise ratio.

[0003] However, most existing dual-channel fusion systems assume the input source is normal and use a fixed algorithm. Once the aforementioned anomalies occur, the system often continues to force fusion, resulting in severe artifacts in the output (such as full-screen noise, ghosting, and light spots), sometimes even worse than a single-channel image. Therefore, how to solve these technical defects has become a pressing technical problem for those skilled in the art. Summary of the Invention

[0004] The purpose of this application is to provide a video fault tolerance method, apparatus, device, and storage medium that enables the system to output more usable images under different abnormal conditions.

[0005] To address the aforementioned technical problems, this application provides a video fault tolerance method, comprising:

[0006] Detect abnormal events and determine the confidence level of the abnormal events;

[0007] The target operating mode is determined based on the abnormal events and their confidence levels.

[0008] If the target working mode is a single-channel pass-through mode, then an infrared image or a visible light image is selected as the target output object based on the confidence level of the abnormal event.

[0009] If the target working mode is a visible light dominant mode or an infrared dominant mode, then the weights corresponding to the infrared image and the visible light image are determined according to the target working mode, and the infrared image and the visible light image are fused according to the weights corresponding to the infrared image and the visible light image to obtain the target output object.

[0010] Output the target output object.

[0011] In some embodiments, outputting the target output object includes:

[0012] Within the transition window, the gradient weight is determined based on the transition frame number and the total number of transition frames.

[0013] The transitional output object is obtained by fusing the target output object with the infrared and visible light images in the original working mode based on the gradual weighting.

[0014] The transition output object is output within the transition window, and the target output object is output after the transition window.

[0015] In some embodiments, detecting anomaly events and determining the confidence level of the anomaly events includes:

[0016] Detect visible light overexposure or underexposure events and determine the confidence level of the visible light overexposure or underexposure events;

[0017] Detect lens contamination or obstruction events and determine the confidence level of said lens contamination or obstruction events;

[0018] Detect infrared non-uniformity correction anomaly events and determine the confidence level of the infrared non-uniformity correction anomaly events;

[0019] Detect infrared dead pixel surge events and determine the confidence level of the infrared dead pixel surge events;

[0020] Detect time synchronization anomalies and determine the confidence level of the time synchronization anomalies.

[0021] In some embodiments, determining the target operating mode based on the anomalous event and its confidence level includes:

[0022] If the abnormal event that occurs includes the time synchronization abnormal event, then the target mode is determined to be a single-channel pass-through mode;

[0023] If the abnormal event does not include the time synchronization abnormal event and the confidence level of the abnormal event determines that both infrared and visible light are severely faulty, then the target working mode is determined to be single-channel pass-through mode.

[0024] If the abnormal event does not include the time synchronization abnormal event and the infrared serious fault is determined based on the confidence level of the abnormal event, then the target working mode is determined to be the visible light dominant mode.

[0025] If the abnormal event does not include the time synchronization abnormal event and the visible light is determined to be severely faulty based on the confidence level of the abnormal event, then the target operating mode is determined to be the infrared-dominated mode.

[0026] In some embodiments, the methods for determining whether there is a serious fault in visible light and whether there is a serious fault in infrared light include:

[0027] If the confidence level of a visible light overexposure or underexposure event exceeds the first preset threshold, then there is a serious visible light failure.

[0028] If the confidence level of a lens contamination or obstruction event corresponding to visible light exceeds the second preset threshold, then there is a serious visible light malfunction.

[0029] If the confidence level of an abnormal event in infrared non-uniformity correction exceeds the third preset threshold, then there is a serious infrared malfunction.

[0030] If the confidence level of the infrared dead pixel surge event exceeds the fourth preset threshold, then the infrared sensor is in serious condition.

[0031] If the confidence level of the lens contamination or obstruction event corresponding to the infrared exceeds the fifth preset threshold, then the infrared is considered to be seriously faulty.

[0032] In some embodiments, the target operating mode is maintained for at least a preset number of frames.

[0033] In some embodiments, the entry threshold for the visible light overexposure or underexposure event is higher than the exit threshold for the visible light overexposure or underexposure event; the entry threshold for the lens contamination or occlusion event is higher than the exit threshold for the lens contamination or occlusion event; the entry threshold for the infrared non-uniformity correction anomaly event is higher than the exit threshold for the infrared non-uniformity correction anomaly event; the entry threshold for the infrared dead pixel surge event is higher than the exit threshold for the infrared dead pixel surge event; and the entry threshold for the time synchronization anomaly event is higher than the exit threshold for the time synchronization anomaly event.

[0034] To address the aforementioned technical problems, this application also provides a video fault-tolerant device, comprising:

[0035] A detection unit is used to detect abnormal events and determine the confidence level of the abnormal events;

[0036] The determining unit is used to determine the target operating mode based on the abnormal event and the confidence level of the abnormal event;

[0037] The selection unit is used to select either an infrared image or a visible light image as the target output object based on the confidence level of the abnormal event if the target working mode is a single-channel pass-through mode.

[0038] The fusion unit is used to determine the weights of the infrared image and the visible light image according to the target working mode if the target working mode is a visible light dominant mode or an infrared dominant mode, and to fuse the infrared image and the visible light image according to the weights of the infrared image and the visible light image to obtain the target output object.

[0039] The output unit is used to output the target output object.

[0040] To address the aforementioned technical problems, this application also provides an electronic device, comprising:

[0041] Memory, used to store computer programs;

[0042] A processor is configured to implement the video fault tolerance method described above when executing the computer program.

[0043] To address the aforementioned technical problems, this application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the video fault tolerance method described above.

[0044] The video fault tolerance method provided in this application includes: detecting abnormal events and determining the confidence level of the abnormal events; determining a target operating mode based on the abnormal events and their confidence levels; if the target operating mode is a single-channel pass-through mode, selecting an infrared image or a visible light image as the target output object based on the confidence level of the abnormal events; if the target operating mode is a visible light-dominated mode or an infrared-dominated mode, determining the weights corresponding to the infrared image and the visible light image respectively based on the target operating mode, and fusing the infrared image and the visible light image according to their respective weights to obtain the target output object; and outputting the target output object.

[0045] As can be seen, the video fault tolerance method provided in this application detects abnormal events and determines their confidence level. Based on the abnormal event and its confidence level, it switches the operating mode, automatically switching to a more suitable fusion mode or single-channel pass-through output under sensor failure or extreme interference, thus making the footage more usable under abnormal conditions. Specifically, the single-channel pass-through mode can reduce the risk of motion ghosting caused by time misalignment and avoid interference with the judgment of personnel and vehicle status.

[0046] The video fault-tolerant device, equipment, and computer-readable storage medium provided in this application all have the aforementioned technical effects. Attached Figure Description

[0047] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the prior art and embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0048] Figure 1A flowchart illustrating a video fault tolerance method provided in an embodiment of this application;

[0049] Figure 2 This is a schematic diagram of a video fault tolerance process provided in an embodiment of this application;

[0050] Figure 3 This application provides a schematic diagram of anomaly detection and state machine decision logic in an embodiment of the present application.

[0051] Figure 4 A schematic diagram of a video fault-tolerant device provided in an embodiment of this application;

[0052] Figure 5 This is a schematic diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0053] The core of this application is to provide a video fault tolerance method, apparatus, device, and storage medium that enables the system to output more usable images under different abnormal conditions.

[0054] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0055] Please refer to Figure 1 , Figure 1 This is a flowchart illustrating a video fault tolerance method provided in an embodiment of this application. (Refer to...) Figure 1 As shown, the method includes:

[0056] S101: Detect an abnormal event and determine the confidence level of the abnormal event.

[0057] The video fault tolerance method provided in this application adopts a closed-loop architecture of event detection + state machine decision + multi-mode execution in its dual-channel fusion system. Step S101 aims to detect abnormal events and determine the confidence level of the abnormal events. The confidence level of the abnormal events can be normalized to [0,1]. A higher confidence level indicates a more severe anomaly.

[0058] In some embodiments, detecting anomaly events and determining the confidence level of the anomaly events includes:

[0059] Detect visible light overexposure or underexposure events and determine the confidence level of the visible light overexposure or underexposure events;

[0060] Detect lens contamination or obstruction events and determine the confidence level of said lens contamination or obstruction events;

[0061] Detect infrared non-uniformity correction anomaly events and determine the confidence level of the infrared non-uniformity correction anomaly events;

[0062] Detect infrared dead pixel surge events and determine the confidence level of the infrared dead pixel surge events;

[0063] Detect time synchronization anomalies and determine the confidence level of the time synchronization anomalies.

[0064] In this embodiment, abnormal events include five categories: visible light overexposure or underexposure events, lens contamination or obstruction events, infrared non-uniformity correction abnormal events, infrared dead pixel sudden increase events, and time synchronization abnormal events.

[0065] Visible light overexposure / underexposure events are recorded as Taking an 8-bit image as an example, the percentage of pixels in highlights and shadows is calculated as follows:

[0066] ;

[0067] .

[0068] Pick The confidence level is calculated as follows:

[0069] .

[0070] , is used to normalize the confidence level to the interval [0,1].

[0071] in, The number of pixels required to meet the conditions. This represents the total number of pixels in the image. This refers to the visible light brightness value. The percentage of overexposed pixels, This represents the percentage of underexposed pixels.

[0072] An alarm will be triggered when the percentage exceeds 30%, and when it exceeds 70%, it will be judged as a serious exposure anomaly (overexposure or underexposure).

[0073] For example, an alarm is triggered when the percentage of overexposed pixels exceeds 30%, and it is determined to be overexposed when it exceeds 70%.

[0074] Lens contamination / obstruction incidents are recorded as follows: The Laplace variance (sharpness) of visible light and infrared light can be calculated separately, as follows:

[0075] ;

[0076] .

[0077] Set threshold and ,For example, , Threshold and The values ​​can be adjusted according to the lens / resolution calibration.

[0078] The confidence level is calculated as follows:

[0079] ;

[0080] .

[0081] in, This is the variance calculation function. For the Laplace operator, Visible light image, Infrared image, This represents the Laplace variance of visible light (reflecting sharpness). This represents the infrared Laplacian variance (reflecting sharpness). The threshold for occlusion events (a resolution scale, calibrated). The severity threshold for occlusion events (a resolution scale, calibrated).

[0082] when Gao Er When the light level is low, determine if the visible light lens is contaminated / obstructed; when... Gao Er When the value is low, infrared lens contamination / occlusion is determined; when both are high, bilateral occlusion is determined and a safety net strategy is triggered.

[0083] To prevent false alarms in textureless scenes such as night skies or flat walls, the detection algorithm can incorporate temporal baseline comparison. An occlusion alarm is only triggered when the current sharpness of one channel is significantly lower than the historical average (Baseline), while the other channel still maintains a certain level of texture.

[0084] Infrared NUC abnormal events are recorded as These are the prominent horizontal and vertical stripes used to detect NUC failure characteristics. The standard deviation of the row and column means can be calculated as follows:

[0085] ;

[0086] .

[0087] .

[0088] The confidence level is calculated as follows:

[0089] .

[0090] in, This is the function for calculating standard deviation. This is the mean calculation function. This indicates calculation by row. This indicates calculation by column. The standard deviation of the row mean. The standard deviation of the column mean. This is an index of stripe noise intensity. This is the threshold for NUC anomaly entry (obtained through calibration). This is the threshold for severe NUC anomalies (obtained through calibration).

[0091] when Exceeding the threshold Triggering a NUC exception, exceeding The condition is classified as severe stripes (threshold can be calibrated).

[0092] An infrared NUC anomaly detection scheme based on stripe and noise characteristics uses the standard deviation of row / column means to detect fixed-pattern noise stripes, which serves as a basis for judging the status of the infrared core, thereby improving the feasibility and calibrability of infrared anomaly identification.

[0093] The event of a sudden increase in infrared dead pixels is recorded as follows: The proportion of outliers that differ significantly from the neighborhood median. and maintain baseline dead pixel rate (Factory calibration or periodic self-test update). Sudden increment is defined as follows:

[0094] .

[0095] The confidence level is calculated as follows:

[0096] .

[0097] in, This represents the percentage of bad pixels in the current frame. The baseline percentage of defective pixels. This represents a sudden increase in the number of bad points.

[0098] An alarm can be triggered when the increase in dead pixels exceeds 1%, and a serious aging abnormality can be determined when the increase in dead pixels exceeds 3%.

[0099] Time synchronization anomalies are recorded as Monitor the difference between the two timestamps. The confidence level is calculated as follows:

[0100] .

[0101] in, This is the difference in timestamps (milliseconds) between the two video frames. The threshold for time synchronization anomalies (milliseconds, 40 for a 25fps scenario). The threshold for severe time synchronization anomalies (milliseconds, 140 can be used in a 25fps scenario).

[0102] When the error exceeds one frame, it can be determined as a synchronization loss, i.e., a synchronization anomaly. This type of anomaly is easily manifested as motion ghosting in the fusion result.

[0103] S102: Determine the target working mode based on the abnormal event and its confidence level.

[0104] Step S102 aims to make a working mode decision. The working modes include four modes: normal fusion mode, single-channel pass-through mode, visible light dominant mode, and infrared dominant mode.

[0105] M0, the normal blending mode, performs dual-path blending and necessary detail enhancement. M1, the infrared-dominant mode, introduces visible light information only in the effective visible light area (not overexposed / underexposed and not contaminated / blocked), with infrared dominating the remaining areas. M2, the visible light-dominant mode, reduces infrared contribution and can be combined with stripe and dead pixel suppression to reduce the risk of infrared noise contamination of the output. M3, the single-channel pass-through mode, directly outputs the healthy channel. or This bypasses the fusion algorithm and serves as a backup output in case of synchronization loss or severe failure on both sides. Channel selection prioritizes the channel with the lowest sum of event confidence.

[0106] In some embodiments, determining the target operating mode based on the anomalous event and its confidence level includes:

[0107] If the abnormal event that occurs includes the time synchronization abnormal event, then the target mode is determined to be a single-channel pass-through mode;

[0108] If the abnormal event does not include the time synchronization abnormal event and the confidence level of the abnormal event determines that both infrared and visible light are severely faulty, then the target working mode is determined to be single-channel pass-through mode.

[0109] If the abnormal event does not include the time synchronization abnormal event and the infrared serious fault is determined based on the confidence level of the abnormal event, then the target working mode is determined to be the visible light dominant mode.

[0110] If the abnormal event does not include the time synchronization abnormal event and the visible light is determined to be severely faulty based on the confidence level of the abnormal event, then the target operating mode is determined to be the infrared-dominated mode.

[0111] In some embodiments, the methods for determining whether there is a serious fault in visible light and whether there is a serious fault in infrared light include:

[0112] If the confidence level of a visible light overexposure or underexposure event exceeds the first preset threshold, then there is a serious visible light failure.

[0113] If the confidence level of a lens contamination or obstruction event corresponding to visible light exceeds the second preset threshold, then there is a serious visible light malfunction.

[0114] If the confidence level of an abnormal event in infrared non-uniformity correction exceeds the third preset threshold, then there is a serious infrared malfunction.

[0115] If the confidence level of the infrared dead pixel surge event exceeds the fourth preset threshold, then the infrared sensor is in serious condition.

[0116] If the confidence level of the lens contamination or obstruction event corresponding to the infrared exceeds the fifth preset threshold, then the infrared is considered to be seriously faulty.

[0117] When multiple abnormal events occur simultaneously, the following priority rule applies:

[0118] Time synchronization anomalies have the highest priority. Upon triggering, the system enters M3 single-channel pass-through mode to avoid ghosting affecting judgment.

[0119] When both sides experience severe failures simultaneously, the system enters the M3 single-channel pass-through mode and selects the healthier channel as the output based on the health score.

[0120] It enters the dominant mode when there is a severe unilateral failure. Among them, severe visible light failure ( The confidence level of visible light overexposure or underexposure events or If the confidence level of a lens contamination or obstruction event corresponding to visible light exceeds a certain threshold, it enters M1 infrared-dominated mode. (Severe infrared malfunction) That is, the confidence level of infrared non-uniformity correction anomaly events or That is, the confidence level of the infrared dead pixel surge event or When the confidence level of the lens contamination or obstruction event corresponding to infrared exceeds the corresponding threshold, it enters the M2 visible light dominant mode.

[0121] S103: If the target working mode is a single-channel pass-through mode, then select an infrared image or a visible light image as the target output object based on the confidence level of the abnormal event.

[0122] S104: If the target working mode is a visible light dominant mode or an infrared dominant mode, then determine the weights of the infrared image and the visible light image respectively according to the target working mode, and fuse the infrared image and the visible light image according to the weights of the infrared image and the visible light image respectively to obtain the target output object.

[0123] If the target operating mode is single-channel straight-through mode, then select the channel with the lower total confidence level. For example, + Less than + + Select a visible light image as the target output object.

[0124] If the target working mode is visible light dominant mode, then the weights corresponding to the infrared image and the visible light image are determined in the visible light dominant mode, and then the infrared image and the visible light image are fused according to their respective weights to obtain the target output object.

[0125] If the target working mode is infrared-dominated mode, then the weights of the infrared image and the visible light image are determined in infrared-dominated mode, and then the infrared image and the visible light image are fused according to their respective weights to obtain the target output object.

[0126] The method for fusing infrared and visible light images based on their respective weights can be as follows:

[0127] . For infrared weights. For visible light weights.

[0128] Infrared Dominant Mode Settings Visible light information is introduced only in the effective visible light area (not overexposed / underexposed and not contaminated / blocked), while infrared light dominates in the remaining areas. This is set in the visible light-dominated mode. It reduces infrared contribution and can be combined with stripe and dead pixel suppression to reduce the risk of infrared noise contaminating the output.

[0129] In some embodiments, it also includes:

[0130] If there are no time synchronization anomalies and neither infrared nor visible light has a serious fault, the normal fusion mode is maintained; in the normal fusion mode, the weights of the infrared image and the visible light image are equal.

[0131] If no significant anomalies are detected, maintain M0, i.e., normal fusion mode. Settings in normal fusion mode. It performs dual-path fusion and enhances necessary details.

[0132] S105: Output the target output object.

[0133] In some embodiments, outputting the target output object includes:

[0134] Within the transition window, the gradient weight is determined based on the transition frame number and the total number of transition frames.

[0135] The transitional output object is obtained by fusing the target output object with the infrared and visible light images in the original working mode based on the gradual weighting.

[0136] The transition output object is output within the transition window, and the target output object is output after the transition window.

[0137] To avoid visual discomfort caused by abrupt mode changes (such as switching from color to black and white instantly) and to achieve a smooth transition during mode switching, this embodiment uses a transition window. For example, a transition window of 10 frames is used.

[0138] Gradient weights can be generated using cosine curves. :

[0139] ;

[0140] .

[0141] in, The current transition frame number ( ), This represents the total number of transition frames (typically 10). For transition weights (dimensionless, value range [0,1]), This is the fusion result of the old mode, i.e., the original mode. This is the fusion result of the new mode, i.e., the target working mode.

[0142] This embodiment uses a non-linear transition, which is smoother and more natural than a linear transition.

[0143] After the transition window, the target output object can be output directly.

[0144] In some embodiments, the target operating mode is maintained for at least a preset number of frames.

[0145] This embodiment adopts the shortest dwell time mechanism, which maintains at least a preset number of frames after entering the new mode, such as at least 30 frames (about 1 second), so as to avoid screen flickering caused by transient interference such as flash.

[0146] In some embodiments, the entry threshold for the visible light overexposure or underexposure event is higher than the exit threshold for the visible light overexposure or underexposure event; the entry threshold for the lens contamination or occlusion event is higher than the exit threshold for the lens contamination or occlusion event; the entry threshold for the infrared non-uniformity correction anomaly event is higher than the exit threshold for the infrared non-uniformity correction anomaly event; the entry threshold for the infrared dead pixel surge event is higher than the exit threshold for the infrared dead pixel surge event; and the entry threshold for the time synchronization anomaly event is higher than the exit threshold for the time synchronization anomaly event.

[0147] This embodiment employs a hysteresis mechanism to prevent repeated jumps at critical points. The threshold for entering an abnormal event is relatively high (e.g., the percentage of pixels with abnormal exposure entering the threshold). The exit threshold is low (e.g.) Each abnormal event can be handled using asymmetric comparison logic with a high entry threshold and a low exit threshold.

[0148] In some embodiments, it also includes:

[0149] Output metadata and display corresponding prompts based on the metadata; the metadata includes operating mode, fault codes and health information.

[0150] The following is an example of the metadata output by the system:

[0151] { "mode": "M3_FALLBACK", "cause": ["SYNC_LOST", "VIS_OVEREXP"], "health": { "vis_score": 0.2, "ir_score": 0.9, "sync_diff": 120}}.

[0152] The platform can then display a "Synchronization error, downgraded to infrared pass-through" message to guide maintenance personnel to check the lines or restart the equipment.

[0153] The output includes metadata containing mode, fault codes, and health information, which facilitates platform-side prompts and remote problem location, reducing operation and maintenance troubleshooting costs.

[0154] The video fault tolerance method provided in the above embodiments of this application is applied to a dual-channel fusion system. The dual-channel fusion system adopts a closed-loop architecture of event detection + state machine decision-making + multi-mode execution. It mainly includes four key steps:

[0155] 1. The event quantification detection step provides mathematical definitions and calculation formulas for 5 types of abnormal events, and outputs the events and their confidence levels.

[0156] 2. The state machine decision-making process defines priority rules and state transition logic, and introduces hysteresis and minimum dwell time to reduce the probability of frequent mode jumps.

[0157] 3. Smooth transition execution: Cosine gradient weighting is used during mode switching to reduce discomfort caused by sudden screen changes.

[0158] 4. The self-diagnostic output section outputs metadata including the current mode and fault codes, which is used for remote operation and maintenance to locate problems.

[0159] The dual-channel fusion system comprises four core modules: an abnormal event detection module, a mode decision state machine, a multi-mode fusion execution module, and a switching transition and metadata module.

[0160] refer to Figure 2 As shown, the abnormal event detection module analyzes the dual light inputs (infrared and visible light) in real time, detecting five key events: overexposure / underexposure, lens contamination / obstruction (detected separately for visible and infrared light), NUC malfunction, sudden increase in dead pixels, and synchronization loss. The output event list is shown. and confidence level (Confidence level is normalized to [0,1], with a larger value indicating a more severe anomaly).

[0161] The mode decision state machine maintains the system's working state (M0-M3) and determines the target mode, i.e. the target working mode, for the next frame based on event priority and hysteresis strategy.

[0162] Multi-mode fusion execution module:

[0163] M0 normal fusion mode. Dual-path weighted and superimposed detail enhancement.

[0164] M1 infrared-dominated mode. Visible light is downweighted, or visible light information is introduced only in locally effective areas.

[0165] M2 visible light dominant mode. Infrared light is downweighted and noise is suppressed to reduce the risk of infrared noise contaminating the output.

[0166] M3 single-channel pass-through mode. Single-channel direct output, bypassing fusion to isolate abnormal input sources.

[0167] Switch between transition and metadata modules, smoothly transition between execution modes, and package and output metadata.

[0168] Combination Figure 3The event detection and state machine decision logic shown in Table 1, using the smooth transition mechanism of mode switching and the cosine weight curve to achieve a smooth image blending, compares the degradation effect under abnormal scenarios. As shown in Table 2, the traditional solution will produce full-screen noise or ghosting when forcibly blended. The solution provided in this application embodiment can output a clear single-channel image after degradation.

[0169] Table 1

[0170]

[0171] Table 2

[0172]

[0173] In summary, the video fault tolerance method provided in this application detects abnormal events and determines their confidence level. Based on the abnormal event and its confidence level, it switches operating modes, automatically switching to a more suitable fusion mode or single-channel pass-through output under sensor failure or extreme interference, thus making the footage more usable under abnormal conditions. Specifically, the single-channel pass-through mode reduces the risk of motion ghosting caused by timing misalignment, avoiding interference with the judgment of personnel and vehicle status.

[0174] This application also provides a video fault-tolerant device, which is described below and can be referred to in conjunction with the method described above. Please refer to... Figure 4 , Figure 4 This is a schematic diagram of a video fault-tolerant device provided in an embodiment of this application, combined with... Figure 4 As shown, the device includes:

[0175] The detection unit 10 is used to detect abnormal events and determine the confidence level of the abnormal events;

[0176] The determining unit 20 is used to determine the target working mode based on the abnormal event and the confidence level of the abnormal event;

[0177] Selection unit 30 is used to select an infrared image or a visible light image as the target output object based on the confidence level of the abnormal event if the target working mode is a single-channel pass-through mode.

[0178] The fusion unit 40 is used to determine the weights of the infrared image and the visible light image according to the target working mode if the target working mode is a visible light dominant mode or an infrared dominant mode, and to fuse the infrared image and the visible light image according to the weights of the infrared image and the visible light image to obtain the target output object.

[0179] Output unit 50 is used to output the target output object.

[0180] Based on the above embodiments, as a specific implementation method, the output unit 50 includes:

[0181] The gradient weight determination sub-unit is used to determine the gradient weight within the transition window based on the transition frame number and the total number of transition frames;

[0182] The fusion subunit is used to fuse the target output object with the fusion results of the infrared image and the visible light image in the original working mode according to the gradient weight, so as to obtain the transition output object;

[0183] An output subunit is used to output the transition output object within the transition window and the target output object after the transition window.

[0184] Based on the above embodiments, as a specific implementation method, the detection unit 10 includes:

[0185] The first detection subunit is used to detect visible light overexposure or underexposure events and determine the confidence level of the visible light overexposure or underexposure events;

[0186] The second detection subunit is used to detect lens contamination or obstruction events and determine the confidence level of the lens contamination or obstruction events.

[0187] The third detection subunit is used to detect infrared non-uniformity correction anomaly events and determine the confidence level of the infrared non-uniformity correction anomaly events.

[0188] The fourth detection subunit is used to detect infrared dead pixel surge events and determine the confidence level of the infrared dead pixel surge events;

[0189] The fifth detection subunit is used to detect time synchronization anomalies and determine the confidence level of the time synchronization anomalies.

[0190] Based on the above embodiments, as a specific implementation method, the determining unit 20 includes:

[0191] The first determining subunit is used to determine the target mode as a single-channel pass-through mode if the abnormal event that occurs includes the time synchronization abnormal event.

[0192] The second determining subunit is used to determine the target working mode as single-channel pass-through mode if the abnormal event does not include the time synchronization abnormal event and the infrared and visible light are both severely faulty based on the confidence level of the abnormal event.

[0193] The third determining subunit is used to determine the target working mode as visible light dominant mode if the abnormal event does not include the time synchronization abnormal event and the infrared serious fault is determined based on the confidence level of the abnormal event.

[0194] The fourth determining subunit is used to determine the target operating mode as infrared-dominated mode if the abnormal event does not include the time synchronization abnormal event and the visible light is determined to be severely faulty based on the confidence level of the abnormal event.

[0195] Based on the above embodiments, as a specific implementation method, the methods for determining whether there is a serious fault in visible light and whether there is a serious fault in infrared light include:

[0196] If the confidence level of a visible light overexposure or underexposure event exceeds the first preset threshold, then there is a serious visible light failure.

[0197] If the confidence level of a lens contamination or obstruction event corresponding to visible light exceeds the second preset threshold, then there is a serious visible light malfunction.

[0198] If the confidence level of an abnormal event in infrared non-uniformity correction exceeds the third preset threshold, then there is a serious infrared malfunction.

[0199] If the confidence level of the infrared dead pixel surge event exceeds the fourth preset threshold, then the infrared sensor is in serious condition.

[0200] If the confidence level of the lens contamination or obstruction event corresponding to the infrared exceeds the fifth preset threshold, then the infrared is considered to be seriously faulty.

[0201] Based on the above embodiments, as a specific implementation method, the target working mode is maintained for at least a preset number of frames.

[0202] Based on the above embodiments, as a specific implementation, the entry threshold for the visible light overexposure or underexposure event is higher than the exit threshold for the visible light overexposure or underexposure event; the entry threshold for the lens contamination or occlusion event is higher than the exit threshold for the lens contamination or occlusion event; the entry threshold for the infrared non-uniformity correction anomaly event is higher than the exit threshold for the infrared non-uniformity correction anomaly event; the entry threshold for the infrared dead pixel surge event is higher than the exit threshold for the infrared dead pixel surge event; and the entry threshold for the time synchronization anomaly event is higher than the exit threshold for the time synchronization anomaly event.

[0203] This application also provides an electronic device, referenced... Figure 5 As shown, the device includes a memory 1 and a processor 2.

[0204] Memory 1 is used to store computer programs;

[0205] Processor 2 is used to execute computer programs to perform the following steps:

[0206] Detect abnormal events and determine the confidence level of the abnormal events;

[0207] The target operating mode is determined based on the abnormal events and their confidence levels.

[0208] If the target working mode is a single-channel pass-through mode, then an infrared image or a visible light image is selected as the target output object based on the confidence level of the abnormal event.

[0209] If the target working mode is a visible light dominant mode or an infrared dominant mode, then the weights corresponding to the infrared image and the visible light image are determined according to the target working mode, and the infrared image and the visible light image are fused according to the weights corresponding to the infrared image and the visible light image to obtain the target output object.

[0210] Output the target output object.

[0211] For a description of the equipment provided in this application, please refer to the above method embodiments; further details will not be provided here.

[0212] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, can perform the following steps:

[0213] Detect abnormal events and determine the confidence level of the abnormal events;

[0214] The target operating mode is determined based on the abnormal events and their confidence levels.

[0215] If the target working mode is a single-channel pass-through mode, then an infrared image or a visible light image is selected as the target output object based on the confidence level of the abnormal event.

[0216] If the target working mode is a visible light dominant mode or an infrared dominant mode, then the weights corresponding to the infrared image and the visible light image are determined according to the target working mode, and the infrared image and the visible light image are fused according to the weights corresponding to the infrared image and the visible light image to obtain the target output object.

[0217] Output the target output object.

[0218] The computer-readable storage medium may include various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0219] For a description of the computer-readable storage medium provided in this application, please refer to the above method embodiments; further details will not be repeated here.

[0220] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatuses, devices, and computer-readable storage media disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple; relevant details can be found in the method section.

[0221] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0222] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0223] The video fault tolerance method, apparatus, device, and storage medium provided in this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are only for the purpose of helping to understand the method and its core ideas. It should be noted that those skilled in the art can make several improvements and modifications to this application without departing from the principles of this application, and these improvements and modifications also fall within the protection scope of this application.

Claims

1. A video fault tolerance method, characterized in that, include: Detect abnormal events and determine the confidence level of the abnormal events; The target operating mode is determined based on the abnormal events and their confidence levels. If the target working mode is a single-channel pass-through mode, then an infrared image or a visible light image is selected as the target output object based on the confidence level of the abnormal event. If the target working mode is a visible light dominant mode or an infrared dominant mode, then the weights corresponding to the infrared image and the visible light image are determined according to the target working mode, and the infrared image and the visible light image are fused according to the weights corresponding to the infrared image and the visible light image to obtain the target output object. Output the target output object.

2. The video fault tolerance method according to claim 1, characterized in that, The target output object includes: Within the transition window, the gradient weight is determined based on the transition frame number and the total number of transition frames. The transitional output object is obtained by fusing the target output object with the infrared and visible light images in the original working mode based on the gradual weighting. The transition output object is output within the transition window, and the target output object is output after the transition window.

3. The video fault tolerance method according to claim 1, characterized in that, Detecting anomalous events and determining the confidence level of said anomalous events includes: Detect visible light overexposure or underexposure events and determine the confidence level of the visible light overexposure or underexposure events; Detect lens contamination or obstruction events and determine the confidence level of said lens contamination or obstruction events; Detect infrared non-uniformity correction anomaly events and determine the confidence level of the infrared non-uniformity correction anomaly events; Detect infrared dead pixel surge events and determine the confidence level of the infrared dead pixel surge events; Detect time synchronization anomalies and determine the confidence level of the time synchronization anomalies.

4. The video fault tolerance method according to claim 3, characterized in that, Based on the abnormal events and their confidence levels, the target operating mode is determined as follows: If the abnormal event that occurs includes the time synchronization abnormal event, then the target mode is determined to be a single-channel pass-through mode; If the abnormal event does not include the time synchronization abnormal event and the confidence level of the abnormal event determines that both infrared and visible light are severely faulty, then the target working mode is determined to be single-channel pass-through mode. If the abnormal event does not include the time synchronization abnormal event and the infrared serious fault is determined based on the confidence level of the abnormal event, then the target working mode is determined to be the visible light dominant mode. If the abnormal event does not include the time synchronization abnormal event and the visible light is determined to be severely faulty based on the confidence level of the abnormal event, then the target operating mode is determined to be the infrared-dominated mode.

5. The video fault tolerance method according to claim 4, characterized in that, The methods for determining whether there is a serious fault in visible light or infrared light include: If the confidence level of a visible light overexposure or underexposure event exceeds the first preset threshold, then there is a serious visible light failure. If the confidence level of a lens contamination or obstruction event corresponding to visible light exceeds the second preset threshold, then there is a serious visible light malfunction. If the confidence level of an abnormal event in infrared non-uniformity correction exceeds the third preset threshold, then there is a serious infrared malfunction. If the confidence level of the infrared dead pixel surge event exceeds the fourth preset threshold, then the infrared sensor is in serious condition. If the confidence level of the lens contamination or obstruction event corresponding to the infrared exceeds the fifth preset threshold, then the infrared is considered to be seriously faulty.

6. The video fault tolerance method according to claim 1, characterized in that, The target working mode is maintained for at least a preset number of frames.

7. The video fault tolerance method according to claim 3, characterized in that, The entry threshold for the visible light overexposure or underexposure event is higher than the exit threshold for the visible light overexposure or underexposure event; the entry threshold for the lens contamination or occlusion event is higher than the exit threshold for the lens contamination or occlusion event; the entry threshold for the infrared non-uniformity correction anomaly event is higher than the exit threshold for the infrared non-uniformity correction anomaly event; the entry threshold for the infrared dead pixel surge event is higher than the exit threshold for the infrared dead pixel surge event; and the entry threshold for the time synchronization anomaly event is higher than the exit threshold for the time synchronization anomaly event.

8. A video fault-tolerant device, characterized in that, include: A detection unit is used to detect abnormal events and determine the confidence level of the abnormal events; The determining unit is used to determine the target operating mode based on the abnormal event and the confidence level of the abnormal event; The selection unit is used to select either an infrared image or a visible light image as the target output object based on the confidence level of the abnormal event if the target working mode is a single-channel pass-through mode. The fusion unit is used to determine the weights of the infrared image and the visible light image according to the target working mode if the target working mode is a visible light dominant mode or an infrared dominant mode, and to fuse the infrared image and the visible light image according to the weights of the infrared image and the visible light image to obtain the target output object. The output unit is used to output the target output object.

9. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, configured to implement the steps of the video fault tolerance method as described in any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the video fault tolerance method as described in any one of claims 1 to 7.

Images