Polar region and polar night cloud layer detection method and device based on event camera and deep learning cooperation

Abstract

The invention discloses a polar region and polar night cloud layer detection method and device based on event camera and deep learning cooperation, and the method comprises the steps: collecting an asynchronous event stream outputted by an event camera and an infrared image sequence outputted by an infrared camera in a polar night environment; performing polarity perception voxelization processing on the asynchronous event stream to generate an event voxel tensor; performing time resolution enhancement on the infrared image sequence based on the event voxel tensor to generate an enhanced infrared sequence; inputting the event voxel tensor and the enhanced infrared sequence into an event-infrared fusion network for feature extraction and cross-modal fusion to obtain a preliminary cloud probability graph; acquiring wind field information in the numerical weather forecast data, constructing a cloud advection motion kernel, and performing motion constraint on the initial cloud probability graph based on the cloud advection motion kernel to generate a cloud probability graph after motion constraint; and determining an extreme night cloud layer detection result based on the cloud probability graph after motion constraint. According to the invention, the accuracy and reliability of cloud layer detection under the extremely night condition are obviously improved.

Description

Technical Field

[0001] This application relates to the field of polar environment monitoring technology, and in particular to a method and apparatus for detecting polar night clouds based on the collaboration of event cameras and deep learning. Background Technology

[0002] During the polar night, lighting conditions in polar regions are extremely poor, making it difficult for traditional optical cameras to function effectively. While infrared cameras can image in complete darkness, their limited temporal resolution makes it difficult to capture rapidly changing cloud movements. Event cameras, as a novel type of sensor, can asynchronously record pixel-level brightness changes with microsecond-level precision, offering advantages such as high temporal resolution, high dynamic range, and low power consumption. However, their output data is an asynchronous event stream, making it difficult to directly use in traditional image processing algorithms. Furthermore, the extremely low temperatures in polar environments affect the trigger threshold of event cameras, leading to a decline in data quality. Existing cloud detection methods are mostly based on visible light or single infrared data, which suffer from low detection accuracy and insensitivity to dynamic cloud layers in polar night environments. Summary of the Invention

[0003] The main purpose of this application is to provide a polar cloud detection method and device based on the collaboration of event camera and deep learning, which aims to solve the technical problems of low cloud detection accuracy and insufficient dynamic capture capability in the polar night environment of the prior art.

[0004] To achieve the above objectives, this application proposes a polar cloud detection method based on the collaboration of event cameras and deep learning. The polar cloud detection method based on the collaboration of event cameras and deep learning includes: In polar night environments, the asynchronous event stream output by the event camera and the infrared image sequence output by the infrared camera are acquired; The asynchronous event stream is subjected to polarity-aware voxelization to generate an event voxel tensor; Based on the event voxel tensor, the infrared image sequence is enhanced with temporal resolution to generate an enhanced infrared sequence; The event voxel tensor and the enhanced infrared sequence are input into the event-infrared fusion network for feature extraction and cross-modal fusion to obtain a preliminary cloud probability map. The event-infrared fusion network includes an event branch, an infrared branch, a cross-modal gated attention fusion module, and a lightweight decoder. The event branch uses lightweight spatiotemporal separation convolution to extract dynamic edge features, and the infrared branch uses a channel-pruned residual network variant to extract thermal structure features. Obtain wind field information from numerical weather forecast data, construct a cloud advection motion kernel based on the wind field information, and apply motion constraints to the preliminary cloud probability map based on the cloud advection motion kernel to generate a motion-constrained cloud probability map. The detection results of polar night clouds are determined based on the cloud probability map after the motion constraints.

[0005] In one embodiment, the step of acquiring the asynchronous event stream output by the event camera and the infrared image sequence output by the infrared camera in a polar night environment includes: Time synchronization was performed on the event camera and infrared camera deployed at the polar observation station to obtain the synchronized event camera and synchronized infrared camera. A temperature-trigger threshold compensation lookup table is established, and the synchronized event camera is calibrated for low-temperature threshold drift based on the temperature-trigger threshold compensation lookup table to obtain the calibrated event camera. The asynchronous event stream output by the calibrated event camera is acquired, and the asynchronous event stream is time-sliced ​​with a preset time window to obtain the total number of events in each time window; The total number of events is compared with a preset sparsity threshold and a preset saturation threshold to obtain the comparison results, and the density state of the asynchronous event stream is determined based on the comparison results. The infrared camera's acquisition parameters are adaptively adjusted based on the density state using a reinforcement learning strategy, and the infrared image sequence output by the adjusted infrared camera is acquired. The acquisition parameters include exposure time and gain coefficient.

[0006] In one embodiment, the polarity-aware voxelization process performed on the asynchronous event stream to generate event voxel tensors includes: The voxel aggregation time window is determined based on the frame interval of the infrared image sequence and the density state of the asynchronous event stream; Within the voxel aggregation time window, events in the asynchronous event stream are separated into positive event sets and negative event sets according to their polarity; For the positive event set and the negative event set respectively, a non-uniform time binning strategy is used to perform discretization mapping in the time dimension to generate discrete spatiotemporal grids for the positive and negative event sets. The event counts are accumulated in the discrete spatiotemporal grids of the positive and negative event sets respectively to generate event voxel tensors.

[0007] In one embodiment, the step of performing temporal resolution enhancement on the infrared image sequence based on the event voxel tensor to generate an enhanced infrared sequence includes: The rate of change of event density over time is calculated based on the event voxel tensor. Local density extreme points are detected based on the rate of change of event density over time, and the time points corresponding to the local density extreme points are determined as characteristic moments of significant changes in cloud structure. Select two original infrared images adjacent to the characteristic time from the infrared image sequence; Based on the two original infrared images, a phase-based motion estimation algorithm is used to estimate the bidirectional optical flow field. Based on the bidirectional optical flow field, a forward twisting and hole filling algorithm is used to generate the virtual infrared frame corresponding to the characteristic moment; The virtual infrared frame is inserted into the infrared image sequence to generate an enhanced infrared sequence.

[0008] In one embodiment, the step of inputting the event voxel tensor and the enhanced infrared sequence into an event-infrared fusion network for feature extraction and cross-modal fusion to obtain a preliminary cloud probability map includes: The event voxel tensor is input into the event branch, and the dynamic edge features of the event are extracted through lightweight spatiotemporal separation convolution. The enhanced infrared sequence is input into the infrared branch, and multi-scale thermal structure features of the infrared image are extracted through a residual network variant with channel pruning. The dynamic edge features and the thermal structure features of the highest layer are input into the cross-modal gated attention module to calculate the event-guided infrared attention weight and the infrared-guided event attention weight. Based on the infrared attention weight and the event attention weight, the dynamic edge features and the thermal structure features of the highest layer are weighted and fused to obtain complementary enhanced features; The complementary enhancement features and the low-level thermal structure features are fed into a lightweight decoder via skip connections to output a preliminary cloud probability map.

[0009] In one embodiment, the steps of acquiring wind field information from numerical weather prediction data, constructing a cloud advection kernel based on the wind field information, and applying motion constraints to the preliminary cloud probability map based on the cloud advection kernel to generate a motion-constrained cloud probability map include: Obtain wind field information from numerical weather forecast data, and extract the horizontal wind vector within a specified pressure layer range from the wind field information; The horizontal wind vector field is mapped onto the image plane through spherical projection and perspective transformation to obtain the pixel displacement vector; A cloud advection motion kernel is constructed using the pixel displacement vector as the core. The cloud advection kernel is spatiotemporally convolved with the preliminary cloud probability map to generate a probability map with enhanced motion consistency. The motion consistency enhancement map is used as a guiding signal to feed back to the lightweight decoder in the event-infrared fusion network, and the cloud probability map after motion constraints is output.

[0010] In one embodiment, before acquiring wind field information from numerical weather forecast data and extracting horizontal wind vectors within a specified pressure layer range from the wind field information, the method further includes: Numerical weather forecast data is acquired, and data quality is checked on the numerical weather forecast data to obtain data missing information and data quality scores. When the data loss occurs in the numerical weather forecast data or the data quality score is lower than a preset score threshold, the backup mode is triggered. In the backup mode, a dense event optical flow field is obtained by calculating the event optical flow estimation algorithm based on the event voxel tensor. An endogenous motion core is constructed using the dense event optical flow field as its core. The endogenous motion kernel is spatiotemporally convolved with the preliminary cloud probability map to generate a cloud probability map based on endogenous motion constraints. The cloud probability map based on endogenous motion constraints is used as a guiding signal to be fed back to the lightweight decoder in the event-infrared fusion network, and the motion-constrained cloud probability map is output.

[0011] In one embodiment, determining the polar night cloud detection result based on the cloud probability map after motion constraints includes: Calculate the Shannon entropy of the cloud probability map after the motion constraints in the local neighborhood, and use the Shannon entropy as the classification uncertainty score; The motion consistency score of each pixel is determined based on the motion information implied in the cloud advection motion kernel and the cloud probability map after motion constraints. A cloud detection confidence map is generated based on the classification uncertainty score and the motion consistency score. The cloud probability map after motion constraints is binarized to obtain the initial cloud mask; The initial cloud mask is filtered based on the cloud detection confidence map to obtain a filtered cloud mask. Morphological operations are performed on the filtered cloud mask to obtain the polar night cloud detection results.

[0012] In one embodiment, generating a cloud detection confidence map based on the classification uncertainty score and the motion consistency score includes: Based on the classification uncertainty score, a spatial uncertainty distribution matrix is ​​constructed, and the spatial uncertainty distribution matrix is ​​decomposed into a multi-scale Gaussian pyramid to obtain a spatial uncertainty feature sequence containing multiple scales. The cloud region boundary is extracted from the cloud probability map after motion constraints, and the derivative of the boundary pixel along the motion direction is calculated based on the motion consistency score to generate a multi-scale motion consistency gradient map. The outputs of each layer of the multi-scale Gaussian pyramid are fused with the gradient magnitudes of the corresponding scales to generate motion-enhanced uncertainty features. The spatial uncertainty feature sequence and the multi-scale motion consistency gradient map are fused element-wise at the corresponding scales to generate a motion-enhanced multi-scale uncertainty feature sequence. Based on the neighborhood structure and direction defined by the cloud advection motion kernel, the motion-enhanced multi-scale uncertainty feature sequence is subjected to top-down anisotropic diffusion filtering to obtain a smooth multi-scale confidence feature sequence with consistent motion trajectories. The smooth, consistent multi-scale confidence feature sequences are upsampled and concatenated, and then input into a lightweight convolutional network for nonlinear mapping and aggregation to generate a cloud detection confidence map.

[0013] Furthermore, to achieve the above objectives, this application also proposes a polar cloud detection device based on the collaboration of event cameras and deep learning. The polar cloud detection device based on the collaboration of event cameras and deep learning includes: The acquisition module is used to acquire asynchronous event streams output by the event camera and infrared image sequences output by the infrared camera in polar night environments. The processing module is used to perform polarity-aware voxelization processing on the asynchronous event stream to generate event voxel tensors; An enhancement module is used to perform temporal resolution enhancement on the infrared image sequence based on the event voxel tensor to generate an enhanced infrared sequence; The fusion module is used to input the event voxel tensor and the enhanced infrared sequence into the event-infrared fusion network for feature extraction and cross-modal fusion to obtain a preliminary cloud probability map. The event-infrared fusion network includes an event branch, an infrared branch, a cross-modal gated attention fusion module, and a lightweight decoder. The event branch uses lightweight spatiotemporal separation convolution to extract dynamic edge features, and the infrared branch uses a channel-pruned residual network variant to extract thermal structure features. The constraint module is used to acquire wind field information from numerical weather forecast data, construct a cloud advection motion kernel based on the wind field information, and apply motion constraints to the preliminary cloud probability map based on the cloud advection motion kernel to generate a motion-constrained cloud probability map. The determination module is used to determine the detection result of polar night clouds based on the cloud probability map after the motion constraints.

[0014] This application proposes one or more technical solutions, which, in a polar night environment, acquire asynchronous event streams output by an event camera and infrared image sequences output by an infrared camera; perform polarity-aware voxelization processing on the asynchronous event stream to generate event voxel tensors; perform temporal resolution enhancement on the infrared image sequences based on the event voxel tensors to generate enhanced infrared sequences; input the event voxel tensors and the enhanced infrared sequences into an event-infrared fusion network for feature extraction and cross-modal fusion to obtain a preliminary cloud probability map, wherein the event-infrared fusion network includes an event branch, an infrared branch, a cross-modal gated attention fusion module, and a lightweight decoder; the event branch uses lightweight spatiotemporal separation convolution to extract dynamic edge features, and the infrared branch uses a channel-pruned residual network variant to extract thermal structure features; acquire wind field information from numerical weather prediction data, construct a cloud advection motion kernel based on the wind field information, and apply motion constraints to the preliminary cloud probability map based on the cloud advection motion kernel to generate a motion-constrained cloud probability map; and determine the polar night cloud detection results based on the motion-constrained cloud probability map. By employing the above methods, and through the collaborative work of event cameras and infrared cameras, combined with deep learning and numerical weather forecast data, high-precision, high-temporal-resolution cloud detection was achieved in polar night environments. Attached Figure Description

[0015] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0016] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a flowchart illustrating an embodiment of the polar cloud detection method based on the collaboration of event camera and deep learning in this application. Figure 2 This is a schematic diagram of the module structure of the polar cloud detection device based on the collaboration of event camera and deep learning in an embodiment of this application.

[0018] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0019] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0020] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0021] This application provides a solution that achieves high-precision, high-temporal-resolution cloud detection in polar night environments by combining event cameras and infrared cameras with deep learning and numerical weather forecast data.

[0022] It should be noted that the executing entity in this embodiment can be a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, or mobile phone, or an electronic device capable of performing the above functions, such as a polar cloud layer detection device based on event camera and deep learning collaboration. The following description uses a polar cloud layer detection device based on event camera and deep learning collaboration as an example to illustrate this embodiment and the subsequent embodiments.

[0023] Based on this, embodiments of this application provide a polar cloud detection method based on the collaboration of event cameras and deep learning, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the polar cloud detection method based on the collaboration of event camera and deep learning in this application.

[0024] In this embodiment, the polar cloud detection method based on the collaboration of event camera and deep learning includes steps S10~S60: Step S10: In a polar night environment, acquire the asynchronous event stream output by the event camera and the infrared image sequence output by the infrared camera.

[0025] It's important to note that the polar night environment is unique in its extremely low light conditions, making it difficult for traditional optical cameras to function effectively. Event cameras, as a novel type of visual sensor, do not rely on the frame rate of traditional cameras but are sensitive to changes in light. They can output dynamic information about the scene as an asynchronous event stream, working effectively even in extremely low light conditions. Infrared cameras, on the other hand, capture the infrared radiation emitted by objects, generating infrared image sequences that reflect the object's thermal distribution. In the polar night environment, they can provide information on the thermal structure of clouds. Event cameras are sensitive to subtle dynamics, while infrared cameras provide thermal radiation information; the two complement each other to address the problem of the lack of visible light in the polar night.

[0026] It is understandable that the asynchronous event stream is a sparse data stream output by the event camera, generated by starlight, airglow, or faint artificial light sources scattered by atmospheric clouds. Each event contains location, timestamp, and polarity information. The infrared image sequence output by the infrared camera is a low frame rate long-wave infrared image sequence, with a frame rate not exceeding 5Hz.

[0027] In practical implementation, during the acquisition process, it is necessary to ensure the synchronization of the event camera and the infrared camera. This can be achieved through hardware synchronization signals or software timestamps, ensuring precise alignment of the acquisition times of the two cameras and guaranteeing the consistency of the asynchronous event stream and the infrared image sequence in the time dimension. In one feasible implementation, step S10 may include: time synchronization of an event camera and an infrared camera deployed at a polar observation station to obtain synchronized event cameras and synchronized infrared cameras; establishing a temperature-trigger threshold compensation lookup table, and performing low-temperature threshold drift calibration on the synchronized event camera based on the temperature-trigger threshold compensation lookup table to obtain calibrated event cameras; acquiring asynchronous event streams output by the calibrated event cameras, and performing time slicing on the asynchronous event streams with preset time windows to obtain the total number of events in each time window; comparing the total number of events with a preset sparse threshold and a preset saturation threshold respectively to obtain comparison results, and determining the density state of the asynchronous event stream based on the comparison results; adaptively adjusting the acquisition parameters of the infrared camera according to the density state based on a reinforcement learning strategy, and acquiring the infrared image sequence output by the infrared camera with adjusted parameters, wherein the acquisition parameters include exposure time and gain coefficient.

[0028] It should be noted that the polar observation station can be an Arctic observation station or an Antarctic observation station. This embodiment does not make specific restrictions on this. This embodiment takes the event camera and infrared camera deployed at an Arctic observation station as an example for illustration.

[0029] Understandably, the event camera and the infrared camera can be synchronized through a precise time protocol, which ensures that the time synchronization accuracy of the two cameras reaches the microsecond level, effectively avoiding image misalignment or information distortion caused by time asynchrony.

[0030] The event camera is based on the principle of a dynamic vision sensor, triggering events by detecting changes in pixel brightness. In the extremely low temperatures of the polar night, its typical temperature range is... 50°C to At 80°C, the leakage current of the CMOS pixel circuitry decreases significantly, but the threshold voltage also drifts, leading to uneven dark count distribution and abnormally high / low sensitivity in some pixels. Dark counts refer to false events generated when there are no real changes in illumination. Therefore, pixel-level low-temperature bias voltage calibration is performed on the event camera to compensate for this. 50°C to Dark count drift at 80°C. Low-temperature threshold drift calibration of the synchronized event camera can be performed by establishing a temperature-trigger threshold compensation lookup table and adjusting the compensation threshold on-site based on the real-time temperature. The temperature-trigger threshold compensation lookup table can be established through laboratory simulation. In an environment ranging from 50℃ to 0℃, the event camera trigger threshold drift was recorded at different temperatures to form a mapping table.

[0031] It's worth noting that the preset time window can be 100ms, meaning the event stream is sliced ​​every 100ms, and the total number of events N_e is counted. The sparsity threshold N_low is set to 500, and the saturation threshold N_high is set to 50000. If N_e < N_low, the asynchronous event stream is considered sparse, requiring a reduction in the infrared camera exposure time to decrease thermal noise. If N_e > N_high, the asynchronous event stream is considered saturated, requiring an increase in the infrared camera gain to enhance weak signals. If N_low ≤ N_e ≤ N_high, it's considered a normal event stream, and the current acquisition parameters are maintained. By dynamically adjusting the infrared camera parameters, collaborative optimization of the asynchronous event stream and the infrared image sequence can be achieved, avoiding subsequent processing errors caused by excessively dense or sparse event streams.

[0032] In practical implementation, the acquisition parameters of the infrared camera can be adaptively adjusted based on the density state of the asynchronous event stream using a reinforcement learning strategy. The reinforcement learning strategy constructs a state-action-reward feedback mechanism, defining the combination of the asynchronous event stream density state (sparse, normal, saturated) and the current acquisition parameters in the state space. The current acquisition parameters can be the signal-to-noise ratio of historical infrared images and the ambient temperature. The action space includes the step size for increasing or decreasing the exposure time and the adjustment range of the gain coefficient. The reward function is designed as a weighted sum of the increase in the infrared image signal-to-noise ratio and the stability of the ambient temperature.

[0033] The reinforcement learning strategy employs an offline pre-training + online fine-tuning approach, deployed in the microcontroller of an edge device, with a decision cycle of 1–2 seconds. By introducing a reinforcement learning strategy to adjust the acquisition parameters of the infrared camera, closed-loop adaptive adaptation between infrared imaging parameters and scene dynamics can be achieved, ensuring the acquisition of high-contrast, low-noise thermal images even under extremely low light conditions, thus improving the robustness of subsequent fusion detection.

[0034] Step S20: Perform polarity-aware voxelization on the asynchronous event stream to generate event voxel tensors.

[0035] It should be noted that polarity-aware voxelization is a voxelization method that considers the positive and negative polarities of events, and can retain more motion information. The event voxel tensor is a four-dimensional data structure. Its three spatial dimensions correspond to the imaging plane of the event camera sensor, while the time dimension records the time information of the event occurrence. At the same time, the polarity (positive or negative) of the event is encoded through different channels.

[0036] In one feasible implementation, step S20 may include: determining a voxel aggregation time window based on the frame interval of the infrared image sequence and the density state of the asynchronous event stream; within the voxel aggregation time window, separating the events in the asynchronous event stream into a positive event set and a negative event set according to polarity; performing discretization mapping in the time dimension on the positive event set and the negative event set respectively using a non-uniform time binning strategy to generate discrete spatiotemporal grids for the positive and negative event sets; and accumulating event counts in the discrete spatiotemporal grids of the positive and negative event sets respectively to generate event voxel tensors.

[0037] It should be noted that the frame interval of an infrared image sequence refers to the time interval between any current infrared frame and the previous frame in the infrared image sequence. Combined with the event sparsity state, the voxel aggregation time window τ is dynamically set. The higher the event flow density, the shorter the aggregation time window is used to preserve high-frequency dynamic details.

[0038] Understandably, a base time window T_base is set, whose value is positively correlated with the infrared image frame interval ΔT_ir. If the density of the asynchronous event stream is sparse, a scaling factor α > 1 is set; if the density of the asynchronous event stream is saturated, a scaling factor 0 < β < 1 is set; and if it is in a normal state, the scaling factor is 1. The final time window is then calculated. , where γ is a scaling factor α, β or 1 determined according to the density state.

[0039] It is worth noting that within the aggregation time window of length T_window, the events in the asynchronous event stream are separated into a positive event set E_positive and a negative event set E_negative according to their polarity. For each event set, a non-uniform time binning strategy is used to perform discretization mapping in the time dimension based on the cumulative distribution of its events over time. In this embodiment, the non-uniform time binning strategy can be the critical event density adaptive binning method.

[0040] Specifically, the process involves calculating the cumulative event count of positive and negative event sets within the aggregation time window over time. The time axis of this curve is divided into N initial equally spaced sub-intervals, representing the discrete spatiotemporal grids of the positive and negative event sets. Event counts within each sub-interval are calculated, and adjacent sub-intervals with event counts below a preset threshold are merged. The boundaries of the resulting M intervals (M≤N) are used as the boundaries for non-uniform time binning. Within each non-uniform time bin, the event counts of the positive and negative event sets are accumulated to obtain the number of positive and negative events within each bin. The three spatial dimensions (corresponding to the imaging plane of the event camera sensor) are combined with the temporal dimension (non-uniform time binning), and the number of positive and negative events is encoded through different channels to generate an event voxel tensor. This tensor not only preserves the spatial location information of the events but also retains the temporal dynamics of the events through the non-uniform binning strategy in the temporal dimension. Furthermore, it utilizes different channels to encode the positive and negative polarities of the events, providing a rich information foundation for subsequent feature extraction and fusion.

[0041] Step S30: Perform temporal resolution enhancement on the infrared image sequence based on the event voxel tensor to generate an enhanced infrared sequence.

[0042] It should be noted that temporal resolution enhancement utilizes the high temporal characteristics of event data to compensate for the insufficient temporal resolution of infrared images.

[0043] Understandably, by analyzing the changes in event voxel tensors over time, extreme points in event density can be detected, corresponding to moments of significant changes in cloud structure. Between adjacent infrared frames, virtual infrared frames corresponding to these characteristic moments are generated using forward twisting and hole-filling algorithms based on a bidirectional optical flow field, thereby improving the temporal resolution of the infrared sequence several times.

[0044] In one feasible implementation, step S30 may include: calculating the rate of change of event density over time based on the event voxel tensor; detecting local density extrema based on the rate of change of event density over time, and determining the time point corresponding to the local density extrema as the characteristic moment of significant cloud structure change; selecting two original infrared images adjacent to the characteristic moment from the infrared image sequence; estimating the bidirectional optical flow field based on the two original infrared images using a phase-based motion estimation algorithm; generating a virtual infrared frame corresponding to the characteristic moment based on the bidirectional optical flow field through a forward twisting and hole-filling algorithm; and inserting the virtual infrared frame into the infrared image sequence to generate an enhanced infrared sequence.

[0045] It should be noted that the rate of change of event density is obtained by differentiating the event voxel tensor along the time axis, as shown in the following formula:

[0046] Where H and W are the height and width of the imaging plane of the event camera sensor. , Let be the values ​​of the positive and negative event voxel tensors at (x,y,t), respectively, and R(t) represent the rate of change of event density at time t.

[0047] A local maximum search algorithm is used to identify local maxima in the event density change rate as characteristic moments of significant changes in cloud structure.

[0048] In the specific implementation, two original infrared images adjacent to the feature time are selected from the infrared image sequence, denoted as I_prev and I_next. A phase-based motion estimation algorithm, such as the Lucas-Kanade method, is used to calculate the bidirectional optical flow fields F_prev→next and F_next→prev between the two frames. The bidirectional optical flow field can more accurately describe the motion direction of pixels and reduce the error in unidirectional optical flow estimation.

[0049] Based on the bidirectional optical flow field, I_prev and I_next are twisted to the feature time step by a forward twisting algorithm to generate two candidate virtual infrared frames. Since holes (i.e., regions without corresponding pixels) may appear during the twisting process, a hole filling algorithm is required, such as a neighborhood interpolation-based or deep learning method, to reasonably fill the hole regions and generate a complete virtual infrared frame I_virtual.

[0050] The generated virtual infrared frame I_virtual is inserted into the original infrared image sequence at the position corresponding to the feature time, thereby improving the temporal resolution of the infrared sequence several times. For example, if the frame rate of the original infrared sequence is 1Hz, inserting a virtual frame can increase it to 5Hz or higher, significantly improving the ability to capture dynamic changes in clouds.

[0051] Step S40: Input the event voxel tensor and the enhanced infrared sequence into the event-infrared fusion network for feature extraction and cross-modal fusion to obtain a preliminary cloud probability map. The event-infrared fusion network includes an event branch, an infrared branch, a cross-modal gated attention fusion module, and a lightweight decoder. The event branch uses lightweight spatiotemporal separation convolution to extract dynamic edge features, and the infrared branch uses a channel-pruned residual network variant to extract thermal structure features.

[0052] It should be noted that the event-infrared fusion network employs a dual-branch architecture to process event voxel tensors and enhanced infrared sequences separately, achieving feature complementarity through cross-modal interaction. The event branch uses lightweight spatiotemporal separable convolutions to extract dynamic edge features, reducing computational complexity and improving processing efficiency. The infrared branch uses a channel-pruned residual network variant to extract thermal structure features, reducing the number of model parameters and adapting to the computational limitations of polar equipment. The cross-modal gated attention fusion module achieves deep fusion of event and infrared features through dynamic weight allocation, enhancing the ability to represent complex cloud structures. The lightweight decoder uses transposed convolutions and skip connections to progressively upsample the fused features to the original resolution, while reducing the number of parameters through depthwise separable convolutions, ultimately outputting a preliminary cloud probability map.

[0053] It is worth noting that this network architecture, while maintaining high accuracy, significantly reduces model complexity, enabling its deployment in resource-constrained polar observation equipment.

[0054] In one feasible implementation, step S40 may include: inputting the event voxel tensor into the event branch and extracting dynamic edge features of the event through lightweight spatiotemporal separation convolution; inputting the enhanced infrared sequence into the infrared branch and extracting multi-scale thermal structure features of the infrared image through a channel-pruned residual network variant; inputting the dynamic edge features and the thermal structure features of the highest layer into a cross-modal gated attention module to calculate event-guided infrared attention weights and infrared-guided event attention weights; performing weighted fusion of the dynamic edge features and the thermal structure features of the highest layer based on the infrared attention weights and the event attention weights to obtain complementary enhanced features; and sending the complementary enhanced features and the thermal structure features of the lower layer into a lightweight decoder through skip connections to output a preliminary cloud probability map.

[0055] It should be noted that the event branch is based on spatiotemporal separation convolution, which decomposes the four-dimensional event voxel tensor into a serial structure of spatial convolution and temporal convolution. First, local spatial features are extracted through 3×3 spatial convolution, and then 1×1 convolution is performed along the temporal dimension to capture temporal correlation. This design significantly reduces computational complexity while preserving the edge details of cloud motion.

[0056] The infrared branch is based on the improved ResNet-18 architecture. It removes redundant feature channels through channel pruning, reducing the number of parameters by 30% while maintaining feature extraction capabilities. Its output multi-scale thermal structure features contain different levels of structural information from shallow to deep layers. The highest layer features are rich in semantic information, while the lower layer features retain more spatial details.

[0057] The cross-modal gated attention module achieves feature interaction through a dual attention mechanism: the event-guided infrared attention weights generate a spatial attention map from dynamic edge features, highlighting regions in the infrared image related to event motion; the infrared-guided event attention weights adjust the temporal weights of event features based on thermal structure features, suppressing background noise interference. This bidirectional attention mechanism enables the network to focus on the most informative regions, enhancing feature complementarity.

[0058] In its implementation, the event-guided infrared attention weights are calculated by the spatial correlation between dynamic edge features and infrared features to generate a spatial attention map, which assigns higher weights to regions in the infrared image related to event motion. The infrared-guided event attention weights are calculated by the temporal correlation between thermal structure features and event features to generate a temporal attention map, which assigns higher weights to temporal segments of event features related to infrared thermal structures. Through this bidirectional attention mechanism, the network can dynamically adjust the weights of feature fusion, achieving deep fusion of events and infrared features. Finally, complementary enhancement features and low-level thermal structure features are fed into a lightweight decoder via skip connections. The decoder uses transposed convolutions to progressively upsample the fused features to the original resolution, while simultaneously reducing the number of parameters through depthwise separable convolutions. The final output is a preliminary cloud probability map, which accurately reflects the spatial distribution probability of clouds.

[0059] Step S50: Obtain wind field information from numerical weather forecast data, construct a cloud advection motion kernel based on the wind field information, and apply motion constraints to the preliminary cloud probability map based on the cloud advection motion kernel to generate a motion-constrained cloud probability map.

[0060] It should be noted that the cloud advection motion kernel, which is the motion propagation kernel that characterizes the cloud advection motion, is a motion model built based on wind field information and is used to constrain the cloud motion trajectory. The wind field information in numerical weather prediction data refers to the horizontal wind vector field (u,v) within a specified pressure layer range extracted from numerical weather prediction data such as ECMWF or GDAS. The specified pressure layer range can be from 850 hPa to 300 hPa.

[0061] Understandably, the cloud advection kernel is presented as a two-dimensional Gaussian filter, with its center position determined by the projection of the wind field vector (u,v) onto the image coordinate system. The standard deviation σ is positively correlated with the wind speed to simulate the diffusion effect of cloud motion.

[0062] In the specific implementation, the horizontal wind vector field of a specified pressure layer is extracted from numerical weather prediction data, and mapped onto the image plane through spherical projection and perspective transformation to construct a cloud advection motion kernel. This motion kernel is then spatiotemporally convolved with the preliminary cloud probability map to enhance motion consistency, and the result is fed back to the decoder for refinement, generating a motion-constrained cloud probability map.

[0063] Step S60: Determine the polar night cloud detection result based on the cloud probability map after motion constraints.

[0064] It should be noted that the local Shannon entropy of the cloud probability map after motion constraints is used as the classification uncertainty, and combined with the motion consistency score to generate a cloud detection confidence map. Smooth and motion-consistent confidence features are generated through multi-scale Gaussian pyramid decomposition and anisotropic diffusion filtering. Based on the confidence map, the initial cloud mask is filtered and morphologically optimized to obtain the final polar night cloud detection results.

[0065] In one feasible implementation, step S60 may include: calculating the Shannon entropy of the motion-constrained cloud probability map in the local neighborhood and using the Shannon entropy as a classification uncertainty score; determining the motion consistency score of each pixel based on the cloud advection kernel and the motion information implied in the motion-constrained cloud probability map; generating a cloud detection confidence map based on the classification uncertainty score and the motion consistency score; binarizing the motion-constrained cloud probability map to obtain an initial cloud mask; filtering the initial cloud mask based on the cloud detection confidence map to obtain a filtered cloud mask; and performing morphological operations on the filtered cloud mask to obtain the polar night cloud detection result.

[0066] It should be noted that when calculating the local Shannon entropy, a 3×3 or 5×5 neighborhood window is divided with each pixel as the center. The entropy value quantifies the degree of disorder in the probability distribution of this region; a higher entropy value indicates greater classification uncertainty. The formula for calculating Shannon entropy is:

[0067] Where (x,y) are the center pixel coordinates, and r is the neighborhood radius. Let be the cloud probability value at the neighborhood (x+i, y+j). This is the normalized probability distribution.

[0068] Based on the cloud probability map after motion constraints, its optical flow field is calculated as the implicit motion information; the cosine similarity between the direction of the optical flow field and the direction indicated by the cloud advection motion kernel is calculated; after normalizing the cosine similarity, it is used as the motion consistency score, which can reflect the degree of matching between pixel motion and cloud advection motion. The higher the score, the stronger the motion consistency.

[0069] In one implementation, the cloud detection confidence map generation employs a weighted fusion strategy, dynamically combining classification uncertainty scores and motion consistency scores. A weighting coefficient α∈[0,1] is set to balance the influence of the two types of scores, and the confidence calculation formula is as follows: C(x,y)=α·(1-H(x,y))+(1-α)·M(x,y) Where H(x,y) is the normalized Shannon entropy value, and M(x,y) is the motion consistency score.

[0070] By adjusting the α value, the detection requirements in different scenarios can be adapted. In polar night environments, α is usually set to 0.6 to highlight motion consistency constraints.

[0071] The initial cloud mask generation employs an adaptive threshold segmentation method, binarizing the motion-constrained cloud probability map. The threshold selection is based on the Otsu algorithm using a global probability distribution, automatically determining the optimal segmentation threshold by maximizing the inter-class variance. This threshold can adapt to different cloud density scenarios, generating a binary mask that includes the initial cloud region.

[0072] Morphological optimization includes: first, opening operations are used to remove isolated noise points in the initial mask, with the kernel size set to 3×3 pixels; second, closing operations are used to fill holes inside the cloud region, with the kernel size set to 5×5 pixels; finally, connected component analysis is applied to remove tiny regions smaller than 20 pixels. The optimized mask accurately describes the cloud geometry while maintaining smooth edges.

[0073] The final polar night cloud detection result is generated by mapping the morphologically optimized cloud mask onto the original image coordinate system. The detection result includes cloud location, contour and confidence information. The confidence value is determined by the value of the corresponding position in the cloud detection confidence map.

[0074] In one feasible implementation, generating a cloud detection confidence map based on the classification uncertainty score and the motion consistency score includes: constructing a spatial uncertainty distribution matrix based on the classification uncertainty score, and performing multi-scale Gaussian pyramid decomposition on the spatial uncertainty distribution matrix to obtain a spatial uncertainty feature sequence containing multiple scales; extracting cloud region boundaries from the motion-constrained cloud probability map, and calculating the derivative of the boundary pixels along the motion direction based on the motion consistency score to generate a multi-scale motion consistency gradient map; and performing cross-scale feature fusion between the outputs of each layer of the multi-scale Gaussian pyramid and the gradient magnitudes of the corresponding scales. The spatial uncertainty feature sequence is fused with the multi-scale motion consistency gradient map at the corresponding scales to generate a motion-enhanced multi-scale uncertainty feature sequence. Based on the neighborhood structure and direction defined by the cloud advection motion kernel, the motion-enhanced multi-scale uncertainty feature sequence is subjected to top-down anisotropic diffusion filtering to obtain a smooth and motion trajectory-consistent multi-scale confidence feature sequence. The smooth and motion trajectory-consistent multi-scale confidence feature sequence is upsampled and concatenated, and then input into a lightweight convolutional network for nonlinear mapping and aggregation to generate a cloud detection confidence map.

[0075] It should be noted that multi-scale Gaussian pyramid decomposition generates spatial uncertainty features at different resolutions through layer-by-layer downsampling. Lower layers retain more detailed information, while higher layers extract global structural features. The spatial uncertainty distribution matrix is ​​used as the 0th layer (highest resolution) of the pyramid, and layers 1, 2, and up to N are generated by successively downsampling by a factor of 2, forming a multi-scale feature representation ranging from fine to coarse. The output of each layer is smoothed for edge artifacts using Gaussian filtering to ensure natural feature transitions.

[0076] In the motion-constrained cloud probabilistic map, the Canny edge detector is used to extract cloud region boundaries, and the gradient magnitude of the boundary pixels along the optical flow direction is calculated by combining the motion consistency score. For each boundary pixel, a first-order difference is calculated along its motion direction to generate a gradient map reflecting motion consistency. This gradient map is fused across scales with the outputs of each layer of the Gaussian pyramid, and feature complementarity is achieved through weighted averaging: low-level gradients enhance detail representation, and high-level gradients strengthen global motion patterns.

[0077] The fused motion-enhanced uncertainty feature sequence is further optimized using anisotropic diffusion filtering. Anisotropic diffusion refers to smoothing along the motion direction while preserving boundaries in the vertical direction. The filtering process is guided by the neighborhood structure defined by the cloud advection motion kernel, employing a larger diffusion coefficient in the motion direction and a smaller diffusion coefficient in the vertical direction to achieve smoothing based on the motion trajectory. This filtering iteratively updates feature values, gradually eliminating isolated noise points while maintaining the clarity of cloud boundaries, ultimately generating a multi-scale confidence feature sequence with motion consistency.

[0078] The filtered feature sequences at each scale are upsampled to the original resolution, and spatial consistency is restored through a concatenation operation. The result is then input into a lightweight convolutional network for nonlinear mapping. This network uses depthwise separable convolutions to reduce the number of parameters and achieves inter-channel information exchange through 1×1 convolutions. Finally, it outputs a normalized cloud detection confidence map with a value range of [0,1]. A larger value indicates a higher probability of cloud presence.

[0079] This embodiment provides a polar night cloud detection method based on the collaboration of event cameras and deep learning. In a polar night environment, asynchronous event streams output by an event camera and infrared image sequences output by an infrared camera are acquired. The asynchronous event stream is subjected to polarity-aware voxelization processing to generate an event voxel tensor. Based on the event voxel tensor, the infrared image sequence is subjected to temporal resolution enhancement to generate an enhanced infrared sequence. The event voxel tensor and the enhanced infrared sequence are input into an event-infrared fusion network for feature extraction and cross-modal fusion to obtain a preliminary cloud probability map. The event-infrared fusion network includes an event branch, an infrared branch, a cross-modal gated attention fusion module, and a lightweight decoder. The event branch uses lightweight spatiotemporal separation convolution to extract dynamic edge features, and the infrared branch uses a channel-pruned residual network variant to extract thermal structure features. Wind field information from numerical weather prediction data is acquired, and a cloud advection motion kernel is constructed based on the wind field information. Motion constraints are applied to the preliminary cloud probability map based on the cloud advection motion kernel to generate a motion-constrained cloud probability map. The polar night cloud detection result is determined based on the motion-constrained cloud probability map. By employing the above methods, and through the collaborative work of event cameras and infrared cameras, combined with deep learning and numerical weather forecast data, high-precision, high-temporal-resolution cloud detection was achieved in polar night environments.

[0080] Based on the first embodiment of this application, in the second embodiment of this application, the content that is the same as or similar to that in the first embodiment described above can be referred to the above description and will not be repeated hereafter. Based on this, step S50 includes steps S501 to S505: Step S501: Obtain wind field information from numerical weather forecast data, and extract the horizontal wind vector within a specified pressure layer range from the wind field information.

[0081] It should be noted that, in this embodiment, the wind field information in the numerical weather prediction data is extracted from numerical weather prediction data such as ECMWF or GDAS, including the horizontal wind vector field (u,v) within the 850 hPa to 300 hPa pressure layer range. This wind vector field data is acquired in real time through an API interface and undergoes data cleaning to remove outliers. Specifically, quality control is performed on the wind vector (u,v) of each pressure layer, removing outlier data points with a modulus exceeding 100 m / s or a change in direction exceeding 180 degrees to ensure the reliability of the input data.

[0082] In one feasible implementation, before step S501, the method may further include: acquiring numerical weather forecast data and performing data quality detection on the numerical weather forecast data to obtain data missing status and data quality score; triggering a backup mode when the data missing status is that the numerical weather forecast data has data missing or the data quality score is lower than a preset score threshold; in the backup mode, calculating a dense event optical flow field based on the event voxel tensor using an event optical flow estimation algorithm; constructing an endogenous motion kernel with the dense event optical flow field as the core; performing spatiotemporal convolution of the endogenous motion kernel with the preliminary cloud probability map to generate a cloud probability map based on endogenous motion constraints; and feeding back the cloud probability map based on endogenous motion constraints as a guiding signal to the lightweight decoder in the event-infrared fusion network to output the motion-constrained cloud probability map.

[0083] It should be noted that a backup mode is introduced in this implementation to address situations where numerical weather forecast data is missing or of poor quality. The delay between the timestamp of the numerical weather forecast data and the current time is checked. If the delay exceeds the tolerance window, the data is considered missing. The physical plausibility index of the wind field in the spatiotemporal dimensions of the numerical weather forecast data is calculated. If the index is abnormal, the data quality is considered unqualified. The backup mode is activated when the numerical weather forecast data missing rate is >20% or the quality score is <0.7.

[0084] In backup mode, the optical flow field of dense events is calculated based on event voxel tensors, and an endogenous motion kernel is constructed to replace the cloud advection motion kernel, ensuring that effective motion constraints can still be performed when numerical weather forecasts are lacking.

[0085] Understandably, the event optical flow estimation algorithm can employ a spatiotemporal optimization-based dense optical flow algorithm. This algorithm generates a pixel-level event motion field by minimizing a joint objective function that combines spatiotemporal gradient constraints and optical flow smoothness constraints. Specifically, an energy function is constructed that includes a data term and a smoothing term: the data term calculates pixel motion consistency based on the event trigger time difference, and the smoothing term constrains the gradient change of the optical flow field using the Laplacian operator; the conjugate gradient method is used to iteratively solve this nonlinear optimization problem, ultimately outputting a dense optical flow field (u_e, v_e) with resolution matching the event voxel tensor, as shown in the following equation:

[0086] in, For the optimal dense event optical flow field, Let I(x,y,t) be the set of spatiotemporal points composed of valid events in the event voxel tensor, and let I(x,y,t) be the spatiotemporal brightness function reconstructed from the event stream. , , , respectively, are the partial derivatives of the brightness function in the time, horizontal, and vertical directions. For the horizontal event optical flow component, For the vertical event optical flow component. For robust loss function, Regularization weights control the smoothing strength. Let be the image spatial domain, and Δ denote the Laplacian operator.

[0087] When constructing the endogenous motion kernel, based on the optical flow field of dense events, an optical flow direction histogram is calculated in a 3×3 neighborhood. The principal direction is selected as the local motion kernel direction, and the kernel radius is dynamically adjusted according to the optical flow amplitude; the larger the amplitude, the smaller the kernel radius to preserve details. During spatiotemporal convolution, the endogenous motion kernel and the preliminary cloud probability map are weighted and summed within the spatiotemporal 3D convolution kernel. The weights are determined by the optical flow amplitude normalization, thereby enhancing the motion consistency region and suppressing noise.

[0088] After receiving the feedback signal, the lightweight decoder uses a 1×1 convolutional layer to compress the channel dimension and fuses multi-scale features through residual connections. The final output is a motion-constrained cloud probability map with the same spatial resolution as the input. The probability values ​​are normalized to [0,1] using a sigmoid activation function. Experiments show that in scenarios with missing numerical weather prediction data, this backup mode can maintain a cloud detection recall rate above 82%, which is 17 percentage points higher than the scheme without motion constraints.

[0089] Step S502: Map the horizontal wind vector field onto the image plane through spherical projection and perspective transformation to obtain the pixel displacement vector.

[0090] It should be noted that mapping the horizontal wind vector field (u,v) within the pressure layer to the image plane requires coordinate transformation and geometric correction. First, a spherical projection model is used to convert the wind vector in the geographic coordinate system into spherical coordinates with the optical center of the event camera as the origin. The projection angle of the wind vector on the sphere is calculated using latitude, longitude, and camera focal length parameters. Then, the perspective transformation principle is used to map the spherical coordinates to the image plane coordinate system, where the projection center corresponds to the intersection of the camera optical axis and the cloud layer.

[0091] To compensate for the effects of Earth's curvature and atmospheric refraction, a correction coefficient is introduced to dynamically correct the displacement vector. During the mapping process, bilinear interpolation is used to handle non-integer pixel displacements, ensuring the spatial continuity of the displacement field. The final generated pixel displacement vector field is consistent with the image resolution, with each pixel corresponding to a two-dimensional displacement vector (Δx, Δy), representing the expected direction and amplitude of motion of that point under the influence of the wind field.

[0092] Step S503: Construct a cloud advection motion kernel with the pixel displacement vector as the core.

[0093] It should be noted that the cloud advection kernel is a two-dimensional Gaussian convolution kernel, the center of which corresponds to the endpoint of the pixel displacement vector, and the kernel size is dynamically adjusted according to the displacement amplitude.

[0094] Specifically, a circular neighborhood with radius r is constructed centered on the endpoint of the displacement vector. The radius r is proportional to the displacement amplitude |Δd| = sqrt(Δx² + Δy²), and the proportionality coefficient is experimentally set to 0.8 to ensure the motion kernel covers the actual movement range of the cloud layer. The weights within the kernel are distributed using a Gaussian function, with the highest weight at the center point decreasing towards the edges. The deceleration rate is controlled by the standard deviation σ, which is set to r / 3 to balance spatial smoothing and edge preservation.

[0095] For non-integer displacements, bicubic interpolation is used to generate subpixel-level motion kernels to ensure that the displacement vector is precisely aligned with the image pixels.

[0096] The constructed cloud advection kernel will be used for subsequent spatiotemporal convolution operations. Weighted summation will smooth the cloud probability map along the motion direction while suppressing noise interference in the vertical motion direction. Experiments show that dynamically adjusting the kernel size and weight distribution can reduce the cloud motion trajectory tracking error to within 1.2 pixels, significantly improving the motion constraint effect.

[0097] Step S504: Perform spatiotemporal convolution between the cloud advection kernel and the preliminary cloud probability map to generate a probability map with enhanced motion consistency.

[0098] It should be noted that the spatiotemporal convolution operation is implemented using a sliding window mechanism, with the window size consistent with the coverage area of ​​the cloud advection motion kernel. At each window location, the kernel weights are weighted and summed with the pixel values ​​of the corresponding region in the initial cloud probability map to generate the enhanced probability value. This operation is performed synchronously in the time dimension to ensure the consistency of motion constraints.

[0099] By dynamically adjusting the size of the motion kernel and the weight distribution, spatiotemporal convolution can effectively enhance the probabilistic response in the direction of cloud motion while suppressing noise interference in the vertical direction of motion. Experimental data show that after spatiotemporal convolution processing, the tracking error of cloud motion trajectory can be reduced to within 1.2 pixels, significantly improving the accuracy of motion constraints. The generated enhanced probability map maintains the same spatial resolution as the input, with probability values ​​still ranging from [0,1], but the values ​​in the motion consistency region are significantly improved.

[0100] Step S505: The motion consistency enhancement map is fed back as a guiding signal to the lightweight decoder in the event-infrared fusion network to output the motion-constrained cloud probability map.

[0101] It should be noted that after receiving the motion consistency enhancement map, the lightweight decoder treats it as an important guiding signal for further processing. The decoder first uses a 1×1 convolutional layer to compress the channel dimension of the enhancement map. This step aims to reduce data redundancy while retaining key motion feature information. Subsequently, through a residual connection mechanism, the decoder fuses the compressed features with multi-scale features. This fusion method can fully utilize information from different levels of features, improving the richness and accuracy of feature representation.

[0102] During the fusion process, the decoder pays special attention to motion-consistent regions, enhancing their features in the final output through weighted processing. Finally, the lightweight decoder, after a series of processing steps, outputs a motion-constrained cloud probability map. This map maintains the same spatial resolution as the input, ensuring the precision of the detection results. Simultaneously, the probability values ​​are normalized to [0,1] using a sigmoid activation function, making the results more intuitive and easier to interpret. Experiments show that by introducing a motion-consistency enhancement map as a guiding signal, the lightweight decoder can significantly improve the accuracy and stability of cloud detection, especially in complex and variable polar night environments, where this advantage is particularly pronounced.

[0103] In this embodiment, motion constraints are introduced by incorporating wind field information from numerical weather forecasts or event optical flow estimation to enhance the spatiotemporal consistency of cloud detection.

[0104] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the polar cloud detection method based on the collaboration of event camera and deep learning in this application. Any simple modifications based on this technical concept are within the protection scope of this application.

[0105] This application also provides a polar cloud detection device based on the collaboration of event camera and deep learning. Please refer to [reference needed]. Figure 2 The polar cloud detection device based on the collaboration of event cameras and deep learning includes: The acquisition module 10 is used to acquire the asynchronous event stream output by the event camera and the infrared image sequence output by the infrared camera in a polar night environment.

[0106] Processing module 20 is used to perform polarity-aware voxelization processing on the asynchronous event stream to generate event voxel tensors.

[0107] Enhancement module 30 is used to perform temporal resolution enhancement on the infrared image sequence based on the event voxel tensor to generate an enhanced infrared sequence.

[0108] The fusion module 40 is used to input the event voxel tensor and the enhanced infrared sequence into the event-infrared fusion network for feature extraction and cross-modal fusion to obtain a preliminary cloud probability map. The event-infrared fusion network includes an event branch, an infrared branch, a cross-modal gated attention fusion module, and a lightweight decoder. The event branch uses lightweight spatiotemporal separation convolution to extract dynamic edge features, and the infrared branch uses a channel-pruned residual network variant to extract thermal structure features.

[0109] The constraint module 50 is used to acquire wind field information from numerical weather forecast data, construct a cloud advection motion kernel based on the wind field information, and apply motion constraints to the preliminary cloud probability map based on the cloud advection motion kernel to generate a motion-constrained cloud probability map.

[0110] The determination module 60 is used to determine the polar night cloud detection result based on the cloud probability map after the motion constraints.

[0111] The polar cloud detection device based on event camera and deep learning collaboration provided in this application, employing the polar cloud detection method based on event camera and deep learning collaboration in the above embodiments, can solve the technical problems of low cloud detection accuracy and insufficient dynamic capture capability in the polar night environment of the prior art. Compared with the prior art, the beneficial effects of the polar cloud detection device based on event camera and deep learning collaboration provided in this application are the same as the beneficial effects of the polar cloud detection method based on event camera and deep learning collaboration provided in the above embodiments, and other technical features in the polar cloud detection device based on event camera and deep learning collaboration are the same as the features disclosed in the methods of the above embodiments, and will not be repeated here.

[0112] The above are only some embodiments of this application and do not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. A polar night cloud detection method based on event camera and deep learning collaboration, characterized in that, The method comprises: In the polar night environment, collect the asynchronous event stream output by the event camera and the infrared image sequence output by the infrared camera; Polarity-aware voxelization processing is performed on the asynchronous event stream to generate an event voxel tensor; Based on the event voxel tensor, the time resolution of the infrared image sequence is enhanced to generate an enhanced infrared sequence; The event voxel tensor and the enhanced infrared sequence are input into an event-infrared fusion network for feature extraction and cross-modal fusion to obtain a preliminary cloud probability map, wherein the event-infrared fusion network comprises an event branch, an infrared branch, a cross-modal gated attention fusion module, and a lightweight decoder, the event branch uses a lightweight spatio-temporal separation convolution to extract dynamic edge features, and the infrared branch uses a channel pruning residual network variant to extract thermal structure features; Obtain wind field information in numerical weather prediction data, construct a cloud advection motion kernel based on the wind field information, and constrain the preliminary cloud probability map based on the cloud advection motion kernel to generate a motion-constrained cloud probability map; Determine the polar night cloud layer detection result based on the motion-constrained cloud probability map.

2. The method of claim 1, wherein, The method comprises: Time synchronization is performed on the event camera and the infrared camera deployed in the polar observation station to obtain a synchronized event camera and a synchronized infrared camera; A temperature-trigger threshold compensation lookup table is established, and low-temperature threshold drift calibration is performed on the synchronized event camera based on the temperature-trigger threshold compensation lookup table to obtain a calibrated event camera; An asynchronous event stream output by the calibrated event camera is collected, and time slicing is performed on the asynchronous event stream in a preset time window to obtain the total number of events in each time window; The total number of events is compared with a preset sparsity threshold and a preset saturation threshold respectively to obtain a comparison result, and the density state of the asynchronous event stream is determined according to the comparison result; Based on the reinforcement learning strategy, the density state is used to adaptively adjust the acquisition parameters of the infrared camera, and an infrared image sequence output by the infrared camera after parameter adjustment is collected, wherein the acquisition parameters include exposure time and gain coefficient.

3. The method of claim 1, wherein, The method comprises: Determine a voxel aggregation time window according to the frame interval of the infrared image sequence and the density state of the asynchronous event stream; In the voxel aggregation time window, separate the events in the asynchronous event stream into a positive event set and a negative event set according to polarity; Discrete mapping in the time dimension is performed on the positive event set and the negative event set respectively using a non-uniform time binning strategy to generate discrete spatio-temporal grids of the positive event set and the negative event set; Event counts are accumulated in the discrete spatio-temporal grids of the positive event set and the negative event set respectively to generate an event voxel tensor.

4. The method of claim 1, wherein, The method comprises: Calculate the rate of change of event density over time based on the event voxel tensor; Detect a local density extreme point according to a rate of change of the event density over time, and determine a time point corresponding to the local density extreme point as a feature moment of a significant change of a cloud structure; Select two original infrared images adjacent to the feature moment from the infrared image sequence; Estimate a bidirectional optical flow field based on a phase-based motion estimation algorithm based on the two original infrared images; Generate a virtual infrared frame corresponding to the feature moment through a forward warping and hole filling algorithm based on the bidirectional optical flow field; Insert the virtual infrared frame into the infrared image sequence to generate an enhanced infrared sequence.

5. The method of claim 1, wherein, The event voxel tensor and the enhanced infrared sequence are input into an event-infrared fusion network for feature extraction and cross-modal fusion to obtain a preliminary cloud probability map, including: The event voxel tensor is input into an event branch to extract dynamic edge features of the event through lightweight spatio-temporal separation convolution; The enhanced infrared sequence is input into an infrared branch to extract multi-scale thermal structure features of the infrared image through a residual network variant with channel pruning; The dynamic edge features and the thermal structure features of the highest layer are input into a cross-modal gated attention module to calculate event-guided infrared attention weights and infrared-guided event attention weights; The dynamic edge features and the thermal structure features of the highest layer are weighted and fused based on the infrared attention weights and the event attention weights to obtain complementary enhanced features; The complementary enhanced features and the thermal structure features of the lower layer are sent to a lightweight decoder through a skip connection to output a preliminary cloud probability map.

6. The method of claim 1, wherein, The wind field information in the numerical weather prediction data is obtained, the cloud advection motion kernel is constructed based on the wind field information, and the preliminary cloud probability map is motion-constrained based on the cloud advection motion kernel to generate a cloud probability map after motion constraint, including: The wind field information in the numerical weather prediction data is obtained, and the horizontal wind vector in the specified pressure layer range is extracted from the wind field information; The horizontal wind vector field is mapped to the image plane through spherical projection and perspective transformation to obtain a pixel displacement vector; The cloud advection motion kernel is constructed with the pixel displacement vector as the core; The cloud advection motion kernel and the preliminary cloud probability map are spatio-temporally convolved to generate a motion consistency enhanced probability map; The motion consistency enhanced map is fed back to the lightweight decoder in the event-infrared fusion network as a guide signal to output a cloud probability map after motion constraint.

7. The method of claim 6, wherein, Before the wind field information in the numerical weather prediction data is obtained, and the horizontal wind vector in the specified pressure layer range is extracted from the wind field information, it further includes: Obtain numerical weather prediction data, and perform data quality detection on the numerical weather prediction data to obtain data missing conditions and data quality scores; When the data missing condition is that the numerical weather prediction data has data missing or the data quality score is lower than a preset score threshold, trigger a backup mode; In the backup mode, a dense event optical flow field is calculated based on the event voxel tensor through an event optical flow estimation algorithm; An endogenous motion kernel is constructed with the dense event optical flow field as the core; spatially and temporally convolve the endogenous motion kernel with the preliminary cloud probability map to generate a cloud probability map based on endogenous motion constraint; feed the cloud probability map based on endogenous motion constraint as a guidance signal to a lightweight decoder in the event-infrared fusion network to output a cloud probability map after motion constraint.

8. The method of claim 1, wherein, determine a polar night cloud layer detection result based on the cloud probability map after motion constraint, including: calculate a Shannon entropy of the cloud probability map after motion constraint in a local neighborhood, and take the Shannon entropy as a classification uncertainty score; determine a motion consistency score of each pixel based on motion information implied by the cloud advection motion kernel and the cloud probability map after motion constraint; generate a cloud detection confidence map according to the classification uncertainty score and the motion consistency score; binarize the cloud probability map after motion constraint to obtain an initial cloud mask; filter the initial cloud mask based on the cloud detection confidence map to obtain a filtered cloud mask; perform morphological operations on the filtered cloud mask to obtain a polar night cloud layer detection result.

9. The method of claim 8, wherein, generate a cloud detection confidence map according to the classification uncertainty score and the motion consistency score, including: construct a spatial uncertainty distribution matrix based on the classification uncertainty score, and perform multi-scale Gaussian pyramid decomposition on the spatial uncertainty distribution matrix to obtain a spatial uncertainty feature sequence containing multiple scales; extract a cloud region boundary in the cloud probability map after motion constraint, and calculate a derivative of a boundary pixel along a motion direction based on the motion consistency score to generate a multi-scale motion consistency gradient map; perform cross-scale feature fusion on outputs of each layer of the multi-scale Gaussian pyramid and gradient amplitudes of corresponding scales to generate motion-enhanced uncertainty features; perform element-by-element fusion on the spatial uncertainty feature sequence and the multi-scale motion consistency gradient map at corresponding scales to generate a motion-enhanced multi-scale uncertainty feature sequence; perform top-down anisotropic diffusion filtering on the motion-enhanced multi-scale uncertainty feature sequence based on a neighborhood structure and a direction defined by the cloud advection motion kernel to obtain a multi-scale confidence feature sequence that is smooth and consistent in motion trajectory; perform upsampling and concatenation on the multi-scale confidence feature sequence that is smooth and consistent in motion trajectory, and input a lightweight convolutional network to generate a cloud detection confidence map through nonlinear mapping and aggregation.

10. A polar polar-night cloud layer detection device based on event camera and deep learning collaboration, characterized in that, The device includes: an acquisition module configured to acquire an asynchronous event stream output by an event camera and an infrared image sequence output by an infrared camera in a polar night environment; a processing module configured to perform polarity-aware voxelization processing on the asynchronous event stream to generate an event voxel tensor; an enhancement module configured to perform time resolution enhancement on the infrared image sequence based on the event voxel tensor to generate an enhanced infrared sequence; a fusion module configured to input the event voxel tensor and the enhanced infrared sequence into an event-infrared fusion network to perform feature extraction and cross-modal fusion, and obtain a preliminary cloud probability map, wherein the event-infrared fusion network comprises an event branch, an infrared branch, a cross-modal gated attention fusion module, and a lightweight decoder, the event branch is configured to extract dynamic edge features by using lightweight space-time separation convolution, and the infrared branch is configured to extract thermal structure features by using a channel-pruned residual network variant; a constraint module configured to obtain wind field information in numerical weather prediction data, construct a cloud advection kernel based on the wind field information, and perform motion constraint on the preliminary cloud probability map based on the cloud advection kernel, to generate a cloud probability map after motion constraint; a determination module configured to determine a polar night cloud layer detection result based on the cloud probability map after motion constraint.

Images