Unsupervised thermal imaging video enhancement method based on event-assisted traffic sign sketch

Abstract

The invention provides an unsupervised thermal imaging video enhancement method based on an event-assisted traffic sign sketch. According to the method, the characteristics of low light perception but texture missing of an infrared camera and low light, high dynamic, noise and incompleteness of event signals are fully combined, a dual lifting mechanism of an infrared frame and the event signals is provided for the traffic semantic reconstruction problem in the driving environment, and high-quality traffic sign sketch drawing without supervision is achieved. According to the method, accurate motion estimation is provided by utilizing infrared signals to serve as a time reference for aligning event signals so as to align incomplete event signals. Event signals are utilized to provide texture details which are usually not perceived by an infrared camera, and event consistency correction is introduced to improve the continuous perception effect between adjacent frames. According to the unsupervised thermal imaging video enhancement method provided by the invention, the applicability of the infrared camera in a low-light traffic environment is improved, and the improvement of the visual task performance in the middle and downstream of automatic driving is facilitated.

Description

Technical Field

[0001] This application belongs to the field of multimodal perception technology for autonomous driving, and specifically relates to an unsupervised thermal imaging video enhancement method based on event-assisted traffic sign sketches. Through the collaborative perception of infrared cameras and event cameras, it solves the problem of traffic sign texture loss under extreme lighting conditions and improves the environmental understanding ability of the autonomous driving system. Background Art

[0002] In low-light autonomous driving environments, infrared (IR) cameras can provide relatively clear images in dark environments. However, because IR cameras have difficulty capturing texture details of homogeneous materials, they are limited in detecting traffic signs and identifying key information. Furthermore, event cameras (i.e., neuromorphic vision cameras), as a new type of asynchronous sensor, can efficiently detect dynamic light intensity changes under high-speed and complex lighting conditions. Although their signals are incomplete and susceptible to noise, they have strong advantages in target detection in dynamic environments.

[0003] In the research field of autonomous driving perception systems, a single sensor cannot achieve high-fidelity and high-precision visual perception under low-light conditions. Although common multimodal fusion solutions (such as the combination of visible light and infrared or event signals) and single-sensor perception have improved, visible light cameras will fail due to excessively low signal-to-noise ratio, resulting in failure of the entire system.

[0004] However, the unsupervised thermal imaging video enhancement method based on event-assisted traffic sign sketches proposed in this application achieves high-quality traffic sign sketching under unsupervised conditions through the dual enhancement mechanism of infrared frames and event signals, as well as cross-modal spatiotemporal collaboration and semantic complementarity, enhances the traffic semantic perception capability of infrared cameras in low-light environments, and provides more robust environmental understanding capabilities for autonomous driving systems. Summary of the Invention

[0005] This paper proposes an unsupervised thermal imaging video enhancement method based on event-assisted traffic sign sketching to achieve high-quality traffic sign sketching under unsupervised conditions.

[0006] The technical solution of the present invention:

[0007] The unsupervised thermal imaging video enhancement method based on event-assisted traffic sign sketches is as follows:

[0008] Step 1: Process the two modal visual data collected by the event camera and the infrared camera to achieve spatial alignment

[0009] First determine the positions of the event camera and the infrared camera ( Figure 2), and then use the hard trigger mode to synchronously capture the infrared frame and event signal. The intrinsic parameter matrix of the infrared camera and event camera is estimated by the checkerboard calibration method. and And the external parameters between the two (rotation matrix With translation vector ), thereby constructing the external parameter matrix and realizing the spatial alignment of infrared frames and event signals as follows:

[0010] ∙ (1)_

[0011] ∙ (2)_

[0012] ∙ (3)_

[0013] in, : represents the normalized 3D point coordinates in the event camera coordinate system; Represents the normalized three-dimensional point coordinates in the infrared camera coordinate system; : represents the pixel coordinates on the event camera image plane; : represents the pixel point on the infrared camera image plane, that is, the pixel coordinates after the three-dimensional point is projected onto the infrared image plane;

[0014] Step 2: Based on the spatially aligned infrared frames and event signals, a rough outline of the semantic information extending from the event signal to the infrared frame is achieved. The event signal is depicted through the obvious edges of the infrared image to align the temporally asynchronous signals to the same moment.

[0015] Construct a multimodal encoding network based on the encoder-decoder bimodal fusion framework ( Figure 3 ), in the multimodal coding network, event encoder and infrared encoder are used to extract features of event signal and infrared frame image data respectively, and the information is fused in the middle layer to generate a low-resolution semantic sketch. The multimodal coding network structure is a symmetric U-Net, the encoding part is a 4-layer convolution unit, and the decoding part recovers spatial information through deconvolution and jump connection ( Figure 3 ).

[0016] Furthermore, in the feature extraction process, the event encoder Operation, the first layer of event features output is expressed as:

[0017] (4)

[0018] in, is the input event signal, Encoding event.

[0019] Then, the features of each layer are generated from the features of the last layer, and the features of the last layer are represented as:

[0020] (5)

[0021] where, are the features generated from the k-1 layer;

[0022] Similarly, for the infrared image encoder , the features of the last layer are represented as:

[0023] (6)

[0024] Further, in the decoding part, a hierarchical decoding structure is constructed to receive features of different scales to reconstruct the sketch result, where the feature of the k-th layer decoder is represented as:

[0025] (7)

[0026] where, denotes the up-sampling process, which has the same size as the down-sampling process, denotes the concatenation operation, denotes the feature of the k-th layer decoder, denotes the decoding operation. When k = 0, the final fused sketch result is obtained .

[0027] Step 3, correspond the event image and the infrared image in time sequence and position

[0028] Using the motion compensation mechanism of event signals Figure 4 , assuming that the event camera and the infrared camera have the same motion trajectory, by calculating the relative pose between the two, and combining the respective internal parameter information, the pixel position corresponding to the event signal is mapped to the infrared image, realizing the correspondence in position.

[0029] Using the motion information between consecutive infrared frames, the motion speed and direction of the event camera are estimated, so as to realize the time alignment and remapping of the event signal, ensure that all event data are aligned to a unified time slot, and realize the consistency in time sequence.

[0030] Step 3 is realized through a spatio-temporal consistency constraint module Figure 5 , in a series of rough sketches obtained after step 2 fusion , the current frame is specified as , in a defined time window The rough outline result after fusion in step 2 is encoded to facilitate the extraction of semantic information from consecutive frames. The generated features are used as embedding representation to guide the reconstruction of the final frame in the window. The value k is obtained through experiments and is preferably set to k=5. The specific process is as follows:

[0031] Step 3.1, Each frame is divided into multiple independent image blocks, and the image blocks are encoded to obtain features , further feature Apply spatial and temporal attention mechanisms in turn for processing:

[0032] (8)

[0033] in, is the result of feature F being processed by the spatial attention mechanism. Processing for the spatial attention mechanism (full name: SpatialAttention);

[0034] Step 3.2: Synthesize the set of frames within the event window and perform temporal attention to capture and integrate temporal correlations:

[0035] (9)

[0036] in, is the space-time fusion feature, Processing operations for the temporal attention mechanism (full name: TemporalAttention).

[0037] Step 3.3, current target frame Feature processing

[0038] (10)

[0039] Among them, FP is the frame feature extraction module, The feature of the current target frame (full name FeatProjection)

[0040] Step 3.4: Fusion of spatial and temporal features and target frame features The three tuples Q, K, and V are calculated together and the final feature fusion reconstruction is completed.

[0041] (11).

[0042] in, A fusion decoder.

[0043] The beneficial effects of the present invention are as follows: the present invention realizes the fusion of neuromorphic event signals and infrared signals under unsupervised conditions, reconstructs semantic information in traffic driving scenes, such as traffic signs and roadblock information, and ensures the continuous perception effect between adjacent frames obtained by preliminary fusion through the temporal consistency module. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 Schematic diagram of the process of the unsupervised thermal imaging video enhancement method of the present invention;

[0045] Figure 2 The camera model used for the method to collect infrared frames and event signals;

[0046] Figure 3 Schematic diagram of the bimodal fusion framework for generating preliminary semantic sketches;

[0047] Figure 4 Schematic diagram of the motion compensation mechanism for event signals;

[0048] Figure 5 Schematic diagram of the spatiotemporal consistency enhancement module.

[0049] Figure 6 This is an example result diagram of the invention. DETAILED DESCRIPTION

[0050] The specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. It should be understood that the embodiments described below are only used to illustrate the present invention and do not constitute a limitation of the present invention.

[0051] In the traffic sign scene, using event cameras and infrared cameras ( Figure 2 ) for synchronous acquisition. In this embodiment, the Davis 346 event camera and the MV-CI003-GLNx infrared camera are used. The two cameras maintain a stable geometric relationship via a fixed platform, and a unified triggering mechanism ensures temporal consistency between frames. During acquisition, the event camera outputs irregular event stream data, while the infrared camera outputs a continuous grayscale image sequence. The collected event data is reconstructed by integrating events within a specific time window into grayscale images (event frames) and aligning them with the infrared images at the corresponding time points. Based on the aligned images, edge information of the event frames is extracted to assist in sketch generation.

[0052] The method extracts multi-scale features by inputting event graphs and infrared images into a multimodal encoding network. Two encoders, an event encoder and an infrared encoder, are used to extract features from the two types of image data. The information is then fused at the intermediate layer to generate a low-resolution semantic sketch. The network structure is a symmetric U-Net, with a 4-layer convolutional unit in the encoding part and a decoding part that recovers spatial information through deconvolution and skip connections ( Figure 3 ).

[0053] After completing the preliminary outline of semantic information on the infrared frame in combination with the event signal, the rough sketch reconstructed in the previous step often shows limited visual continuity, considering the uneven distribution of attributes in the event signal. The method implements an event compensation mechanism ( Figure 4 ), assuming that the time window is 4 infrared frames, the specific approach is: the motion trajectory state between the infrared image sequences is used to estimate the camera motion information. Based on the fact that the motion trajectories of the two cameras are basically consistent, the event points corresponding to the four infrared frames are time-normalized and spatially projected, so that the event graph corresponds to the infrared image in terms of time sequence and position.

[0054] After obtaining all the rough semantic sketches constructed in the previous order within a complete time window, the spatiotemporal consistency constraint module ( Figure 5 ), which can capture inter-frame correlation under non-uniform sampling and achieve the forced spatiotemporal consistency across multiple frames. The spatiotemporal consistency constraint module is based on the temporal attention mechanism to build an inter-frame correlation graph, align and enhance the regions with the same position or similar features in the previous and next frames, and obtain a final sketch that integrates important information in the temporal neighborhood, has richer content, and more accurate traffic semantic information ( Figure 6 ).

Claims

1. Unsupervised thermal imaging video enhancement method based on event-assisted traffic sign sketches, characterized by: Here are the steps: Step 1: Process the two modal visual data collected by the event camera and the infrared camera First, the positions of the event camera and the infrared camera are determined, and then the hard trigger mode is used to synchronously capture the infrared frame and event signal. The intrinsic parameter matrix of the infrared camera and the event camera is estimated by the checkerboard calibration method. and And the external parameter rotation matrix between the two With translation vector , thus constructing the external parameter matrix and realizing the spatial alignment of infrared frames and event signals as follows: ∙ (1)_ ， ∙ (2)_ ， ∙ (3)_ ， in, : represents the normalized 3D point coordinates in the event camera coordinate system; Represents the normalized three-dimensional point coordinates in the infrared camera coordinate system; : represents the pixel coordinates on the event camera image plane; : represents the pixel point on the infrared camera image plane, that is, the pixel coordinates after the three-dimensional point is projected onto the infrared image plane; Step 2: Based on the spatially aligned infrared frames and event signals, a rough outline of the semantic information extending from the event signal to the infrared frame is achieved. The event signal is depicted by the obvious edges of the infrared image to align the temporally asynchronous signals to the same moment. A multimodal encoding network based on an encoder-decoder bimodal fusion framework was constructed. In the multimodal encoding network, an event encoder and an infrared encoder were used to extract features from event signals and infrared frames, respectively. The information was fused at the intermediate layer to generate a low-resolution semantic sketch. The multimodal encoding network structure is a symmetric U-Net, with a four-layer convolutional unit in the encoding part and a decoding part that recovers spatial information through deconvolution and skip connections. Step 3: Match the event graph and infrared image in terms of time sequence and position Using the motion compensation mechanism of the event signal, the event camera and the infrared camera are assumed to have the same motion trajectory. By calculating the relative posture between the two and combining their respective intrinsic reference information, the pixel position corresponding to the event signal is mapped to the infrared image to achieve positional correspondence. The motion information between consecutive infrared frames is used to estimate the motion speed and direction of the event camera, thereby achieving time alignment and remapping of event signals, ensuring that all event data are aligned to a unified time slot and achieving temporal consistency.

2. The unsupervised thermal imaging video enhancement method based on event-assisted traffic sign sketches according to claim 1 is characterized in that: In step 2, during the feature extraction process, the event encoder Operation, the first layer of event features output is expressed as: （4） ， in, is the input event signal, is the event encoder; After that, the features of each layer are generated from the features of the last layer, so the features of the last layer are expressed as: （5）， in, Features generated for the k-1 layer; The same is true for infrared image encoders , the last layer of features is represented as: （6）。 3. The unsupervised thermal imaging video enhancement method based on event-assisted traffic sign sketches according to claim 1 is characterized in that: In step 2, in the decoding part, a hierarchical decoding structure is constructed to receive features of different scales to reconstruct the sketch result, where the decoder features of each layer are represented as: (7) ， in, represents the upsampling process, which has the same size as the downsampling process, Indicates cascade operation, represents the k-th layer decoder feature representation, Represents the decoding operation; when k=0, the final fused sketch result is obtained .

4. The unsupervised thermal imaging video enhancement method based on event-assisted traffic sign sketches according to claim 1 is characterized in that: Step 3 is implemented by the spatiotemporal consistency constraint module, which is a series of rough outlines obtained after fusion in step 2. In the , specify the current frame as , in the defined time window The rough outline result after fusion in step 2 is encoded to facilitate the extraction of semantic information from consecutive frames. The generated features are used as embedding representation to guide the reconstruction of the final frame in the window. The specific process is: Step 3.1, Each frame is divided into multiple independent image blocks, and the image blocks are encoded to obtain features , further feature Apply spatial and temporal attention mechanisms in sequence for processing: (8) ， in, is the result of feature F being processed by the spatial attention mechanism. Processed by spatial attention mechanism; Step 3.2: Synthesize the set of frames within the event window and perform temporal attention to capture and integrate temporal correlations: (9) ， in, is the space-time fusion feature, Processing operations for the temporal attention mechanism; Step 3.3, current target frame Feature processing (10) ， Among them, FP is the frame feature extraction module, is the feature of the current target frame Step 3.4: Fusion of spatial and temporal features and target frame features They are input into the fusion decoding module together, and the co-attention mechanism is introduced to calculate the triples Q, K, and V respectively to complete the final feature fusion reconstruction: (11)； in, A fusion decoder.

Images