A video coding fusion method of infrared and microwave information based on adaptive registration

Abstract

The present invention discloses a method for infrared and microwave information video coding fusion based on adaptive registration, Step 1, collecting information fragments of infrared and microwave imaging devices in the same scene; Step 2, preprocessing the collected infrared video fragments, pulse compressing the collected microwave information fragments and calculating the frequency vector to convert them into radar RD map preprocessing operations; Step 3, aligning and fusing infrared video frames and microwave information fragments to generate fused video frames; Step 4, post-processing operations using local contrast enhancement technology to enhance details and features in the image; Step 5, using H.265 / HEVC to encode the fused video frame sequence into a video file. The present invention makes full use of the characteristics of infrared and microwave information, realizes target visualization in special scenes, ensures accurate matching and alignment of target objects in infrared and microwave information, improves the accuracy of fusion, and broadens the application field of infrared video and microwave information video coding fusion technology.

Description

Technical Field

[0001] The invention relates to the technical field of video encoding processing, and in particular to a method for fusion of infrared and microwave information video encoding based on adaptive registration. Background Art

[0002] In modern video coding and processing technology, the fusion of infrared and microwave information plays an important role in many application fields, especially in scenarios where multimodal data needs to be efficiently processed and transmitted, such as security monitoring, military target identification, unmanned driving, etc. Infrared imaging can capture clear thermal radiation information in low-light or nighttime environments, while microwave radar can penetrate complex meteorological conditions such as smoke and haze, and provide information on the shape and movement of the target.

[0003] However, the images of these two modalities have different physical properties and spatial resolutions. How to effectively align, fuse and compress them for transmission has become a challenge in current technology. Traditional multimodal image fusion methods often ignore the differences between targets in different modalities, resulting in information loss or poor fusion quality. Summary of the invention

[0004] In view of the above-mentioned technical deficiencies, the purpose of the present invention is to provide a method for infrared and microwave information video coding fusion based on adaptive registration. By introducing adaptive registration and efficient video coding, it is possible to greatly improve the video compression efficiency, reduce the transmission bandwidth, and improve the real-time and reliability of processing while ensuring the accuracy of information fusion.

[0005] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0006] The present invention provides a method for encoding and fusing infrared and microwave information video based on adaptive registration, comprising the following steps:

[0007] Step 1, collect information fragments of infrared and microwave imaging devices in the same scene to obtain infrared video fragments and microwave information fragments;

[0008] Step 2: Extract video frames from the infrared video clips collected in Step 1 and perform preprocessing using lightweight convolution kernel sharpening, perform pulse compression on the collected microwave information clips and calculate the frequency vector to convert it into a radar RD (Range-Doppler) image;

[0009] Step 3, using the adaptive anchor frame registration and fusion method to align and fuse the infrared video frame and microwave information fragments to generate a fused video frame;

[0010] Step 4, using local contrast enhancement technology to post-process the fused video frames in Step 3 to enhance the details and features in the image;

[0011] Step 5: Use H.265 / HEVC to encode the fused video frame sequence into a video file.

[0012] Preferably, step 2 specifically includes:

[0013] Step 2.1-1, use Python-OpenCV to extract each frame of the infrared video clip, and save each frame image in memory for subsequent processing;

[0014] Step 2.1-2, use the lightweight convolution kernel sharpening method to pre-operate the infrared video frame. Represents the image at coordinates The pixel value at Represents the convolution kernel In coordinates The value at , m and n are half of the number of rows and columns of the convolution kernel respectively, and the calculation formula is:

[0015] ;

[0016] ;

[0017] The image is sharpened by assigning positive weights to the central pixel and negative weights to the surrounding pixels.

[0018] Step 2.2-1. Use the matched filter method to compress the microwave information fragments. First, define some constants for calculation in MATLAB: sampling frequency fs, pulse repetition frequency PRF and speed of light c. Use the matched filter to compress the pulse to improve the radar resolution. The output of the matched filter is:

[0019] ;

[0020] in, is the received radar echo signal, is the impulse response of the matched filter. After pulse compression processing, the distance of the target is calculated by measuring the time difference between the transmitted signal and the received signal, as shown in the formula:

[0021] ;

[0022] Step 2.2-2, use the method based on calculating the frequency vector to convert the microwave information fragment into the radar RD (Range-Doppler) diagram to enhance the visualization of the target. The time for the radar signal to propagate to the target and return is proportional to the target distance. The distance vector can be obtained by dividing by the sampling frequency and multiplying by half the speed of light. Doppler processing is to obtain the speed information of the target. The radial speed of the target It will cause the frequency of the received radar signal to shift. In order to analyze and extract the Doppler frequency component in the signal, a frequency vector needs to be generated. Therefore, a fast Fourier transform is performed on the received radar signal to convert the time domain signal into a frequency domain signal to obtain the frequency component. The formula for calculating the frequency vector and FFT is:

[0023] ;

[0024] ;

[0025] The linspace function generates a linear space vector from -0.5×PRF to 0.5×PRF. PRF is the pulse repetition frequency, which determines the range of the Doppler frequency. is a time domain signal, is the frequency domain signal and N is the signal length.

[0026] Preferably, step Step 3 specifically includes:

[0027] The target detection boxes in the radar RD (Range-Doppler) image and the infrared image are respectively denoted as and , by calculating a transformation matrix , so that after the transformation and Overlap, define the transformation matrix , which is expressed in the form of:

[0028] ;

[0029] in, represents the scale factor, represents the rotation angle, and Represent the translation amount respectively, and optimize the transformation matrix by minimizing the Euclidean distance d between the target anchor boxes. , by iteratively adjusting the transformation matrix The parameters in make the objective function reach the minimum value, and the objective function is as follows:

[0030] ;

[0031] in, Represents the weight factor of the i-th anchor box, which is used to adjust the contribution of different anchor boxes to the objective function; Represents the Euclidean distance between the transformed anchor box and the target anchor box; represents the regularization term; Represents the weight of the regularization term, which is used to balance the influence of the registration error and the regularization term;

[0032] Due to the unique properties of infrared images and their blurred and tiny target features, after comparing various excellent image fusion methods at this stage, it is found that complex fusion methods are not suitable for image fusion processing in such scenarios, which often lead to image artifacts and target information loss. Adding consistency strategy to simple weighted fusion can achieve better performance. The fused features are expressed as , the calculation formula is:

[0033] ;

[0034] in, and They are the features extracted from radar RD images and infrared images respectively. It is to adjust the weight of the contribution of two features. In the process of feature fusion, in order to ensure the consistency of the fused features, a consistency test can be performed. Specifically, the difference between the features before and after fusion and the original features can be compared. If the difference exceeds a certain threshold, the weight needs to be readjusted or other processing needs to be performed to ensure the consistency of the fusion results.

[0035] Preferably, step 4 specifically includes:

[0036] The non-local mean filtering enhancement method is used to calculate similar blocks in the image to smooth the noise. First, the original image is taken as input ,in Represents the pixel coordinates in the image. For each pixel , calculate it with other pixels The weighted average of is determined by the similarity between the two. The formula for non-local mean filtering is:

[0037] ;

[0038] in, Indicates that the enhanced image has pixels The value at It's pixels and pixels The similarity weight between is determined by the distance between them and is defined as:

[0039] ;

[0040] in, and Pixels and pixels is the image patch centered, h is the smoothing parameter, is the normalization factor. After non-local means filtering, the noise is effectively removed, while the details and edges are enhanced.

[0041] Preferably, step 5 specifically includes:

[0042] Install and configure a video encoder that supports H.265 / HEVC encoding, and convert the fused image sequence into a video file format. First, by analyzing the redundant information of the fused frames, dynamically optimize the encoding parameters, and adjust the size and shape of the encoding block to adapt to complex scenes; second, use the complementarity of infrared and microwave information to improve motion estimation and enhance prediction accuracy; finally, by optimizing entropy coding, give priority to encoding the target area, reduce the background information bit rate, and improve compression efficiency. The core formula is:

[0043] ;

[0044] Where R represents the bit rate after encoding, Indicates the size of the encoding block, Indicates the accuracy of intra-block prediction; finally, the encoded video file is checked and tested to ensure that the video quality and compression efficiency meet the requirements.

[0045] The beneficial effect of the present invention is that the method can solve the problems of insufficient registration accuracy, serious information loss and low compression efficiency in the existing multimodal information fusion process. Through the invention, more efficient video compression and transmission can be achieved on the basis of ensuring high precision and multimodal information fusion, which is suitable for scenarios requiring low latency and high reliability, such as intelligent monitoring, unmanned driving and military reconnaissance. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0047] Figure 1 It is a schematic diagram of the process of the present invention;

[0048] Figure 2 This is a diagram showing the sharpening effect of a lightweight convolution kernel in an embodiment of the present invention;

[0049] Figure 3 Generate a radar RD diagram for microwave information in an embodiment of the present invention;

[0050] Figure 4 Schematic diagram of the adaptive registration fusion algorithm used in the embodiment of the present invention

[0051] Figure 5 This is a flowchart of encoding a fused video frame sequence into a video file using H.265 / HEVC in an embodiment of the present invention. DETAILED DESCRIPTION

[0052] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0053] Embodiment 1, as Figure 1 As shown, a method for infrared and microwave information video coding fusion based on adaptive registration includes the following steps:

[0054] Step 1: Collect information fragments of infrared and microwave imaging devices in the same scene to obtain infrared video fragments and microwave information fragments;

[0055] Step 2: Extract video frames from the infrared video clips collected in Step 1 and perform preprocessing using lightweight convolution kernel sharpening, perform pulse compression on the collected microwave information clips and calculate the frequency vector to convert it into a radar RD (Range-Doppler) map;

[0056] Step 3: Adopt the adaptive anchor frame registration and fusion method to align and fuse the infrared video frame and microwave information fragments to generate a fused video frame;

[0057] Step 4: Use local contrast enhancement technology to post-process the fused video frames in Step 3 to enhance the details and features in the image;

[0058] Step 5: Use H.265 / HEVC to encode the fused video frame sequence into a video file.

[0059] As a preferred implementation, the steps of extracting video frames from the collected infrared video segments and performing preprocessing operations using lightweight convolution kernel sharpening, performing pulse compression on the collected microwave information segments and calculating the frequency vector to convert them into radar RD (Range-Doppler) images in Step 2 are as follows:

[0060] Step 2.1-1, use Python-OpenCV to extract each frame of the infrared video clip, and save each frame image in memory for subsequent processing;

[0061] Step 2.1-2, use the lightweight convolution kernel sharpening method to pre-operate the infrared video frame. Represents the image at coordinates The pixel value at Represents the convolution kernel In coordinates The value at , m and n are half of the number of rows and columns of the convolution kernel respectively, and the calculation formula is:

[0062] ;

[0063] ;

[0064] By assigning a positive weight (5) to the central pixel and a negative weight (-1) to the surrounding pixels, the edge features of the infrared image can be enhanced, the image clarity and detail can be improved, and the image sharpening effect can be achieved. Figure 2 As shown;

[0065] Step 2.2-1. Use the matched filter method to compress the microwave information fragments. First, define some constants for calculation in MATLAB, such as sampling frequency fs, pulse repetition frequency PRF and speed of light c. Use the matched filter to compress the pulse to improve the radar resolution. The output of the matched filter is:

[0066] ;

[0067] in, is the received radar echo signal, is the impulse response of the matched filter. After the pulse compression process, the distance of the target is calculated by measuring the time difference between the transmitted signal and the received signal, as shown in the formula:

[0068] ;

[0069] Step 2.2-2, use the method based on calculating the frequency vector to convert the microwave information fragment into the radar RD (Range-Doppler) diagram; the time for the radar signal to propagate to the target and return is proportional to the target distance. The distance vector can be obtained by dividing by the sampling frequency and multiplying by half the speed of light; Doppler processing is to obtain the speed information of the target, the radial speed of the target It will cause the frequency of the received radar signal to shift. In order to analyze and extract the Doppler frequency component in the signal, a frequency vector needs to be generated. Therefore, the received radar signal is subjected to a fast Fourier transform (FFT) to transform the time domain signal into a frequency domain signal and obtain the frequency component. The formula for calculating the frequency vector and FFT is:

[0070] ;

[0071] ;

[0072] Among them, the linspace function generates a linear space vector from -0.5×PRF to 0.5×PRF, PRF is the pulse repetition frequency, which determines the range of Doppler frequency. is a time domain signal, is the frequency domain signal, and N is the signal length. By converting microwave data into a visual radar RD graph, it is convenient for subsequent processing, such as Figure 3 As shown;

[0073] As a preferred implementation, the adaptive anchor frame registration and fusion method is used in Step 3 to align and fuse the infrared video frame and the microwave information fragment to generate a fused video frame:

[0074] The target detection boxes in the radar RD image and the infrared image are respectively denoted as and , by calculating a transformation matrix , so that the transformed anchor frames overlap as much as possible. The overall process is as follows Figure 4 As shown, define the transformation matrix , which is expressed in the form of:

[0075] ;

[0076] in, represents the scale factor, represents the rotation angle, and Represent the translation respectively. The transformation matrix is ​​optimized by minimizing the Euclidean distance d between the target anchor boxes. , by iteratively adjusting the transformation matrix The parameters in make the objective function reach the minimum value, and the objective function is as follows:

[0077] ;

[0078] in, Represents the weight factor of the i-th anchor box, which is used to adjust the contribution of different anchor boxes to the objective function; Represents the Euclidean distance between the transformed anchor box and the target anchor box; represents the regularization term; Represents the weight of the regularization term, which is used to balance the influence of the registration error and the regularization term;

[0079] Due to the unique nature of infrared images and their blurred and tiny target features, after comparing various excellent image fusion methods at this stage, it is found that complex fusion methods are not suitable for image fusion processing in such scenes, which often leads to image artifacts and target information loss. Adding a consistency strategy to the simple weighted fusion will achieve better performance. The fused features are expressed as , the calculation formula is:

[0080] ;

[0081] in, and They are the features extracted from radar RD images and infrared images respectively. It is to adjust the weight of the contribution of two features. In the process of feature fusion, in order to ensure the consistency of the fused features, a consistency test can be performed. Specifically, the difference between the features before and after fusion and the original features can be compared. If the difference exceeds a certain threshold, the weight needs to be readjusted or other processing needs to be performed to ensure the consistency of the fusion results.

[0082] As a preferred implementation, in Step 4, the local contrast enhancement technology is used to post-process the video frames fused in Step 3 to enhance the details and features in the image as follows:

[0083] The non-local mean filtering enhancement method is used to calculate similar blocks in the image to smooth the noise. First, the original image is taken as input ,in Represents the pixel coordinates in the image. For each pixel , calculate it with other pixels The weighted average of is determined by the similarity between the two. The formula for non-local mean filtering is:

[0084] ;

[0085] in, Indicates that the enhanced image has pixels The value at It's pixels and pixels The similarity weight between is determined by the distance between them and is defined as:

[0086] ;

[0087] in, and Pixels and pixels is the image patch centered, h is the smoothing parameter, is the normalization factor. After non-local means filtering, the noise is effectively removed, while the details and edges are enhanced.

[0088] As a preferred implementation, the step of encoding the fused video frame sequence into a video file using H.265 / HEVC in Step 5 is:

[0089] Install and configure a video encoder that supports H.265 / HEVC encoding, and convert the fused image sequence into a video file format. First, by analyzing the redundant information of the fused frames, dynamically optimize the encoding parameters, and adjust the size and shape of the encoding block to adapt to complex scenes. Secondly, use the complementarity of infrared and microwave information to improve motion estimation and enhance prediction accuracy. Finally, by optimizing entropy coding, give priority to encoding the target area, reduce the background information bit rate, and improve compression efficiency. The overall process is as follows: Figure 5 As shown, the core formula is:

[0090] ;

[0091] Where R represents the bit rate after encoding, Indicates the size of the encoding block, It indicates the accuracy of intra-block prediction. Finally, the encoded video file is checked and tested to ensure that the video quality and compression efficiency meet the requirements.

[0092] In summary, the infrared and microwave information video coding fusion technology based on adaptive registration can transform abstract point cloud data into visualized data, realize feature-level fusion according to the orientation of the target object, make full use of multimodal fusion to perceive the object coordinates, and improve the fusion effect through targeted preprocessing, fusion and post-processing strategies. It is suitable for target detection and positioning tasks in various complex environments.

[0093] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A method for infrared and microwave information video coding fusion based on adaptive registration, characterized in that: The following steps are involved: Step 1, collect information fragments of infrared and microwave imaging devices in the same scene to obtain infrared video fragments and microwave information fragments; Step 2, extracting video frames from the infrared video clips collected in Step 1 and performing preprocessing operations using lightweight convolution kernel sharpening, performing pulse compression on the collected microwave information clips and calculating the frequency vector to convert it into a radar RD map preprocessing operation; Step 3, using the adaptive anchor frame registration and fusion method to align and fuse the infrared video frame and microwave information fragments to generate a fused video frame; Step 4, using local contrast enhancement technology to post-process the fused video frames in Step 3 to enhance the details and features in the image; Step 5, use H.265 / HEVC to encode the fused video frame sequence into a video file; Step 2 specifically includes: Step 2.1-1, use Python-OpenCV to extract each frame of the infrared video clip, and save each frame image in memory for subsequent processing; Step 2.1-2, use the lightweight convolution kernel sharpening method to pre-operate the infrared video frame. Represents the image at coordinates The pixel value at Represents the convolution kernel In coordinates The value at , m and n are half of the number of rows and columns of the convolution kernel respectively, and the calculation formula is: ； ； The image is sharpened by assigning positive weights to the central pixel and negative weights to the surrounding pixels. Step 2.2-1. Use the matched filter method to compress the microwave information fragments. First, define some constants for calculation in MATLAB: sampling frequency fs, pulse repetition frequency PRF and speed of light c. Use the matched filter to compress the pulse to improve the radar resolution. The output of the matched filter is: ； in, is the received radar echo signal, is the impulse response of the matched filter. After pulse compression processing, the distance of the target is calculated by measuring the time difference between the transmitted signal and the received signal, as shown in the formula: ； Step 2.2-2, use the method based on calculating the frequency vector to convert the microwave information fragment into the radar RD map, enhance the visualization effect of the target, perform fast Fourier transform on the received radar signal, convert the time domain signal into the frequency domain signal, obtain the frequency component, and calculate the frequency vector and FFT formula as follows: ； ； The linspace function generates a linear space vector from -0.5×PRF to 0.5×PRF. PRF is the pulse repetition frequency, which determines the range of the Doppler frequency. is a time domain signal, is the frequency domain signal, N is the signal length; Step 3 specifically includes: The target detection boxes in the radar RD image and the infrared image are respectively denoted as and , by calculating a transformation matrix , so that after the transformation and Overlap, define the transformation matrix , which is expressed in the form of: ； in, represents the scale factor, represents the rotation angle, and Represent the translation amount respectively, and optimize the transformation matrix by minimizing the Euclidean distance d between the target anchor boxes. , by iteratively adjusting the transformation matrix The parameters in make the objective function reach the minimum value, and the objective function is as follows: ； in, Represents the weight factor of the i-th anchor box, which is used to adjust the contribution of different anchor boxes to the objective function; Represents the Euclidean distance between the transformed anchor box and the target anchor box; represents the regularization term; Represents the weight of the regularization term, which is used to balance the influence of the registration error and the regularization term; The fused features are expressed as , the calculation formula is: ； in, and They are the features extracted from radar RD images and infrared images respectively. It is to adjust the weight of the contribution of the two features.

2. The method for infrared and microwave information video coding fusion based on adaptive registration as claimed in claim 1, characterized in that: Step 4 specifically includes: The non-local mean filtering enhancement method is used to calculate similar blocks in the image to smooth the noise. First, the original image is taken as input ,in Represents the pixel coordinates in the image. For each pixel , calculate it with other pixels The weighted average of is determined by the similarity between the two. The formula for non-local mean filtering is: ； in, Indicates that the enhanced image has pixels The value at It's pixels and pixels The similarity weight between is determined by the distance between them and is defined as: ； in, and Pixels and pixels is the image patch centered, h is the smoothing parameter, is the normalization factor.

3. The method for infrared and microwave information video coding fusion based on adaptive registration as claimed in claim 1, characterized in that: Step 5 specifically includes: Install and configure a video encoder that supports H.265 / HEVC encoding, and convert the fused image sequence into a video file format. First, by analyzing the redundant information of the fused frames, dynamically optimize the encoding parameters and adjust the size and shape of the encoding block; second, use the complementarity of infrared and microwave information to improve motion estimation; finally, optimize entropy coding and give priority to encoding the target area. The core formula is: ； Where R represents the bit rate after encoding, Indicates the size of the encoding block, Indicates the accuracy of intra-block prediction; finally, the encoded video file is checked and tested.

Images