Dual-spectrum image transmission pretreatment method and system

Abstract

The invention relates to the technical field of image processing and transmission, in particular to a dual-spectrum image transmission pretreatment method and system. The dual-spectrum image transmission pre-processing method comprises the following steps: performing high-speed scanning shooting by adopting a galvanometer in cooperation with an infrared and visible light camera rotating by 360 degrees to obtain a dual-spectrum image frame; carrying out rapid feature extraction and matching on the shot dual-spectrum image frames by using ResNet and a batch FLANN search method according to an image azimuth angle and an equipment rotating speed, and carrying out image alignment, fusion and smoothing processing to obtain a panoramic spliced image; and carrying out H.264 coding compression on a private protocol message formed by the panoramic spliced image and the information of the equipment rotating speed, the equipment pitch angle and the image azimuth angle so as to carry out wireless transmission. According to the method, the defects that an existing radar cannot acquire image information, the false alarm rate is high, 360-degree panoramic information cannot be acquired in real time, and a missing detection area exists are overcome.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing and transmission, and in particular to a dual-spectrum image transmission pre-processing method and system. Background Art

[0002] In field border surveillance, due to the vastness of the border and the complex terrain, some areas are even difficult to reach manually. Therefore, traditional manual patrols have significant limitations, often relying on radar and infrared photoelectric PTZ equipment for 24 / 7 monitoring. However, radar equipment cannot obtain image information and is significantly affected by weather, resulting in a high false alarm rate and requiring manual on-site secondary confirmation. Infrared photoelectric PTZ equipment also cannot obtain 360-degree panoramic information in real time, resulting in missed monitoring areas. Therefore, there is an urgent need to develop a dual-spectrum image transmission pre-processing method and system to achieve remote 360-degree panoramic, all-weather, visual, real-time monitoring. Summary of the Invention

[0003] The purpose of the present invention is to solve at least one of the technical problems existing in the prior art and to provide a dual-spectrum image transmission pre-processing method and system.

[0004] To achieve the above object, the present invention adopts the following technical solution: a dual-spectrum image transmission pre-processing method, comprising the following steps:

[0005] Step 1: Use a galvanometer to perform high-speed scanning and shooting with a 360° rotating infrared and visible light camera to obtain a dual-spectrum image frame;

[0006] Step 2: The captured dual-spectral image frames are subjected to rapid feature extraction and matching using ResNet and batch FLANN search methods according to the image azimuth and device rotation speed, and a panoramic mosaic image is obtained through image alignment, fusion, and smoothing.

[0007] Step 3: The panoramic mosaic image and the private protocol message consisting of the device rotation speed, device pitch angle, and image azimuth angle information are compressed together with H.264 encoding for wireless transmission.

[0008] Furthermore, in said step 2, it specifically includes:

[0009] The input image frames are pre-processed with distortion correction and illumination equalization, and then projected into cylindrical space. The precise position of each image in the panorama is determined based on its azimuth information. The overlapping areas are stitched using a weighted fusion method with fade-in and fade-out, and finally exposure compensation and seam optimization are performed.

[0010] Furthermore, in said step 2, it specifically includes:

[0011] Step 201, data acquisition: 360° rotating infrared and visible light cameras perform high-speed scanning and shooting via galvanometers, capturing dual-spectral image frames while simultaneously acquiring information on device rotation speed, device pitch angle, and image azimuth angle. The dual-spectral image frames cover a 360° field of view of the entire scene, and adjacent dual-spectral image frames are guaranteed to have >30% overlap.

[0012] Step 202: Feature extraction and feature matching:

[0013] Feature extraction uses the ResNet model in the convolutional neural network, which is pre-trained on a dataset of a preset magnitude. The pre-trained ResNet model calls the evaluation mode and uses the output of the 4th and 5th blocks as feature maps.

[0014] The feature matching steps specifically include:

[0015] Identify the adjacent relationship between dual-spectrum image frames by image azimuth angle, and identify the overlap rate between adjacent image frames by device rotation speed;

[0016] The computational efficiency of feature matching is improved through the batch FLANN search method. The specific formula is as follows:

[0017] {\text{NN}_1, \text{NN}_2, \dots, \text{NN}_M} =\text{FLANN.search}({\mathbf{f}'_1, \mathbf{f}'_2, \dots,\mathbf{f}'_M},k);

[0018] Where {\mathbf{f}'_1,\mathbf{f}'_2,\dots,\mathbf{f}'_M} is a set of M query vectors; k is the number of nearest neighbors to be returned for each query vector; the \text{FLANN.search} method returns the set of k nearest neighbor indices {\text{NN}_1,\text{NN}_2,\dots,\text{NN}_M} for each query vector;

[0019] Step 203: Image alignment and fusion:

[0020] The image alignment process first calculates the homography matrix, and then performs perspective transformation on the image to align it with the target image. The homography matrix calculation process uses the PROSAC method, selecting sample points with higher scores in the feature matching stage.

[0021] The fusion process can adopt weighted average fusion, and after fusion, seam optimization and splicing area smoothing operations are performed; among them, the seam optimization adopts the dynamic programming method, and the splicing area smoothing adopts the fast Gaussian bilateral filtering method.

[0022] Furthermore, in said step 3, it specifically includes:

[0023] The acquired dual-spectrum image frame is in BGR24 format;

[0024] Perform pixel format conversion through FFmpeg's SwsContext to convert the BGR24 format to the YUV420P format that complies with the video coding standard;

[0025] The YUV420P format divides the image into a luminance plane and two chrominance planes, where the sampling rate of the chrominance plane is 1 / 4 of the luminance plane;

[0026] The converted YUV420P frame is assigned an incremental display timestamp and then fed into the H.264 / AVC encoder for compression encoding;

[0027] The encoder encodes the image data according to preset parameters. During the encoding process, it uses intra-frame prediction and inter-frame prediction technologies to achieve data compression and output an H.264 raw bitstream.

[0028] The encoded data is output in the form of AVPacket. Each packet contains the encoded frame data and its corresponding display timestamp, decoding timestamp and duration stamp. These timestamps are converted to adapt to the time scale of the output stream through time base conversion.

[0029] The processed data is encapsulated into the FLV container format, and the necessary container metadata and header information are added by the FLV encapsulator;

[0030] The encapsulated data is pushed to the nginx streaming server via the RTMP protocol.

[0031] The present invention also provides a dual-spectrum image transmission pre-processing system, comprising:

[0032] The image acquisition module is configured to use a galvanometer with a 360° rotating infrared and visible light camera to perform high-speed scanning and shooting to obtain a dual-spectrum image frame;

[0033] The image stitching module is configured to perform fast feature extraction and matching on the captured dual-spectral image frames based on the image azimuth and device rotation speed using the ResNet and batch FLANN search methods, respectively, and obtain a panoramic stitching image through image alignment, fusion, and smoothing;

[0034] The image transmission module is configured to perform H.264 encoding compression on the private protocol message consisting of the panoramic mosaic image and the information of the device rotation speed, the device pitch angle, and the image azimuth angle for wireless transmission.

[0035] Furthermore, the image stitching module is specifically configured as follows:

[0036] The input image frames are pre-processed with distortion correction and illumination equalization, and then projected into cylindrical space. The precise position of each image in the panorama is determined based on its azimuth information. The overlapping areas are stitched using a weighted fusion method with fade-in and fade-out, and finally exposure compensation and seam optimization are performed.

[0037] Furthermore, the image transmission module is specifically configured as follows:

[0038] The acquired dual-spectrum image frame is in BGR24 format;

[0039] Perform pixel format conversion through FFmpeg's SwsContext to convert the BGR24 format to the YUV420P format that complies with the video coding standard;

[0040] The YUV420P format divides the image into a luminance plane and two chrominance planes, where the sampling rate of the chrominance plane is 1 / 4 of the luminance plane;

[0041] The converted YUV420P frame is assigned an incremental display timestamp and then fed into the H.264 / AVC encoder for compression encoding;

[0042] The encoder encodes the image data according to preset parameters. During the encoding process, it uses intra-frame prediction and inter-frame prediction technologies to achieve data compression and output an H.264 raw bitstream.

[0043] The encoded data is output in the form of AVPacket. Each packet contains the encoded frame data and its corresponding display timestamp, decoding timestamp and duration stamp. These timestamps are converted to adapt to the time scale of the output stream through time base conversion.

[0044] The processed data is encapsulated into the FLV container format, and the necessary container metadata and header information are added by the FLV encapsulator;

[0045] The encapsulated data is pushed to the nginx streaming server via the RTMP protocol.

[0046] From the above description of the present invention, it can be seen that, compared with the prior art, the dual-spectrum image transmission pre-processing method of the present invention has at least one of the following beneficial effects:

[0047] 1. This invention uses a dual-spectral camera to capture image frames, encodes and compresses them, transmits the video streams remotely and wirelessly, and finally decodes the streams to create a 360-degree panoramic image. This achieves remote, all-weather, 360-degree panoramic, real-time monitoring. This invention comprehensively considers quality and real-time requirements, combining the feature extraction and data generalization capabilities of convolutional neural networks with the computational efficiency of traditional image processing to achieve the entire image stitching process.

[0048] 2. The present invention overcomes the defects of existing radars, such as the inability to obtain image information, the high false alarm rate, the inability to obtain 360-degree panoramic information in real time, and the existence of missed detection areas. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 The figure is a flowchart of the steps of a dual-spectrum image transmission pre-processing method in a preferred embodiment of the present invention. DETAILED DESCRIPTION

[0050] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making any creative efforts are within the scope of protection of the present invention.

[0051] Reference Figure 1 As shown, a preferred embodiment of the present invention, a dual-spectrum image transmission pre-processing method, includes the following steps:

[0052] Step 1: Use a galvanometer to perform high-speed scanning and shooting with a 360° rotating infrared and visible light camera to obtain a dual-spectrum image frame;

[0053] Step 2: The captured dual-spectral image frames are subjected to rapid feature extraction and matching using ResNet and batch FLANN search methods according to the image azimuth and device rotation speed, and a panoramic mosaic image is obtained through image alignment, fusion, and smoothing.

[0054] Step 3: The panoramic mosaic image and the private protocol message consisting of the device rotation speed, device pitch angle, and image azimuth angle information are compressed together with H.264 encoding for wireless transmission.

[0055] The present invention uses a dual-spectral camera to capture image frames, encodes and compresses them, transmits video streams remotely and wirelessly, and finally decodes the video streams and stitches them into a 360-degree panoramic image, achieving remote, all-weather, visual, real-time monitoring. The present invention comprehensively considers quality and real-time requirements, combining the feature extraction and data generalization capabilities of convolutional neural networks with the high computational efficiency of traditional image processing to achieve the entire image stitching process. This invention overcomes the shortcomings of existing radars, which are unable to obtain image information, have a high false alarm rate, are unable to obtain 360-degree panoramic information in real time, and often miss detection areas.

[0056] As a preferred embodiment of the present invention, it may also have the following additional technical features:

[0057] In this embodiment, step 2 specifically includes:

[0058] The input image frames are pre-processed with distortion correction and illumination equalization, and then projected into cylindrical space. The precise position of each image in the panorama is determined based on its azimuth information. The overlapping areas are stitched using a weighted fusion method with fade-in and fade-out, and finally exposure compensation and seam optimization are performed.

[0059] In this embodiment, step 2 specifically includes:

[0060] Step 201, data acquisition: 360° rotating infrared and visible light cameras perform high-speed scanning and shooting via galvanometers, capturing dual-spectral image frames while simultaneously acquiring information on device rotation speed, device pitch angle, and image azimuth angle. The dual-spectral image frames cover a 360° field of view of the entire scene, and adjacent dual-spectral image frames are guaranteed to have >30% overlap.

[0061] Step 202: Feature extraction and feature matching:

[0062] Feature extraction uses the ResNet model in the convolutional neural network, which is pre-trained on a dataset of a preset magnitude. The pre-trained ResNet model calls the evaluation mode and uses the output of the 4th and 5th blocks as feature maps.

[0063] The feature matching steps specifically include:

[0064] Identify the adjacent relationship between dual-spectrum image frames by image azimuth angle, and identify the overlap rate between adjacent image frames by device rotation speed;

[0065] The computational efficiency of feature matching is improved through the batch FLANN search method. The specific formula is as follows:

[0066] {\text{NN}_1, \text{NN}_2, \dots, \text{NN}_M} =\text{FLANN.search}({\mathbf{f}'_1, \mathbf{f}'_2, \dots,\mathbf{f}'_M},k);

[0067] Where {\mathbf{f}'_1,\mathbf{f}'_2,\dots,\mathbf{f}'_M} is a set of M query vectors; k is the number of nearest neighbors to be returned for each query vector; the \text{FLANN.search} method returns the set of k nearest neighbor indices {\text{NN}_1,\text{NN}_2,\dots,\text{NN}_M} for each query vector;

[0068] Step 203: Image alignment and fusion:

[0069] The image alignment process first calculates the homography matrix, and then performs perspective transformation on the image to align it with the target image. The homography matrix calculation process uses the PROSAC method, selecting sample points with higher scores in the feature matching stage.

[0070] The fusion process can adopt weighted average fusion, and after fusion, seam optimization and splicing area smoothing operations are performed; among them, the seam optimization adopts the dynamic programming method, and the splicing area smoothing adopts the fast Gaussian bilateral filtering method.

[0071] In this embodiment, step 3 specifically includes:

[0072] The acquired dual-spectrum image frame is in BGR24 format;

[0073] Perform pixel format conversion through FFmpeg's SwsContext to convert the BGR24 format to the YUV420P format that complies with the video coding standard;

[0074] The YUV420P format divides the image into a luminance plane and two chrominance planes, where the sampling rate of the chrominance plane is 1 / 4 of the luminance plane;

[0075] The converted YUV420P frame is assigned an incremental display timestamp and then fed into the H.264 / AVC encoder for compression encoding;

[0076] The encoder encodes the image data according to preset parameters. During the encoding process, it uses intra-frame prediction and inter-frame prediction technologies to achieve data compression and output an H.264 raw bitstream.

[0077] The encoded data is output in the form of AVPacket. Each packet contains the encoded frame data and its corresponding display timestamp, decoding timestamp and duration stamp. These timestamps are converted to adapt to the time scale of the output stream through time base conversion.

[0078] The processed data is encapsulated into the FLV container format, and the necessary container metadata and header information are added by the FLV encapsulator;

[0079] The encapsulated data is pushed to the nginx streaming server via the RTMP protocol.

[0080] The present invention also provides a dual-spectrum image transmission pre-processing system, comprising:

[0081] The image acquisition module is configured to use a galvanometer with a 360° rotating infrared and visible light camera to perform high-speed scanning and shooting to obtain a dual-spectrum image frame;

[0082] The image stitching module is configured to perform fast feature extraction and matching on the captured dual-spectral image frames based on the image azimuth and device rotation speed using the ResNet and batch FLANN search methods, respectively, and obtain a panoramic stitching image through image alignment, fusion, and smoothing;

[0083] The image transmission module is configured to perform H.264 encoding compression on the private protocol message consisting of the panoramic mosaic image and the information of the device rotation speed, the device pitch angle, and the image azimuth angle for wireless transmission.

[0084] In this embodiment, the image stitching module is specifically configured as follows:

[0085] The input image frames are pre-processed with distortion correction and illumination equalization, and then projected into cylindrical space. The precise position of each image in the panorama is determined based on its azimuth information. The overlapping areas are stitched using a weighted fusion method with fade-in and fade-out, and finally exposure compensation and seam optimization are performed.

[0086] In this embodiment, the image transmission module is specifically configured as follows:

[0087] The acquired dual-spectrum image frame is in BGR24 format;

[0088] Perform pixel format conversion through FFmpeg's SwsContext to convert the BGR24 format to the YUV420P format that complies with the video coding standard;

[0089] The YUV420P format divides the image into a luminance plane and two chrominance planes, where the sampling rate of the chrominance plane is 1 / 4 of the luminance plane;

[0090] The converted YUV420P frame is assigned an incremental display timestamp and then fed into the H.264 / AVC encoder for compression encoding;

[0091] The encoder encodes the image data according to preset parameters. During the encoding process, it uses intra-frame prediction and inter-frame prediction technologies to achieve data compression and output an H.264 raw bitstream.

[0092] The encoded data is output in the form of AVPacket. Each packet contains the encoded frame data and its corresponding display timestamp, decoding timestamp and duration stamp. These timestamps are converted to adapt to the time scale of the output stream through time base conversion.

[0093] The processed data is encapsulated into the FLV container format, and the necessary container metadata and header information are added by the FLV encapsulator;

[0094] The encapsulated data is pushed to the nginx streaming server via the RTMP protocol.

[0095] The above description is only a preferred specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes based on the technical solutions and improved concepts of the present invention within the technical scope disclosed by the present invention, and these changes should be covered by the scope of protection of the present invention.

Claims

1. A dual-spectrum image transmission pre-processing method, characterized in that: The following steps are involved: Step 1: Use a galvanometer to perform high-speed scanning and shooting with a 360° rotating infrared and visible light camera to obtain a dual-spectrum image frame; Step 2: The captured dual-spectral image frames are subjected to rapid feature extraction and matching using ResNet and batch FLANN search methods according to the image azimuth and device rotation speed, and a panoramic mosaic image is obtained through image alignment, fusion, and smoothing. Step 3: The panoramic mosaic image and the private protocol message consisting of the device rotation speed, device pitch angle, and image azimuth angle information are compressed together with H.264 encoding for wireless transmission.

2. A dual-spectrum image transmission pre-processing method according to claim 1, characterized in that: In the step 2, it specifically includes: The input image frames are pre-processed with distortion correction and illumination equalization, and then projected into cylindrical space. The precise position of each image in the panorama is determined based on its azimuth information. The overlapping areas are stitched using a weighted fusion method with fade-in and fade-out, and finally exposure compensation and seam optimization are performed.

3. The method for pre-processing a dual-spectrum image transmission according to claim 1, characterized in that: In the step 2, it specifically includes: Step 201, data acquisition: 360° rotating infrared and visible light cameras perform high-speed scanning and shooting via galvanometers, capturing dual-spectral image frames while simultaneously acquiring information on device rotation speed, device pitch angle, and image azimuth angle. The dual-spectral image frames cover a 360° field of view of the entire scene, and adjacent dual-spectral image frames are guaranteed to have >30% overlap. Step 202: Feature extraction and feature matching: Feature extraction uses the ResNet model in the convolutional neural network, which is pre-trained on a dataset of a preset magnitude. The pre-trained ResNet model calls the evaluation mode and uses the output of the 4th and 5th blocks as feature maps. The feature matching steps specifically include: Identify the adjacent relationship between dual-spectrum image frames by image azimuth angle, and identify the overlap rate between adjacent image frames by device rotation speed; The computational efficiency of feature matching is improved through the batch FLANN search method. The specific formula is as follows: {\text{NN}_1,\text{NN}_2,\dots,\text{NN}_M}=\text{FLANN.search}({\mathbf{f}'_1,\mathbf{f}'_2,\dots,\mathbf{f}'_M},k); Where {\mathbf{f}'_1,\mathbf{f}'_2,\dots,\mathbf{f}'_M} is a set of M query vectors; k is the number of nearest neighbors to be returned for each query vector; the \text{FLANN.search} method returns the set of k nearest neighbor indices {\text{NN}_1,\text{NN}_2,\dots,\text{NN}_M} for each query vector; Step 203: Image alignment and fusion: The image alignment process first calculates the homography matrix, and then performs perspective transformation on the image to align it with the target image. The homography matrix calculation process uses the PROSAC method, selecting sample points with higher scores in the feature matching stage. The fusion process can adopt weighted average fusion, and after fusion, seam optimization and splicing area smoothing operations are performed; among them, the seam optimization adopts the dynamic programming method, and the splicing area smoothing adopts the fast Gaussian bilateral filtering method.

4. A dual-spectrum image transmission pre-processing method according to claim 1, characterized in that: In the step 3, it specifically includes: The acquired dual-spectrum image frame is in BGR24 format; Perform pixel format conversion through FFmpeg's SwsContext to convert the BGR24 format to the YUV420P format that complies with the video coding standard; The YUV420P format divides the image into a luminance plane and two chrominance planes, where the sampling rate of the chrominance plane is 1 / 4 of the luminance plane; The converted YUV420P frame is assigned an incremental display timestamp and then fed into the H.264 / AVC encoder for compression encoding; The encoder encodes the image data according to preset parameters. During the encoding process, it uses intra-frame prediction and inter-frame prediction technologies to achieve data compression and output an H.264 raw bitstream. The encoded data is output in the form of AVPacket. Each packet contains the encoded frame data and its corresponding display timestamp, decoding timestamp and duration stamp. These timestamps are converted to adapt to the time scale of the output stream through time base conversion. The processed data is encapsulated into the FLV container format, and the necessary container metadata and header information are added by the FLV encapsulator; The encapsulated data is pushed to the nginx streaming server via the RTMP protocol.

5. A dual-spectrum image transmission pre-processing system, characterized in that: include: The image acquisition module is configured to use a galvanometer with a 360° rotating infrared and visible light camera to perform high-speed scanning and shooting to obtain a dual-spectrum image frame; The image stitching module is configured to perform fast feature extraction and matching on the captured dual-spectral image frames based on the image azimuth and device rotation speed using the ResNet and batch FLANN search methods, respectively, and obtain a panoramic stitching image through image alignment, fusion, and smoothing; The image transmission module is configured to perform H.264 encoding compression on the private protocol message consisting of the panoramic mosaic image and the information of the device rotation speed, the device pitch angle, and the image azimuth angle for wireless transmission.

6. A dual-spectrum image transmission pre-processing system according to claim 5, characterized in that: The image stitching module is specifically configured as follows: The input image frames are pre-processed with distortion correction and illumination equalization, and then projected into cylindrical space. The precise position of each image in the panorama is determined based on its azimuth information. The overlapping areas are stitched using a weighted fusion method with fade-in and fade-out, and finally exposure compensation and seam optimization are performed.

7. The dual-spectrum image transmission pre-processing system according to claim 5, characterized in that: The image transmission module is specifically configured as follows: The acquired dual-spectrum image frame is in BGR24 format; Perform pixel format conversion through FFmpeg's SwsContext to convert the BGR24 format to the YUV420P format that complies with the video coding standard; The YUV420P format divides the image into a luminance plane and two chrominance planes, where the sampling rate of the chrominance plane is 1 / 4 of the luminance plane; The converted YUV420P frame is assigned an incremental display timestamp and then fed into the H.264 / AVC encoder for compression encoding; The encoder encodes the image data according to preset parameters. During the encoding process, it uses intra-frame prediction and inter-frame prediction technologies to achieve data compression and output an H.264 raw bitstream. The encoded data is output in the form of AVPacket. Each packet contains the encoded frame data and its corresponding display timestamp, decoding timestamp and duration stamp. These timestamps are converted to adapt to the time scale of the output stream through time base conversion. The processed data is encapsulated into the FLV container format, and the necessary container metadata and header information are added by the FLV encapsulator; The encapsulated data is pushed to the nginx streaming server via the RTMP protocol.

Images