Monitoring system based on dual-light fusion video technology

Abstract

The invention relates to the technical field of monitoring, in particular to a monitoring system based on a dual-light fusion video technology, which is used for solving the problems that factory correction parameters are not suitable any more and imaging quality is affected due to the fact that the structure of a dual-light camera is changed in the long-time use process of dual-light fusion equipment. According to the invention, the operation effects of the visible light channel and the infrared light channel are synchronously acquired, and the acquired infrared light picture and the visible light picture are respectively subjected to visible light imaging quality judgment, infrared light fusion decision and infrared light regional contrast enhancement processing, so that the sample quality is improved; and then frame disassembly and picture fusion are carried out based on the time axis to obtain a dual-light fusion picture, so that clear imaging can be obtained in various different environments, the monitoring effect is improved, the fusion effect can be verified according to the imaging difference after the fusion imaging, and an early warning prompt is generated in time when the fusion effect has imaging deviation. Therefore, stable operation of equipment is ensured.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of monitoring, in particular to a monitoring system based on double light fusion video technology. BACKGROUND

[0002] Double light fusion is a dual-channel of "micro light + thermal imager", that is, a combination of visible light channel and infrared light channel. It simultaneously uses infrared and micro light technology to image at different wavelengths, synchronously detects the two-dimensional geometric space and one-dimensional spectral information of the target, and then uses certain image processing algorithms to analyze and process the multi-band image. The useful information in various channels is fully utilized to synthesize the image. The visible light channel can display the real-time dynamics of the target, which is equivalent to a camera. The infrared light channel can display the temperature measurement results in the form of a thermal image to show the temperature difference and temperature value on site. In this way, the drawbacks of single thermal imaging devices or personnel on site to take blurred images and the need to recheck the problem points on site are avoided, and the work is more convenient and efficient. At present, the double light fusion devices of visible light images and infrared thermal imaging images in the non-civilian industries such as security and unmanned aerial vehicles on the market correct the double light fusion algorithm before the devices are shipped, that is, the double light fusion correction of visible light images and infrared thermal imaging images is completed during production. However, during the subsequent use process, the structure of the visible light lens and the thermal imaging lens may be offset or the position of the double light camera itself may be offset with the increase of the use time, so that the factory correction parameters are no longer applicable, resulting in interference such as ghosting in the formed image, which affects the imaging accuracy. In view of the above technical problems, the present application provides a solution. SUMMARY

[0003] After fusion imaging, the fusion result and the single light channel image participating in fusion are compared respectively to obtain the imaging gap between the fusion result and the sample participating in fusion. The fusion effect is verified according to the imaging gap, and a warning reminder is generated in time when the imaging deviation occurs in the fusion effect, so as to ensure the stable operation of the device and solve the problem that the structure of the double light camera of the double light fusion device changes during long-term use, resulting in that the factory correction parameters are no longer applicable and affecting the imaging quality.

[0004] The purpose of the present application can be achieved by the following technical solutions: A monitoring system based on double light fusion video technology, comprising a visible light capture module, an infrared light capture module, a double light fusion module, an abnormal calibration module and a monitoring interface analysis module. The visible light capture module captures visible light pictures through a visible light camera to generate visible light pictures. The infrared light capture module captures infrared light images through an infrared camera, preprocesses the captured infrared light images using environmental parameters to improve the contrast of the infrared light images, and generates infrared images. The dual-light fusion module decomposes the visible light image and the infrared image into frames, and then fuses the decomposed images one by one through time series and algorithm to obtain the corrected image. After acquiring the corrected image, the monitoring interface analysis module imports the corrected image into the cloud for storage via the interface. The anomaly calibration module can acquire the corrected image and compare and analyze it with the infrared image and the visible light image respectively to obtain the edge information and image size information of the corrected image. The anomaly calibration module obtains the edge repair degree based on the edge information and the fusion symmetry degree based on the image size information. The anomaly calibration module performs a comprehensive analysis based on the degree of edge repair and the degree of fusion symmetry to obtain a fusion effect rating, and feeds the fusion effect rating back to the dual-light fusion module; After obtaining the fusion effect rating, the dual-light fusion module provides feedback and reminders.

[0005] In a preferred embodiment of the present invention, when the visible light capture module acquires a visible light signal, it collects the signal through a visible light camera, automatically analyzes the brightness of the acquired visible light image, and obtains a fusion signal or an output signal based on whether the visible light brightness reaches a preset setting.

[0006] In a preferred embodiment of the present invention, the method by which the visible light capture module performs image brightness analysis on a visible light image is as follows: The visible light capture module performs matrix sampling on the visible light image and performs grayscale conversion on each sampling point to obtain multiple single-channel grayscale images. The grayscale values ​​of the pixels in each single-channel grayscale image are averaged, and the average value is used as the brightness value of the single-channel grayscale image. Then, the brightness values ​​of each sample in the matrix sampling are averaged to obtain the overall brightness of the visible light image.

[0007] In a preferred embodiment of the present invention, the environmental parameter collected by the infrared light capture module is the ambient air temperature. After collecting the environmental parameter, the infrared light capture module records the collected ambient air temperature as the reference temperature. The infrared light capture module obtains the set amplitude threshold η and generates two sets of endpoint ratios through (1+η) and (1-η). The reference temperature is multiplied by the two sets of endpoint ratios respectively to obtain two sets of temperature endpoints. The temperature between the two sets of temperature endpoints is taken as the high contrast temperature, and the temperature outside the two sets of endpoint temperatures is taken as the normal temperature.

[0008] In a preferred embodiment of the present invention, when the infrared light capture module preprocesses the infrared image, it obtains the temperature values ​​of different regions in the infrared image through a pre-calibrated temperature mapping relationship, marks the regions with high contrast temperatures, linearly stretches the gray values ​​of the marked regions to increase the gray value range, and compresses the gray values ​​of those not belonging to the marked regions to reduce the gray value range. After changing the grayscale value, the infrared light capture module optimizes color mapping based on temperature, improves the contrast of areas with temperatures similar to the ambient air temperature, and enhances image clarity by utilizing high contrast. In areas with large temperature differences between the infrared light and the ambient air temperature, the image clarity is maintained even when the grayscale value range is compressed by utilizing the temperature difference of the infrared light itself.

[0009] In a preferred embodiment of the present invention, when the dual-light fusion module performs frame decomposition on the visible light image and the infrared image, it converts the video image into an independent single-frame static image and obtains the specific time information of the single-frame static image. Then, it fuses the single-frame static images of the infrared light image and the single-frame static images of the visible light image with the same time information. The dual-light fusion module merges the fused static images to obtain a corrected image, and then merges them into a video image based on time information.

[0010] In a preferred embodiment of the present invention, the method by which the dual-light fusion module merges static images is as follows: The visible light and infrared light images are subjected to perspective transformation based on camera parameters to eliminate image deviation. An attention mechanism is introduced to dynamically weight the infrared image, and features are extracted by an encoder. The fusion layer integrates the feature information to complete the fusion of the infrared and visible light images.

[0011] As a preferred embodiment of the present invention, the method for the abnormal calibration module to obtain the edge repair degree is as follows: the object edges of the corrected image, the visible light image, and the infrared light image are identified to obtain the number of edges in the three sets of images, and the number of edges is compared. If the number of edges in the corrected image is less than the number of edges in the visible light image or the infrared light image, the edge repair degree is determined to be insufficient; otherwise, the edge repair degree is determined to be up to standard. The method for the abnormal calibration module to obtain image frame information is as follows: extract feature points in the corrected image, and extract the same feature points through the visible light image and the infrared light image. Locate multiple feature points in the corresponding images to obtain feature point coordinates. Compare the feature point coordinates in different images and count the number of feature points with the same coordinates. If the proportion of the number of feature points with the same coordinates in the total number of feature points is greater than a set value, the image frame is determined to be normal; otherwise, the image frame is determined to be abnormal. The anomaly calibration module records image aberrations and normal image aberrations as the degree of fusion symmetry, and records insufficient repair and satisfactory repair as the degree of edge repair.

[0012] In a preferred embodiment of the present invention, the anomaly calibration module generates a fusion failure signal when it simultaneously detects both image format anomaly and insufficient repair level, generates a fusion normal signal when it simultaneously detects both image format anomaly and adequate repair level, and generates a fusion optimization signal otherwise.

[0013] Compared with the prior art, the beneficial effects of the present invention are: 1. In this invention, during the operation of the video surveillance system, the operating effects of the visible light channel and the infrared light channel are collected synchronously, and the collected infrared light images and visible light images are preprocessed separately to improve the sample quality. Frame decomposition and image fusion are performed based on the time axis to obtain a dual-light fused image, thereby obtaining clear imaging in various environments and improving the monitoring effect.

[0014] 2. In this invention, when processing visible light, the imaging quality of visible light is judged to make a scientific decision on the fusion of infrared light, thereby reducing the local computation load and imaging delay. Infrared light is processed in regions, and grayscale stretching is performed according to different temperature ranges to improve the contrast of the approximate temperature range. This ensures that each temperature range of infrared imaging can obtain clear imaging edges, which improves the sample quality during dual-light fusion and guarantees the individual imaging clarity of a single light channel.

[0015] 3. In this invention, after fusion imaging, the fusion result is compared with the single optical channel image participating in the fusion, thereby obtaining the imaging difference between the fusion result and the sample participating in the fusion. The fusion effect is verified based on the imaging difference, and when the fusion effect shows imaging deviation, an early warning reminder is generated in a timely manner, thereby ensuring the stable operation of the equipment. Attached Figure Description

[0016] To facilitate understanding by those skilled in the art, the present invention will be further described below with reference to the accompanying drawings.

[0017] Figure 1 This is a system block diagram of the present invention; Figure 2 This is a system flowchart of the present invention. Detailed Implementation

[0018] The technical solution of the present invention will be clearly and completely described below with reference to the embodiments. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0019] Example 1: Please refer to Figure 1 - Figure 2 As shown, a monitoring system based on dual-light fusion video technology includes a visible light capture module, an infrared light capture module, a dual-light fusion module, an anomaly calibration module, and a monitoring interface analysis module. The visible light capture module captures visible light images through a visible light camera to generate visible light images. When acquiring visible signals, the visible light capture module uses a visible light camera to collect data and automatically analyzes the brightness of the captured visible light image. The method used by the visible light capture module to analyze the brightness of the visible light image is as follows: The visible light capture module performs matrix sampling on the visible light image and performs grayscale conversion on each sampling point to obtain multiple single-channel grayscale images. The grayscale values ​​of the pixels in each single-channel grayscale image are averaged, and the average value is used as the brightness value of the single-channel grayscale image. Then, the brightness values ​​of each sample in the matrix sampling are averaged to obtain the overall brightness of the visible light image. The visible light capture module judges the brightness of the visible light image. If the brightness of the visible light image reaches the preset brightness, it generates an image output signal. If the brightness of the visible light image does not reach the preset brightness, it generates an image fusion signal. After the visible light capture module generates an image output signal, it indicates that the image brightness is sufficient and the image details can be observed intuitively and effectively. In this case, the image is directly output to the monitoring interface analysis module. After generating an image fusion signal, it indicates that the image brightness is insufficient and the image details cannot be observed intuitively and effectively. It is necessary to fuse the infrared light image to improve the image clarity and recognition ability. The infrared light capture module acquires infrared light images through an infrared camera, preprocesses the acquired infrared light images using environmental parameters to improve the contrast of the infrared light images, and generates infrared images. The environmental parameter acquired by the infrared light capture module is the ambient air temperature. After acquiring the environmental parameter, the infrared light capture module records the acquired ambient air temperature as the reference temperature. The infrared light capture module obtains the set amplitude threshold η and generates two sets of endpoint ratios through (1+η) and (1-η). The reference temperature is multiplied by the two sets of endpoint ratios respectively to obtain two sets of temperature endpoints. The temperature between the two sets of temperature endpoints is taken as the high contrast temperature, and the temperature outside the two sets of endpoint temperatures is taken as the normal temperature. When the infrared light capture module preprocesses the infrared image, it obtains the temperature values ​​of different areas in the infrared image through a pre-calibrated temperature mapping relationship, marks the areas with high contrast temperatures, linearly stretches the gray values ​​of the marked areas to increase the gray value range, and compresses the gray values ​​of areas that do not belong to the marked areas to reduce the gray value range. After changing the grayscale value, the infrared light capture module optimizes color mapping based on temperature, improves the contrast of areas with temperatures similar to the ambient air temperature, and enhances image clarity by utilizing high contrast. In areas with large temperature differences between the infrared light and the ambient air temperature, the image clarity is maintained even when the grayscale value range is compressed by utilizing the temperature difference of the infrared light itself.

[0020] The dual-light fusion module decomposes the visible light image and the infrared image into frames. When the dual-light fusion module decomposes the visible light image and the infrared image into frames, it converts the video image into an independent single-frame static image and obtains the specific time information of the single-frame static image. It then fuses the single-frame static images of the infrared image and the single-frame static images of the visible light image with the same time information to obtain the corrected image. The dual-light fusion module merges the fused static images to obtain a corrected image, and then merges them into a video image based on time information. The method by which the dual-light fusion module merges static images is as follows: The visible light and infrared light images are subjected to perspective transformation based on camera parameters to eliminate image deviation. An attention mechanism is introduced to dynamically weight the infrared image, and features are extracted by an encoder. The fusion layer integrates the feature information to complete the fusion of the infrared and visible light images.

[0021] After acquiring the corrected image, the monitoring interface analysis module imports the corrected image into the cloud for storage via the interface, making it convenient for managers to view or retrieve it.

[0022] Example 2: Please refer to Figure 1 - Figure 2 As shown, the anomaly calibration module can acquire the corrected image and compare and analyze it with the infrared image and the visible light image respectively to obtain the edge information and image size information of the corrected image. The anomaly calibration module obtains the edge repair degree based on the edge information and the fusion symmetry degree based on the image size information. The abnormal calibration module obtains the edge repair level by identifying the object edges in the corrected image, visible light image, and infrared light image, obtaining the number of edges in the three sets of images. The number of edges is represented by the line length of the edges. The edge numbers are compared. If the number of edges in the corrected image is less than the number of edges in the visible light image or the infrared light image, the edge repair level is determined to be insufficient. If the number of edges in the corrected image is greater than or equal to the number of edges in the visible light image and the infrared light image, the edge repair level is determined to be up to standard. The abnormal calibration module obtains image frame information by extracting feature points from the corrected image and extracting the same feature points from the visible light image and the infrared light image. Multiple feature points are located in the corresponding images to obtain the feature point coordinates (Xi, Yi), where i is the feature point number. The feature point coordinates in different images are compared, and the number of feature points with the same coordinates is counted. If the proportion of the number of feature points with the same coordinates in the total number of feature points is greater than a set value, the image frame is determined to be normal. If the proportion of the number of feature points with the same coordinates in the total number of feature points is not greater than a set value, the image frame is determined to be abnormal. The anomaly calibration module records image aberration and normal image aberration as the degree of blending symmetry, and records insufficient repair and satisfactory repair as the degree of edge repair. The anomaly calibration module performs a comprehensive analysis based on the degree of edge repair and the degree of fusion symmetry. When the anomaly calibration module simultaneously detects both image anomaly and insufficient repair, it generates a fusion failure signal. When it simultaneously detects both image anomaly and adequate repair, it generates a fusion normal signal. Otherwise, it generates a fusion optimization signal. The fusion failure signal, fusion normal signal, or fusion optimization signal are used as the fusion effect rating result, and the fusion effect rating is fed back to the dual-light fusion module. After obtaining the fusion effect rating, the dual-light fusion module provides feedback and reminders, enabling managers to perform maintenance and repairs on the equipment after receiving fusion optimization signals or fusion failure signals. This allows them to adjust the dual-light fusion parameters of the equipment and ensure the imaging quality of the dual-light fusion monitoring system.

[0023] The preferred embodiments of the present invention disclosed above are merely illustrative of the invention. These preferred embodiments do not exhaustively describe all details, nor do they limit the invention to any specific implementation. Clearly, many modifications and variations can be made based on the content of this specification. This specification selects and specifically describes these embodiments to better explain the principles and practical applications of the invention, thereby enabling those skilled in the art to better understand and utilize the invention. The invention is limited only by the claims and their full scope and equivalents.

Claims

1. A monitoring system based on dual light fusion video technology, characterized in that, Including visible light capture module, infrared light capture module, dual light fusion module, abnormal calibration module and monitoring interface analysis module, the visible light capture module captures visible light picture through visible light camera, generates visible light picture; The infrared light capture module collects infrared light picture through infrared camera, and pre-processes the collected infrared light picture through environmental parameters, improves the contrast of the infrared light picture, and generates an infrared picture; The dual light fusion module frame decomposes the visible light picture and the infrared picture, and one by one corresponds the frame decomposed picture through time sequence, and fuses through algorithm to obtain a corrected picture; The monitoring interface analysis module stores the corrected picture in the cloud after obtaining the corrected picture through the interface; The abnormal calibration module can obtain the corrected picture, and compare and analyze the corrected picture with the infrared light picture and the visible light picture respectively, obtain the edge information and the frame information of the corrected picture, and the abnormal calibration module obtains the edge repair degree according to the edge information and the fusion symmetry degree according to the frame information; The abnormal calibration module comprehensively analyzes the edge repair degree and the fusion symmetry degree to obtain a fusion effect rating, and feeds back the fusion effect rating to the dual light fusion module; The dual light fusion module feeds back after obtaining the fusion effect rating.

2. The monitoring system based on dual light fusion video technology according to claim 1, characterized in that, When the visible light capture module obtains visible signals, it collects through the visible light camera, and automatically analyzes the brightness of the collected visible light picture, and according to whether the visible light brightness reaches the preset setting, obtains a fusion signal or an output signal according to the judgment result.

3. The monitoring system based on dual light fusion video technology according to claim 2, characterized in that, The method for the visible light capture module to analyze the brightness of the visible light picture is: The visible light capture module samples the matrix of the visible light picture, converts the gray scale of each sampling point, obtains a plurality of single-channel gray scale images, averages the gray scale values of the pixel points in each single-channel gray scale image, takes the average value as the brightness value of the single-channel gray scale image, and then averages the brightness values of each sample in the matrix sampling to obtain the overall picture brightness of the visible light picture.

4. The monitoring system based on dual light fusion video technology according to claim 1, characterized in that, The environmental parameters collected by the infrared light capture module are environmental air temperature, the infrared light capture module records the collected environmental air temperature as a reference temperature after collecting the environmental parameters, the infrared light capture module obtains a set amplitude threshold η, generates two sets of end point ratios through (1+η) and (1-η), multiplies the reference temperature by the two sets of end point ratios to obtain two sets of temperature end points, and takes the temperature between the two sets of temperature end points as a high-contrast temperature, and takes the temperature outside the two sets of end point temperatures as a normal temperature.

5. The monitoring system based on dual light fusion video technology according to claim 1, characterized in that, When the infrared light capture module pre-processes the infrared picture, it obtains the temperature values of different regions in the infrared picture through the pre-calibrated temperature mapping relationship, marks the regions with temperature values belonging to high-contrast temperature, linearly stretches the gray scale values of the marked regions to increase the gray scale value interval range, and compresses the gray scale values not belonging to the marked regions to reduce the gray scale value interval range; The infrared light capture module optimizes the color mapping according to the temperature after changing the gray scale value.

6. The monitoring system based on dual light fusion video technology according to claim 1, characterized in that, The dual-light fusion module converts the video picture into independent single-frame static images when frame decomposing the visible light picture and the infrared picture, and obtains specific time information of the single-frame static images, and fuses the single-frame static images of the infrared light picture and the single-frame static images of the visible light picture corresponding to the same time information. The dual-light fusion module merges the fused static images to obtain a corrected picture, and then combines the corrected picture into a video picture according to the time information.

7. The monitoring system based on dual light fusion video technology according to claim 1, characterized in that, The method for merging the static images by the dual-light fusion module is: The visible light picture and the infrared light picture are subjected to perspective transformation based on camera parameters to eliminate picture deviation, an attention mechanism is introduced, the infrared picture is dynamically weighted, features are extracted through an encoder, and a fusion layer integrates the feature information to complete fusion of the infrared picture and the visible light picture.

8. The monitoring system based on dual light fusion video technology according to claim 1, characterized in that, The method for obtaining the edge repair degree by the abnormal calibration module is: identifying object edges of the corrected picture, the visible light picture and the infrared light picture to obtain the number of edges in the three pictures, and comparing the number of edges, if the number of edges in the corrected picture is less than the number of edges in the visible light picture or the infrared light picture, it is determined that the edge repair degree is insufficient, otherwise it is determined that the edge repair degree meets the standard. The method for obtaining the frame information by the abnormal calibration module is: extracting feature points in the corrected picture, and extracting the same feature points in the visible light picture and the infrared light picture, positioning the multiple feature points in the corresponding pictures to obtain feature point coordinates, and comparing the feature point coordinates in different pictures, counting the number of feature points with the same coordinates, if the proportion of the number of feature points with the same coordinates in the total number of feature points is greater than a set value, it is determined that the frame is normal, otherwise it is determined that the frame is abnormal. The abnormal calibration module records the frame abnormality and the frame normality as the fusion symmetry degree, and records the insufficient repair degree and the repair degree meeting the standard as the edge repair degree.

9. The monitoring system based on dual light fusion video technology according to claim 1, characterized in that, The abnormal calibration module generates a fusion failure signal when the frame abnormality and the insufficient repair degree are obtained at the same time, generates a fusion normal signal when the frame abnormality and the repair degree meeting the standard are obtained at the same time, and otherwise generates a fusion optimization signal.

Images