Binocular visible light and infrared image fusion method and device and computer equipment

Abstract

The invention relates to a binocular visible light and infrared image fusion method and device and computer equipment, and the method comprises the steps: reading two paths of visible light image data when the real-time cache line number of each path of visible light image data collected by a binocular sensor reaches a preset line number; the preset line number is determined by a lens distortion parameter of the binocular sensor and a preset correction algorithm parameter; reading infrared image data corresponding to the visible light image data, and respectively correcting each path of visible light image data and infrared image data; and splicing the corrected two paths of visible light image data in real time, and performing real-time fusion based on a splicing result and the corrected infrared image data to obtain a target image. According to the invention, the problems that the output delay of the fused image is high and the real-time output cannot be ensured are solved, the output delay of the fused image is effectively reduced, and the real-time output is ensured.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a method, device and computer equipment for fusing binocular visible light and infrared images. Background Art

[0002] Visible light and infrared image fusion technology is widely used in various fields, such as security monitoring, intelligent driving, and industrial inspection. However, existing fusion methods typically use C language algorithms for data processing. Due to the large amount of computational data, the output results are delayed and cannot ensure the real-time output of the fused image.

[0003] There is currently no effective solution to the problem that the fused image output has high delay and cannot ensure real-time output in related technologies. Summary of the Invention

[0004] In this embodiment, a binocular visible light and infrared image fusion method, apparatus, and computer device are provided to solve the problem in related technologies of high output delay of fused images and inability to ensure real-time output.

[0005] First, in this embodiment, a method for fusing binocular visible light and infrared images is provided, including:

[0006] When the number of real-time cache lines of each channel of first visible light image data collected by the binocular sensor reaches a preset number of lines, reading the two channels of first visible light image data; the preset number of lines is determined by the lens distortion parameters of the binocular sensor and the preset correction algorithm parameters;

[0007] reading first infrared image data corresponding to the first visible light image data, and respectively correcting each channel of the first visible light image data and the first infrared image data to obtain two channels of second visible light image data and second infrared image data;

[0008] The two channels of the second visible light image data are spliced ​​in real time; and the splicing result and the second infrared image data are fused in real time to obtain a target image.

[0009] In some embodiments, the method further comprises:

[0010] Obtaining the lens distortion parameters of the binocular sensor and the preset correction algorithm parameters; wherein the lens distortion parameters include the number of lens distortion lines of the binocular sensor, and the preset correction algorithm parameters include the interval of the number of original image lines required for correction;

[0011] The preset number of rows is determined according to the number of lens distortion rows and the interval of the number of original image rows required for correction.

[0012] In some embodiments, the correcting each channel of the first visible light image data and the first infrared image data to obtain two channels of second visible light image data and second infrared image data includes:

[0013] Correcting each channel of the first visible light image data based on a first lookup table corresponding to the first visible light image data to obtain two channels of the second visible light image data;

[0014] The first infrared image data is corrected based on a second lookup table corresponding to the first infrared image data to obtain the second infrared image data.

[0015] In some embodiments, when the number of real-time cache lines of each channel of first visible light image data collected by the binocular sensor reaches a preset number of lines, before reading the two channels of first visible light image data, the method further includes:

[0016] Obtaining a first calibration file of the binocular sensor and a second calibration file of the first infrared image data acquisition device;

[0017] Based on the first calibration file, generating the corresponding first lookup table;

[0018] Based on the second calibration file, a corresponding second lookup table is generated.

[0019] In some embodiments, after reading the first infrared image data corresponding to the first visible light image data, the method further includes:

[0020] The two channels of the first visible light image data and the first infrared image data are synchronously processed; the synchronous processing includes image frame rate alignment and image data alignment.

[0021] In some embodiments, the step of performing real-time fusion based on the stitching result and the second infrared image data to obtain the target image includes:

[0022] Analyzing the stitching result and the second infrared image data by a fusion algorithm to obtain corresponding fusion coordinate information;

[0023] Based on the fusion coordinate information, the stitching result and the second infrared image data are fused in real time to obtain the target image; wherein the fusion process adopts a pipeline processing architecture.

[0024] In some embodiments, when the number of real-time cache lines of each channel of first visible light image data collected by the binocular sensor reaches a preset number of lines, before reading the two channels of first visible light image data, the method further includes:

[0025] Through the external trigger mode, the binocular sensor is controlled to synchronously collect the two channels of the first visible light image data; wherein, the acquisition process controls the sampling of the binocular sensor according to the line field blanking of the fused image.

[0026] In some embodiments, the real-time stitching of the two channels of the second visible light image data includes:

[0027] storing the two channels of the second visible light image data to be spliced ​​into a splicing storage unit;

[0028] Whenever a single row of data in the splicing storage unit is completed, the corresponding row of data is read based on a preset timing, and the read image data is spliced ​​in real time.

[0029] Secondly, in this embodiment, a binocular visible light and infrared image fusion device is provided, comprising:

[0030] a reading module, configured to read two channels of first visible light image data when the number of real-time cache lines of each channel of first visible light image data collected by the binocular sensor reaches a preset number of lines; the preset number of lines is determined by the lens distortion parameters of the binocular sensor and preset correction algorithm parameters;

[0031] a correction module, configured to read first infrared image data corresponding to the first visible light image data, and respectively correct each channel of the first visible light image data and the first infrared image data to obtain two channels of second visible light image data and second infrared image data;

[0032] The fusion module is used to perform real-time splicing of the two channels of the second visible light image data; and to perform real-time fusion based on the splicing result and the second infrared image data to obtain a target image.

[0033] In a third aspect, a computer device is provided in this embodiment, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, the binocular visible light and infrared image fusion method described in the first aspect above is implemented.

[0034] In a fourth aspect, a storage medium is provided in this embodiment, on which a computer program is stored. When the program is executed by a processor, the binocular visible light and infrared image fusion method described in the first aspect is implemented.

[0035] Compared with the related art, the binocular visible light and infrared image fusion method, device and computer equipment provided in this embodiment read two channels of visible light image data when the number of real-time cache lines of each channel of visible light image data collected by the binocular sensor reaches a preset number of lines; the preset number of lines is determined by the lens distortion parameters of the binocular sensor and the preset correction algorithm parameters; read the infrared image data corresponding to the visible light image data, and correct each channel of visible light image data and infrared image data respectively; the corrected two channels of visible light image data are spliced ​​in real time, and the target image is obtained based on the real-time fusion of the splicing result and the corrected infrared image data, which solves the problem of high delay in fused image output and inability to ensure real-time output, and effectively reduces the delay in fused image output and ensures real-time output.

[0036] The details of one or more embodiments of the present application are set forth in the following drawings and description to make other features, objects, and advantages of the present application more readily apparent. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0038] Figure 1 This is a hardware structure block diagram of a terminal device for a binocular visible light and infrared image fusion method provided in one embodiment of the present application;

[0039] Figure 2 This is a flow chart of a binocular visible light and infrared image fusion method provided in one embodiment of the present application;

[0040] Figure 3 This is a flow chart of a binocular visible light and infrared image correction method provided by an embodiment of the present application;

[0041] Figure 4 This is a flow chart of a binocular visible light and infrared image fusion method provided by another embodiment of the present application;

[0042] Figure 5 This is a schematic diagram of the structure of a binocular visible light and infrared image fusion system provided in one embodiment of the present application;

[0043] Figure 6 This is a flow chart of a binocular visible light and infrared image fusion method provided by a preferred embodiment of the present application;

[0044] Figure 7 This is a structural block diagram of a binocular visible light and infrared image fusion device provided in one embodiment of the present application.

[0045] In the figure: 102, processor; 104, memory; 106, transmission device; 108, input and output device; 10, reading module; 20, correction module; 30, fusion module. DETAILED DESCRIPTION

[0046] In order to more clearly understand the purpose, technical solutions and advantages of the present application, the present application is described and illustrated below in conjunction with the accompanying drawings and embodiments.

[0047] Unless otherwise defined, technical or scientific terms used in this application shall have the ordinary meanings as understood by persons of ordinary skill in the art to which this application belongs. The terms "a," "an," "the," "these," and similar expressions in this application do not denote limitations on quantity and may be singular or plural. The terms "comprise," "include," "have," and any variations thereof, as used in this application, are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or device comprising a series of steps or modules (units) is not limited to the listed steps or modules (units) but may include unlisted steps or modules (units) or other steps or modules (units) inherent to the process, method, product, or device. The terms "connected," "connected," "coupled," and similar expressions used in this application are not limited to physical or mechanical connections but may include electrical connections, whether direct or indirect. As used in this application, "plurality" means two or more. "And / or" describes an association between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can mean: A exists alone; A and B exist simultaneously; or B exists alone. Generally, the character " / " indicates that the objects in the preceding and following relationship are in an "or" relationship. The terms "first", "second", "third", etc. involved in this application are only used to distinguish similar objects and do not represent a specific ordering of the objects.

[0048] The method embodiment provided in this embodiment can be executed in a terminal, a computer or a similar computing device. For example, running on a terminal, Figure 1 FIG. 1 is a block diagram of the hardware structure of the terminal of the binocular visible light and infrared image fusion method of this embodiment. Figure 1 As shown, the terminal may include one or more ( Figure 1 The processor 102 (only one is shown) and a memory 104 for storing data, wherein the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA. The terminal may also include a transmission device 106 for communication functions and an input / output device 108. It will be understood by those skilled in the art that Figure 1The structure shown is only for illustration and does not limit the structure of the above terminal. Figure 1 More or fewer components than shown, or with Figure 1 Different configurations shown.

[0049] The memory 104 can be used to store computer programs, such as application software programs and modules, such as the computer program corresponding to the binocular visible light and infrared image fusion method in this embodiment. The processor 102 executes the computer programs stored in the memory 104 to execute various functional applications and data processing, thereby implementing the aforementioned method. The memory 104 may include high-speed random access memory (RAM) and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include memory located remotely from the processor 102, and such remote memory may be connected to the terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0050] Transmission device 106 is used to receive or transmit data via a network. This network may include a wireless network provided by the terminal's communications provider. In one embodiment, transmission device 106 includes a network interface controller (NIC), which can be connected to other network devices via a base station to enable communication with the Internet. In another embodiment, transmission device 106 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0051] In this embodiment, a method for fusing binocular visible light and infrared images is provided. Figure 2 FIG. 1 is a flow chart of the binocular visible light and infrared image fusion method of this embodiment. Figure 2 As shown, the process includes the following steps:

[0052] Step S210: When the number of real-time cache lines of each channel of first visible light image data collected by the binocular sensor reaches a preset number of lines, reading the two channels of first visible light image data; the preset number of lines is determined by the lens distortion parameters of the binocular sensor and the preset correction algorithm parameters;

[0053] Step S220, reading the first infrared image data corresponding to the first visible light image data, and respectively correcting each channel of the first visible light image data and the first infrared image data to obtain two channels of second visible light image data and second infrared image data;

[0054] In step S230 , the two channels of second visible light image data are spliced ​​in real time; and the splicing result and the second infrared image data are fused in real time to obtain a target image.

[0055] Specifically, the binocular sensor collects first visible light image data row by row and stores the collected first visible light image data. When the number of real-time cache rows of each channel of first visible light image data reaches a preset number of rows, both channels of first visible light image data are read. The preset number of rows is the minimum number of cache rows and is determined by the lens distortion parameters of the binocular sensor and preset correction algorithm parameters.

[0056] For example, the first visible light image data collected is converted to a YUV format and stored separately, with the luminance (Y) component and chrominance (UV) component stored separately. The Y component is stored using four Block Random Access Memory (BRAM) groups to ensure simultaneous access to four peripheral data points during reading. The UV component is stored using one BRAM group, with one group of UV component data read at a time. In other embodiments, the YUV format can be replaced with RGB or other formats, without limitation.

[0057] Furthermore, first infrared image data corresponding to the first visible light image data is read, and correction is performed on each channel of the first visible light image data and the first infrared image data to obtain two channels of second visible light image data and second infrared image data. In this embodiment, the collected first infrared image data is pre-stored in a double data rate (DDR) memory.

[0058] The first lookup table used to correct each channel of first visible light image data and the second lookup table used to correct the first infrared image data are both pre-stored in flash memory. When the binocular sensor and the infrared imaging sensor are powered on, the lookup tables in the flash memory are written into the lookup table BRAM memory for the corresponding image data. When performing a correction operation, each channel of image data is corrected using the corresponding lookup table. That is, each channel of first visible light image data is corrected based on the first lookup table corresponding to the first visible light image data, and the first infrared image data is corrected based on the second lookup table corresponding to the first infrared image data.

[0059] The two channels of second visible light image data are stored in a splicing storage unit. Whenever a single row of data in the splicing storage unit is completed, that is, a row is filled, the corresponding row data is read based on a preset timing, and the read image data is spliced ​​in real time. Through real-time splicing through the pipeline, a large field-of-view image after the two channels of second visible light image data are obtained, which helps to reduce computing delays. Afterwards, the splicing result and the second infrared image data are analyzed by a fusion algorithm to obtain the corresponding fusion coordinate information. Based on the fusion coordinate information, the splicing result and the second infrared image data are fused in real time to obtain the target image. Among them, the fusion process can adopt a pipeline processing architecture, that is, execute pipeline superposition to achieve pipeline fusion, so that the delay reaches the nanosecond level, significantly reducing computing delays.

[0060] It's important to note that the aforementioned binocular visible light and infrared image fusion can be implemented using a field-programmable gate array (FPGA) device. This accelerates processing via the FPGA, avoiding output delays caused by large computational data volumes and effectively resolving the issue of ensuring real-time performance when processing high-frame-rate images. The FPGA employs a multi-buffer architecture, caching the collected visible light and infrared image data separately according to specific rules. It also performs steps such as image data reading, image data correction, data stitching, and fusion, which will not be detailed here.

[0061] Visible light and infrared image fusion technology is widely used in various fields, such as security monitoring, intelligent driving, and industrial inspection. However, existing fusion methods typically use C language algorithms for data processing. Due to the large amount of computational data, the output results are delayed and cannot ensure the real-time output of the fused image.

[0062] Compared to the prior art, this application reads both channels of visible light image data when the number of real-time cache lines of visible light image data collected by the binocular sensor reaches a preset number of lines; the preset number of lines is determined by the lens distortion parameters of the binocular sensor and the preset correction algorithm parameters; the infrared image data corresponding to the visible light image data is read, and each channel of visible light image data and infrared image data are corrected separately; the corrected two channels of visible light image data are spliced ​​in real time, and the target image is obtained based on the splicing result and the corrected infrared image data in real time. Based on this, according to the pre-set minimum number of cache lines, the start time of data reading is strictly controlled to ensure the minimum delay of the data flow, solving the problem of high delay in fused image output and the inability to ensure real-time output, and effectively reducing the delay in fused image output and ensuring real-time output.

[0063] In some embodiments, the binocular visible light and infrared image fusion method further includes the following steps:

[0064] Obtaining lens distortion parameters of the binocular sensor and preset correction algorithm parameters; wherein the lens distortion parameters include the number of lens distortion lines of the binocular sensor, and the preset correction algorithm parameters include the interval of the number of original image lines required for correction;

[0065] The preset number of rows is determined based on the number of lens distortion rows and the range of original image rows required for correction.

[0066] Specifically, the lens distortion parameters of the binocular sensor and the preset correction algorithm parameters are obtained. The lens distortion parameters include the number of rows of lens distortion of the binocular sensor. The preset correction algorithm parameters include the range of raw image rows required for correction, that is, the difference between the maximum and minimum rows of the raw image required for correction. Based on the number of rows of lens distortion and the range of raw image rows required for correction, a preset number of rows is calculated to cover the range of the maximum and minimum rows of the raw image required for the target image. The specific calculation formula for the preset number of rows is as follows:

[0067] m=a×2+b (1)

[0068] In formula (1), m represents the preset number of rows; a represents the number of rows of lens distortion; and b represents the interval of the number of rows of the original image required for correction.

[0069] Through this embodiment, the lens distortion parameters of the binocular sensor and the preset correction algorithm parameters are obtained. The lens distortion parameters include the number of lens distortion lines of the binocular sensor, and the preset correction algorithm parameters include the interval of the number of original image lines required for correction. Based on the number of lens distortion lines and the interval of the number of original image lines required for correction, the preset number of lines is determined. This accurately sets the number of cache lines that trigger data reading, strictly controls the start time of reading, and ensures the minimum delay of the data stream.

[0070] In some of these embodiments, Figure 3 As shown, step S220 of respectively correcting each channel of first visible light image data and first infrared image data to obtain two channels of second visible light image data and second infrared image data includes the following steps:

[0071] Step S221: Correct each channel of first visible light image data based on a first lookup table corresponding to the first visible light image data to obtain two channels of second visible light image data;

[0072] Step S222: Correct the first infrared image data based on a second lookup table corresponding to the first infrared image data to obtain second infrared image data.

[0073] Specifically, a first lookup table for correcting each channel of first visible light image data and a second lookup table for correcting first infrared image data are pre-stored in a Flash memory. When the binocular sensor and the infrared image data acquisition device are powered on, the lookup tables in the Flash memory are written into the lookup table BRAM memory corresponding to the image data. The first and second lookup tables are both preset data mapping tables, namely, LUT tables for image correction, which store the mapping relationship between original pixel values ​​and corrected pixel values.

[0074] Afterwards, when performing the correction operation, each channel of image data is interpolated and corrected using the corresponding lookup table, that is, based on the first lookup table corresponding to the first visible light image data, each channel of first visible light image data is corrected to obtain two channels of second visible light image data, and based on the second lookup table corresponding to the first infrared image data, the first infrared image data is corrected to obtain the second infrared image data.

[0075] The correction process can employ interpolation correction algorithms such as nearest neighbor interpolation, bilinear interpolation, and bicubic interpolation, though these are not specifically limited here. The interpolation correction algorithm calculates the original image coordinate information required for the target data. When the first row of the target image is calculated, the original image information is immediately selected from the BRAM cache according to a fixed time sequence of one frame.

[0076] It should be noted that, in addition to the above-mentioned image correction method based on the lookup table, correction operations may be performed using algorithms such as polynomial fitting correction and image correction based on the physical model of the imaging system, which are not specifically limited here.

[0077] Through this embodiment, based on the first lookup table corresponding to the first visible light image data, each channel of the first visible light image data is corrected to obtain two channels of second visible light image data, and based on the second lookup table corresponding to the first infrared image data, the first infrared image data is corrected to obtain the second infrared image data, thereby realizing image correction.

[0078] In some embodiments, when the number of real-time cache lines of each channel of first visible light image data collected by the binocular sensor reaches a preset number of lines, before reading the two channels of first visible light image data, the following steps are further included:

[0079] Obtaining a first calibration file of the binocular sensor and a second calibration file of the first infrared image data acquisition device;

[0080] Based on the first calibration file, generating a corresponding first lookup table;

[0081] Based on the second calibration file, a corresponding second lookup table is generated.

[0082] Specifically, the binocular sensor and the device for collecting the first infrared image data are pre-calibrated to obtain a first calibration file for the binocular sensor and a second calibration file for the device for collecting the first infrared image data. The calibration file includes intrinsic parameters (such as focal length and principal point coordinates) and extrinsic parameters (such as camera position and posture) of the corresponding camera.

[0083] Furthermore, based on the first calibration file, a corresponding first lookup table is generated, and the first lookup table is used to correct the first visible light image data collected by the binocular sensor; and based on the second calibration file, a corresponding second lookup table is generated, and the first lookup table is used to correct the first infrared image data.

[0084] Through this embodiment, a first calibration file of the binocular sensor and a second calibration file of the first infrared image data acquisition device are obtained, a corresponding first lookup table is generated based on the first calibration file, and a corresponding second lookup table is generated based on the second calibration file, so as to facilitate subsequent image data correction to correct image geometric distortion, etc.

[0085] In some embodiments, after reading the first infrared image data corresponding to the first visible light image data, the following steps are further included:

[0086] The two channels of first visible light image data and first infrared image data are synchronously processed; the synchronous processing includes image frame rate alignment and image data alignment.

[0087] Specifically, the collected first infrared image data is pre-stored in the DDR, and after the first visible light image data is read, the corresponding first infrared image data is read from the DDR using the data valid signal of the first visible light image data.

[0088] Furthermore, because the frame rate of infrared image data is typically lower than that of visible light image data, the two channels of first visible light image data and first infrared image data are synchronized. This synchronization process includes image frame rate alignment and image data alignment to reduce DDR buffering time and, therefore, latency. Specifically, the two channels of first visible light image data and first infrared image data are synchronized at the beginning of each row to ensure image data alignment.

[0089] Through this embodiment, the two channels of first visible light image data and first infrared image data are synchronously processed. The synchronous processing includes image frame rate alignment and image data alignment, which helps to reduce the overall processing delay.

[0090] In some of these embodiments, Figure 4 As shown, the step S230 of performing real-time fusion based on the stitching result and the second infrared image data to obtain the target image includes the following steps:

[0091] Step S231, analyzing the stitching result and the second infrared image data through a fusion algorithm to obtain corresponding fusion coordinate information;

[0092] In step S232 , the stitching result and the second infrared image data are fused in real time based on the fusion coordinate information to obtain a target image; wherein the fusion process adopts a pipeline processing architecture.

[0093] Specifically, a fusion algorithm analyzes the stitching results of the two second visible light image data and the second infrared image data to obtain corresponding fused coordinate information, including horizontal and vertical coordinates. Based on the fused coordinate information provided by the fusion algorithm, a parameterized design is performed to fuse the stitching results and the second infrared image data in real time based on the fused coordinate information to obtain the target image, achieving real-time and variable data overlay at different locations.

[0094] It should be noted that, based on the data synchronization of the visible light image data and the infrared image data, the above-mentioned fusion process can adopt a pipeline processing architecture to pipeline-superimpose the spliced ​​large field-of-view visible light image and the second infrared image data to achieve pipeline fusion.

[0095] Through this embodiment, the stitching result and the second infrared image data are analyzed by a fusion algorithm to obtain corresponding fusion coordinate information, and based on the fusion coordinate information, the stitching result and the second infrared image data are fused in real time to obtain the target image. The fusion process can adopt a pipeline processing architecture to reduce the delay to the nanosecond level, significantly reducing the computing delay.

[0096] In some embodiments, when the number of real-time cache lines of each channel of first visible light image data collected by the binocular sensor reaches a preset number of lines, before reading the two channels of first visible light image data, the following steps are further included:

[0097] Through the external trigger mode, the binocular sensor is controlled to synchronously collect two channels of first visible light image data; wherein, the acquisition process controls the sampling of the binocular sensor according to the line field blanking of the fused image.

[0098] Specifically, an external trigger mode is used to control the binocular sensors to synchronously collect two channels of first visible light image data, ensuring synchronization of the two channels. This eliminates the need to use DDR to cache the visible light image data before synchronization, helping to reduce the complexity of the control logic design. The acquisition process controls the binocular sensor sampling based on the horizontal and vertical blanking of the fused image. Vertical blanking refers to the time period during the image scanning process used to eliminate interference signals generated by electron beam retracement. Maintaining consistent horizontal and vertical blanking ensures image integrity and stability during display and processing.

[0099] It should be noted that when controlling the binocular sensor acquisition, the sampling control of the binocular sensor is performed according to the row field blanking of the fused image to ensure the consistency of the previous and subsequent timings, that is, to ensure that the timing of the image data collected by the binocular sensor is consistent with the timing required for the final fused image output, thereby reducing the delay time of image processing.

[0100] Through this embodiment, the binocular sensor is controlled to synchronously collect two channels of first visible light image data through the external trigger mode. The acquisition process controls the sampling of the binocular sensor according to the line field blanking of the fused image, which helps to reduce the calculation delay.

[0101] In some embodiments, the step S230 of real-time stitching of the two channels of second visible light image data includes the following steps:

[0102] storing the two channels of second visible light image data to be spliced ​​into a splicing storage unit;

[0103] Whenever a single row of data in the splicing storage unit is completed, the corresponding row of data is read based on a preset timing, and the read image data is spliced ​​in real time.

[0104] Specifically, the two-way second visible light image data are stored in a stitching storage unit. Whenever a single row of data in the stitching storage unit is completed, that is, a row is full, the corresponding row data is read based on a preset timing, and the read image data is stitched in real time to obtain a stitched image with a large field of view angle.

[0105] It should be noted that the above stitching process adopts pipeline real-time stitching. On the basis of controlling the image data acquisition timing at the front end, the stitched data can be read out in the pipeline at the same clock, which helps to reduce computing delay.

[0106] Through this embodiment, the two-way second visible light image data to be spliced ​​are stored in the splicing storage unit. Whenever a single row of data is stored in the splicing storage unit, the corresponding row data is read based on a preset timing, and the read image data is spliced ​​to realize image data splicing. At the same time, real-time splicing through the pipeline can effectively reduce computing delay.

[0107] The following combination Figure 5 , this embodiment is described and illustrated through specific embodiments.

[0108] Using external triggering, the binocular sensor is controlled to synchronously collect two channels of first visible light image data, each with an m×n size and a frame rate of 120 fps. These two channels of first visible light image data are stored in the first and second BRAMs, respectively. Simultaneously, corresponding first infrared image data with an a×b size and a frame rate of 50 fps is collected and stored in the DDR. When the number of real-time cache lines for each channel of first visible light image data reaches a preset number, the two channels of first visible light image data are read, and the corresponding first infrared image data is read from the DDR into the third BRAM using the data valid signal of the first visible light image data. DDR control is implemented in the ddr_ctrl unit, and the two channels of first visible light image data and the first infrared image data are synchronized. This synchronization process includes image frame rate alignment and image data alignment.

[0109] Furthermore, each channel of first visible light image data after synchronous processing is interpolated and corrected using a first lookup table pre-stored in Flash, and the first infrared image data after synchronous processing is interpolated and corrected using a second lookup table pre-stored in Flash, thereby obtaining two channels of second visible light image data and second infrared image data. The two channels of second visible light image data to be spliced ​​are stored in a splicing storage unit. Whenever a single row of data in the splicing storage unit is completed, the corresponding row of data is read based on a preset timing, and the read image data is spliced ​​in real time to obtain a large field of view image. The large field of view image has a size of p×q and a frame rate of 120fps. Subsequently, in the fusion unit, the splicing result and the second infrared image data are fused in real time to obtain and output a target image. The target image has a size of p×q and a frame rate of 120fps.

[0110] The present embodiment is described and illustrated below through preferred embodiments.

[0111] Figure 6 FIG. 1 is a flow chart of the binocular visible light and infrared image fusion method of the preferred embodiment. Figure 6 As shown, the binocular visible light and infrared image fusion method includes the following steps:

[0112] Step S610, obtaining the number of lines of lens distortion of the binocular sensor and preset correction algorithm parameters; wherein the preset correction algorithm parameters are the interval of the number of lines of the original image required for correction;

[0113] Step S620, determining a preset number of lines according to the number of lines of lens distortion and the interval of the number of lines of the original image required for correction;

[0114] Step S630, when the number of real-time cache lines of each channel of first visible light image data collected by the binocular sensor reaches a preset number of lines, reading two channels of first visible light image data;

[0115] Step S640: Synchronize the two channels of first visible light image data and first infrared image data; the synchronization process includes image frame rate alignment and image data alignment;

[0116] Step S650 , respectively correcting each channel of the first visible light image data and the first infrared image data after the synchronous processing to obtain two channels of second visible light image data and second infrared image data;

[0117] Step S660 , performing real-time stitching on the two channels of second visible light image data; and performing real-time fusion on the stitching result and the second infrared image data to obtain a target image.

[0118] This embodiment obtains the number of lines of lens distortion and preset correction algorithm parameters of the binocular sensor. The preset correction algorithm parameters represent the range of original image lines required for correction. The preset number of lines is determined based on the number of lines of lens distortion and the range of original image lines required for correction. When the number of real-time cached lines of first visible light image data collected by each channel of the binocular sensor reaches the preset number of lines, the two channels of first visible light image data are read and synchronized with the first infrared image data. This synchronization includes image frame rate alignment and image data alignment.

[0119] Furthermore, each channel of first visible light image data and first infrared image data after synchronous processing is corrected respectively to obtain two channels of second visible light image data and second infrared image data, the two channels of second visible light image data are spliced ​​in real time, and based on the splicing result and the second infrared image data, they are fused in real time to obtain the target image, which solves the problem of high delay in outputting the fused image and inability to ensure real-time output, and effectively reduces the delay in outputting the fused image and ensures real-time output.

[0120] It should be noted that the steps shown in the above process or the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0121] This embodiment also provides a binocular visible light and infrared image fusion device, which is used to implement the above-mentioned embodiments and preferred embodiments. Details already described will not be repeated here. The terms "module," "unit," "subunit," etc. used below may refer to a combination of software and / or hardware that implements a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, implementation using hardware, or a combination of software and hardware, is also possible and contemplated.

[0122] Figure 7 This is a structural block diagram of the binocular visible light and infrared image fusion device of this embodiment. Figure 7 As shown, the device includes:

[0123] A reading module 10 is configured to read two channels of first visible light image data when the number of real-time cache lines of each channel of first visible light image data collected by the binocular sensor reaches a preset number of lines; the preset number of lines is determined by the lens distortion parameters of the binocular sensor and preset correction algorithm parameters;

[0124] a correction module 20 configured to read the first infrared image data corresponding to the first visible light image data, and to correct each channel of the first visible light image data and the first infrared image data to obtain two channels of second visible light image data and second infrared image data;

[0125] The fusion module 30 is used to perform real-time splicing of the two channels of second visible light image data; and to perform real-time fusion based on the splicing result and the second infrared image data to obtain a target image.

[0126] Through the device provided by this embodiment, when the number of real-time cache lines of each channel of visible light image data collected by the binocular sensor reaches a preset number of lines, the two channels of visible light image data are read; the preset number of lines is determined by the lens distortion parameters of the binocular sensor and the preset correction algorithm parameters; the infrared image data corresponding to the visible light image data is read, and each channel of visible light image data and infrared image data are corrected respectively; the two channels of corrected visible light image data are spliced ​​in real time, and the target image is obtained based on the real-time fusion of the splicing result and the corrected infrared image data, thereby solving the problem of high output delay of the fused image and the inability to ensure real-time output, and effectively reducing the output delay of the fused image and ensuring real-time output.

[0127] In some of these embodiments, Figure 7 On the basis of the invention, the device also includes a calculation module for obtaining the lens distortion parameters of the binocular sensor and the preset correction algorithm parameters; wherein the lens distortion parameters include the number of lens distortion rows of the binocular sensor, and the preset correction algorithm parameters include the interval of the number of original image rows required for correction; the preset number of rows is determined according to the number of lens distortion rows and the interval of the number of original image rows required for correction.

[0128] In some embodiments, the correction module 20 is further used to correct each path of first visible light image data based on a first lookup table corresponding to the first visible light image data to obtain two paths of second visible light image data; and to correct the first infrared image data based on a second lookup table corresponding to the first infrared image data to obtain second infrared image data.

[0129] In some of these embodiments, Figure 7On the basis of, the device also includes a calibration module, which is used to obtain a first calibration file of the binocular sensor and a second calibration file of the first infrared image data acquisition device; based on the first calibration file, a corresponding first lookup table is generated; based on the second calibration file, a corresponding second lookup table is generated.

[0130] In some of these embodiments, Figure 7 On the basis of, the device also includes a synchronization module for synchronously processing the two-way first visible light image data and the first infrared image data; the synchronization processing includes image frame rate alignment and image data alignment.

[0131] In some of the embodiments, the fusion module 30 is further used to analyze the stitching result and the second infrared image data through a fusion algorithm to obtain corresponding fusion coordinate information; based on the fusion coordinate information, the stitching result and the second infrared image data are fused in real time to obtain a target image; wherein the fusion process adopts a pipeline processing architecture.

[0132] In some of these embodiments, Figure 7 On the basis of this, the device also includes a sampling module, which uses an external trigger mode to control the binocular sensor to synchronously collect two channels of first visible light image data; wherein, the acquisition process controls the sampling of the binocular sensor according to the line field blanking of the fusion image.

[0133] In some embodiments, the fusion module 30 is further used to store the two second visible light image data to be stitched into a stitching storage unit; whenever a single row of data in the stitching storage unit is completed, the corresponding row data is read based on a preset timing, and the read image data is stitched in real time.

[0134] It should be noted that the above modules can be functional modules or program modules, and can be implemented through software or hardware. For modules implemented through hardware, the above modules can be located in the same processor; or the above modules can be located in different processors in any combination.

[0135] This embodiment further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0136] Optionally, the computer device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0137] Optionally, in this embodiment, the processor may be configured to execute the following steps through a computer program:

[0138] S1, when the number of real-time cache lines of each channel of first visible light image data collected by the binocular sensor reaches a preset number of lines, reading the two channels of first visible light image data; the preset number of lines is determined by the lens distortion parameters of the binocular sensor and the preset correction algorithm parameters;

[0139] S2, reading the first infrared image data corresponding to the first visible light image data, and respectively correcting each channel of the first visible light image data and the first infrared image data to obtain two channels of second visible light image data and second infrared image data;

[0140] S3, performing real-time stitching on the two channels of second visible light image data; and performing real-time fusion on the stitching result and the second infrared image data to obtain a target image.

[0141] It should be noted that, for specific examples in this embodiment, reference may be made to the examples described in the above embodiments and optional implementation modes, and will not be repeated in this embodiment.

[0142] In addition, in conjunction with the binocular visible light and infrared image fusion methods provided in the above embodiments, a storage medium may also be provided in this embodiment to implement the method. The storage medium stores a computer program; when the computer program is executed by a processor, it implements any of the binocular visible light and infrared image fusion methods in the above embodiments.

[0143] It should be understood that the specific embodiments described herein are only used to explain this application and are not used to limit it. Based on the embodiments provided in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0144] Obviously, the accompanying drawings are merely examples or embodiments of the present application. A person skilled in the art can also apply the present application to other similar situations based on these drawings without inventive effort. Furthermore, it is understandable that, although the work involved in this development process may be complex and lengthy, certain design, manufacturing, or production changes based on the technical content disclosed in this application are merely routine technical means for a person skilled in the art and should not be considered to constitute a deficiency in the disclosure of the present application.

[0145] The term "embodiment" as used in this application refers to specific features, structures, or characteristics described in conjunction with the embodiment that can be included in at least one embodiment of this application. The appearance of this phrase in various places in the specification does not necessarily mean that the embodiment is the same, nor does it mean that it is mutually exclusive with other embodiments and is independent or optional. It is understood, either explicitly or implicitly, by those skilled in the art that the embodiments described in this application can be combined with other embodiments when there is no conflict.

[0146] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of patent protection. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. A binocular visible light and infrared image fusion method, characterized in that: include: When the number of real-time cache lines of each channel of first visible light image data collected by the binocular sensor reaches a preset number of lines, reading two channels of the first visible light image data; The preset number of rows is determined by the lens distortion parameters of the binocular sensor and the preset correction algorithm parameters; Reading first infrared image data corresponding to the first visible light image data, and respectively correcting each channel of the first visible light image data and the first infrared image data to obtain two channels of second visible light image data and second infrared image data; The two channels of the second visible light image data are spliced ​​in real time; and the splicing result and the second infrared image data are fused in real time to obtain a target image.

2. The binocular visible light and infrared image fusion method according to claim 1, characterized in that: The method further comprises: Obtaining the lens distortion parameters of the binocular sensor and the preset correction algorithm parameters; wherein the lens distortion parameters include the number of lens distortion lines of the binocular sensor, and the preset correction algorithm parameters include the interval of the number of original image lines required for correction; The preset number of rows is determined according to the number of lens distortion rows and the interval of the number of original image rows required for correction.

3. The binocular visible light and infrared image fusion method according to claim 1, characterized in that: The correcting each channel of the first visible light image data and the first infrared image data to obtain two channels of second visible light image data and second infrared image data includes: Correcting each channel of the first visible light image data based on a first lookup table corresponding to the first visible light image data to obtain two channels of the second visible light image data; The first infrared image data is corrected based on a second lookup table corresponding to the first infrared image data to obtain the second infrared image data.

4. The binocular visible light and infrared image fusion method according to claim 3, characterized in that: When the number of real-time cache lines of each channel of first visible light image data collected by the binocular sensor reaches a preset number of lines, before reading the two channels of first visible light image data, the method further includes: Obtaining a first calibration file of the binocular sensor and a second calibration file of the first infrared image data acquisition device; Based on the first calibration file, generating the corresponding first lookup table; Based on the second calibration file, a corresponding second lookup table is generated.

5. The binocular visible light and infrared image fusion method according to claim 1, characterized in that: After reading the first infrared image data corresponding to the first visible light image data, the method further includes: The two channels of the first visible light image data and the first infrared image data are synchronously processed; the synchronous processing includes image frame rate alignment and image data alignment.

6. The binocular visible light and infrared image fusion method according to claim 5, characterized in that: The step of performing real-time fusion based on the stitching result and the second infrared image data to obtain a target image includes: Analyzing the stitching result and the second infrared image data by a fusion algorithm to obtain corresponding fusion coordinate information; Based on the fusion coordinate information, the stitching result and the second infrared image data are fused in real time to obtain the target image; wherein the fusion process adopts a pipeline processing architecture.

7. The binocular visible light and infrared image fusion method according to claim 1, characterized in that: When the number of real-time cache lines of each channel of first visible light image data collected by the binocular sensor reaches a preset number of lines, before reading the two channels of first visible light image data, the method further includes: Through the external trigger mode, the binocular sensor is controlled to synchronously collect the two channels of the first visible light image data; wherein, the acquisition process controls the sampling of the binocular sensor according to the line field blanking of the fused image.

8. The binocular visible light and infrared image fusion method according to claim 7, characterized in that: The real-time stitching of the two channels of the second visible light image data includes: storing the two channels of the second visible light image data to be spliced ​​into a splicing storage unit; Whenever a single row of data in the splicing storage unit is completed, the corresponding row of data is read based on a preset timing, and the read image data is spliced ​​in real time.

9. A binocular visible light and infrared image fusion device, characterized in that: include: a reading module, configured to read two channels of first visible light image data when the number of real-time cache lines of each channel of first visible light image data collected by the binocular sensor reaches a preset number of lines; the preset number of lines is determined by the lens distortion parameters of the binocular sensor and preset correction algorithm parameters; a correction module, configured to read first infrared image data corresponding to the first visible light image data, and respectively correct each channel of the first visible light image data and the first infrared image data to obtain two channels of second visible light image data and second infrared image data; The fusion module is used to perform real-time splicing of the two channels of the second visible light image data; and to perform real-time fusion based on the splicing result and the second infrared image data to obtain a target image.

10. A computer device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to run the computer program to perform the steps of the binocular visible light and infrared image fusion method according to any one of claims 1 to 8.

Images