Image fusion method, device and equipment and computer readable storage medium

Abstract

This application relates to the field of image processing and provides an image fusion method, device, and computer-readable storage medium based on a field-programmable gate array (FPGA), enabling rapid fusion of images with different exposure areas. The method includes: performing a serial-to-parallel conversion on the original image to obtain parallel data to be processed; preprocessing and caching the parallel data to be processed to obtain a target image; reading the cached target image and fusing the image data corresponding to different exposure areas of the original image in the target image using the FPGA to obtain a fused image. Compared to existing PC-based image fusion technologies, the solution presented in this application offers faster processing speeds and higher real-time performance, making it particularly suitable for scenarios with high real-time requirements for image processing.

Description

Technical Field

[0001] This invention relates to the field of image processing, and in particular to an image fusion method, apparatus, device, and computer-readable storage medium. Background Technology

[0002] Image sensors are widely used in various security monitoring fields such as traffic surveillance. However, in low-light conditions such as rainy days or nighttime, the images captured by image sensors may exhibit anomalies. For example, in rainy weather, traffic lights captured by image sensors may appear red with a color cast, while green and yellow lights may be overexposed. At night, all lights may be overexposed. Current image sensor exposure designs are typically linear, and their dynamic range cannot meet the brightness requirements of these low-light scenarios. To address these issues, image fusion methods based on field-programmable gate arrays (FPGAs) are generally employed. However, existing image fusion technologies are PC-based, and due to the serial computation involved, this PC-based image fusion technique suffers from poor practicality. Summary of the Invention

[0003] This application provides an image fusion method, device, and computer-readable storage medium based on a field-programmable gate array (FPGA), which can achieve rapid fusion of images from different exposure areas.

[0004] On one hand, this application provides an image fusion method based on field-programmable gate arrays, the method comprising:

[0005] The original image is converted from serial to parallel to obtain parallel data to be processed;

[0006] The parallel data to be processed is preprocessed and cached to obtain the target image.

[0007] The target image is read from the cache, and the image data corresponding to different exposure areas of the original image in the target image are fused using the field-programmable gate array to obtain a fused image.

[0008] On the other hand, this application provides an image fusion device based on a field-programmable gate array (FPGA). The image fusion device includes a serial-to-parallel conversion module, a preprocessing buffer module, and a fusion processing module disposed on the FPGA, wherein:

[0009] A serial-to-parallel conversion module is used to perform serial-to-parallel conversion on the original image to obtain the parallel data to be processed;

[0010] A preprocessing cache module is used to preprocess and cache the parallel data to be processed to obtain the target processed image;

[0011] The fusion processing module is used to read the cached target image and use the field-programmable gate array to fuse image data in the target image corresponding to different exposure areas of the original image to obtain a fused image.

[0012] Thirdly, this application provides an apparatus comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described image fusion method based on a field-programmable gate array.

[0013] Fourthly, this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the image fusion method based on field-programmable gate arrays described above.

[0014] As can be seen from the technical solution provided in this application, since the technical solution is based on field-programmable gate arrays and performs serial-to-parallel conversion on the original image at the beginning of processing, and after reading the cached target processing image, the pipeline mechanism of the field-programmable gate array can be used to fuse the image data in these target processing images corresponding to different exposure areas of the original image. Therefore, compared with the existing technology that implements image fusion based on the PC, the technical solution of this application has a faster processing speed and thus higher real-time performance, and is particularly suitable for scenarios with high real-time requirements for image processing. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 This is a flowchart of an image fusion method based on a field-programmable gate array provided in an embodiment of this application;

[0017] Figure 2 This is a schematic diagram of the structure of the image fusion device based on field-programmable gate array provided in the embodiments of this application;

[0018] Figure 3 This is a schematic diagram of an example of an image fusion device based on a field-programmable gate array provided in this application embodiment;

[0019] Figure 4This is a schematic diagram of the device provided in the embodiments of this application. Detailed Implementation

[0020] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0021] In this specification, adjectives such as "first" and "second" are used only to distinguish one element or action from another, without necessarily requiring or implying any actual such relationship or order. Where circumstances permit, reference to an element, component, or step (etc.) should not be construed as limited to only one element, component, or step, but may include one or more of the elements, components, or steps, etc.

[0022] For ease of description, the dimensions of the various parts shown in the accompanying drawings are not drawn to actual scale.

[0023] This application proposes an image fusion method based on field-programmable gate arrays (FPGAs), as shown in the attached figure. Figure 1 As shown, the image fusion method based on field-programmable gate arrays mainly includes steps S101 to S103, which are detailed below:

[0024] Step S101: Perform serial-to-parallel conversion on the original image to obtain parallel data to be processed.

[0025] In this embodiment, the original image refers to the image acquired by the image sensing device (sensor). These original images are serial data in format; in one example, they can be 8-bit raw image data. To improve processing speed, the data parsing module in the Field Programmable Gate Array (FPGA) receives the original image acquired by the image sensing device through the MIPI interface. The serial-to-parallel conversion chip within the FPGA can convert the serial-formatted original image into parallel data to be processed; in one example, the format is 32-bit MIPI parallel data. In an optional example, the FPGA data parsing module utilizes the Xilinx IOSERDES serial-to-parallel conversion IP to convert the sensor's raw serial data into parallel data.

[0026] Step S102: Preprocess and cache the parallel data to be processed to obtain the target image.

[0027] In the embodiments of this application, parallel data to be processed can be preprocessed and cached based on high-speed read / write memory, such as Double Data Rate (DDR) synchronous dynamic random access memory.

[0028] As an embodiment of this application, step S102, which preprocesses and buffers the parallel data to be processed to obtain the target processed image, can be as follows: For the parallel data to be processed, the average pixel value corresponding to different exposure areas of the original image is calculated based on the pipeline characteristics of the field-programmable gate array (FPGA) to obtain the target processed image. In one example, the average value of the parallel data to be processed corresponding to different areas can be calculated using a computing module located in the FPGA. The computing module can use existing adders, dividers, etc., on the FPGA. Furthermore, different exposure areas in the original image can be exposure areas of different times. For example, it can be a long exposure time area and a short exposure time area to simultaneously obtain the bright and dark areas of the scene, in order to obtain a high dynamic range. In this embodiment, the original image includes two adjacent exposure areas, namely a long exposure time area and a short exposure time area. Here, the average value of the long exposure time area and the average value of the short exposure time area are calculated separately.

[0029] In the caching process, in one example, this embodiment of the application can arrange a cache control module inside the FPGA and a cache module outside the FPGA. For example, a cache control module such as DDR can be arranged inside the FPGA, and a cache module such as DDR3 (DDR3 has higher read and write performance than DDR and can be considered an upgraded version of DDR) can be arranged outside the FPGA; the cache control module and the cache module can interact. Of course, the cache module can also be other types of storage modules.

[0030] After the cache control module receives the first frame of data to be processed obtained in step S101, it processes it into the format required by subsequent cache modules, such as the format required by DDR3, and transmits it to DDR3. The cache control module can be programmed on the FPGA according to the requirements of subsequent cache modules. In another example, the read / write timing of subsequent cache modules can also be written into the cache control module, which can be set according to actual needs. Additionally, the cache module can transmit cached data to the cache control module when needed; then it receives the second frame of data to be processed from the cache control module and caches it; simultaneously, it receives the third frame of data to be processed from the cache control module, and so on; each subsequent frame of data to be processed can be processed in a similar way to the previous frames. At this time, the calculation module can calculate the average pixel value of different exposure areas for each frame of data to be processed. DDR3 receives and caches the current frame of data to be processed, the calculation module completes the calculation of the average pixel value of different exposure areas of the current frame of data to be processed, and outputs the calculated value of the current frame of data to be processed.

[0031] This FPGA-based pipeline feature effectively enables parallel processing of the data to be processed, thereby increasing computational efficiency.

[0032] Step S103: Read the cached target processing image, and use a field-programmable gate array to fuse the image data in the target processing image corresponding to different exposure areas of the original image to obtain a fused image.

[0033] It should be noted that after the ISP module of the field-programmable gate array (FPGA) reads the target image from the DDR3 module cache, it includes the average pixel values ​​corresponding to different exposure areas from the parallel data obtained from the computation modules on the FPGA column. First, the data is decoded to restore it to, for example, 48-bit RAW format. Since a 48-bit RAW format data consists of the pixel values ​​of 4 pixels, it means that 4 pixel values ​​can be processed in one clock cycle, thus improving the computation speed. As an embodiment of this application, the above-mentioned use of the FPGA to fuse image data corresponding to different exposure areas of the original image in the target image to obtain a fused image can be achieved by: adjusting the brightness of the target image corresponding to different exposure areas in the original image to be consistent, obtaining an intermediate image to be processed; then, performing boundary brightness smoothing processing on adjacent areas between different exposure areas in the intermediate image to be processed, obtaining the fused image.

[0034] In the above embodiments, adjusting the brightness of different exposure areas in the target image corresponding to different exposure areas in the original image to be consistent aims to restore the overall brightness of the images in different exposure areas to the brightness of a normally exposed image. Specifically, adjusting the brightness of different exposure areas in the target image corresponding to different exposure areas in the original image to obtain an intermediate image to be processed can be achieved by: taking one exposure area in the target image corresponding to different exposure areas as a reference area, using the average pixel value of the reference area as the reference pixel value, and calculating the ratio of the average pixel value of the non-reference area in the corresponding different exposure areas to the reference pixel value; adjusting the brightness of the non-reference area according to the brightness and ratio of the reference area to make the brightness of the different exposure areas in the target image corresponding to different exposure areas in the original image consistent, thus obtaining an intermediate image to be processed. The reference area can be selected as the area with the longest exposure time among the different exposure areas.

[0035] For example, assuming the average pixel value of the reference region is P0 and the average pixel value of any non-reference region is P, then calculate the ratio P0 / P between the average pixel value of any non-reference region R in different exposure regions and the average pixel value of the reference region. Then adjust the brightness of the non-reference region to make the brightness of different exposure regions in the target image consistent with that in the original image. Specifically, adjust the brightness of any pixel L in any non-reference region R to L*(P0 / P).

[0036] Furthermore, considering that adjusting the brightness of different exposure areas according to L*(P0 / P) may result in excessively high brightness, the above embodiment for forming an intermediate image to be processed also includes a threshold adjustment process. Specifically, this threshold adjustment can be:

[0037] The algorithm compares the value of each pixel in the brightness-adjusted image with a preset pixel threshold. If any pixel has a value higher than the preset pixel threshold, its value is replaced with the preset pixel threshold to obtain an intermediate image. In other words, if the brightness-adjusted image contains pixels with values ​​higher than the preset pixel threshold, these values ​​are forcibly compressed to the preset pixel threshold, ensuring that the brightness-adjusted image does not contain excessively bright pixels. The preset pixel threshold can be set according to actual needs.

[0038] As for performing boundary brightness smoothing on adjacent regions between different exposure areas in the intermediate image to be processed to obtain a fused image, that is, reading the cached target image and using the field-programmable gate array to fuse the image data corresponding to different exposure areas of the original image in the target image to obtain a fused image.

[0039] As an embodiment of this application, it can be implemented through steps S1031 to S1034, as detailed below:

[0040] Step S1031: Read the target row data of the intermediate image to be processed from the Nth row buffer of the multiple row buffers, wherein the target row data of the intermediate image to be processed is the row data immediately preceding the current row data, and N is an integer greater than or equal to 1.

[0041] In this embodiment, the multiple line buffers are cascaded, meaning the input to the current line buffer is the output of the line buffer preceding it, and the output of the current line buffer is the input of the line buffer following it. In one example, target line data of the intermediate image to be processed is read from the first line buffer of the multiple line buffers. Here, N is chosen to be 1. That is, this step can be to read the target line data from the first line buffer in the threshold-adjusted data, where the target line data of the intermediate image to be processed is the line data immediately preceding the current line data. Alternatively, the multiple line buffers can be existing storage areas in the FPGA.

[0042] Step S1032: Perform boundary brightness smoothing on the target row data of the intermediate image to be processed.

[0043] Step S1032 can be implemented by first defining a transition zone with the boundary line between adjacent areas of different exposure areas as the center line, and then replacing the pixel values ​​of the transition zone on one side of the center line with the weighted sum of the original pixel values ​​and the corresponding pixel values ​​on the other side of the center line to obtain a fused image. Specifically, replacing the pixel values ​​of the transition zone on one side of the center line with the weighted sum of the original pixel values ​​and the corresponding pixel values ​​on the other side of the center line to obtain a fused image can be:

[0044] In the transition zone, take two pixels, M1 and M2, located on either side of the center line and corresponding to different exposure areas along any line perpendicular to the center line. For ease of description later, we denote the pixel value of pixel M1 as IMG(p1,q1) and the pixel value of pixel M2 as IMG(p2,q2). Calculate the weighted pixel values ​​of M1 and M2 in the transition zone located on one side of the center line using the following formulas:

[0045]

[0046]

[0047] In the above calculation formulas for IMG(p1,q1)' and IMG(p2,q2)', MASK(p1,q1) is the pixel value corresponding to pixel point M1 in the template image, and IMG(p2,q2) is the pixel value corresponding to pixel point M2 in the template image.

[0048] In one implementation, the template image can be set as follows: First, an initial template image with the same size as the original image is defined, for example, it can be a data array with the same rows and columns as the original image array; then, regions in the initial template image corresponding to different exposure zones of the original image are assigned different initial template pixel values, and the initial template pixel values ​​of regions corresponding to the same exposure zone are the same. For example, in one example, it includes an upper half zone corresponding to short exposure and a lower half zone corresponding to high exposure, and the data of each pixel in the upper half zone is set to 2. n -1, where n is the quantization bit depth of the original pixel (e.g., 10 bits), and the data in the lower half of the image is all 0, resulting in an initial template image consisting of 1023 and 0, with the same size as the original image array. Next, the initial template image, to which the initial template pixel values ​​are assigned, undergoes multiple mean filtering processes, such as twice (the number can be set according to actual needs), to obtain the template image to be selected. Finally, a region of the same size as the transition region and centered on the center line of the transition region is selected from the template image to obtain the template image, thus creating a data matrix with different data of the same size as the transition region (i.e., the region to be smoothed), forming the template image. Of course, other methods can also be used to form the template image participating in the weighted calculation; it is not limited to this. In this example, an area-for-speed approach is used in the FPGA, pre-storing the mean filtering results at the boundaries of different exposure areas of the template image MASK in the FPGA, which can be retrieved at any time for accelerated computation.

[0049] Step S1033: Store the data after boundary brightness smoothing into the Mth row buffer of multiple row buffers, where the Mth row buffer is the row buffer after the Nth row buffer.

[0050] In one example, M is chosen as 2, and the second row buffer is the row buffer immediately following the first row buffer described above. Here, storing the data after boundary brightness smoothing into the Mth row (such as the second row) of multiple row buffers is to free up space in the Nth row (such as the first row) buffer so that the current row data of the intermediate image to be processed can be stored into the Nth row (such as the first row) buffer.

[0051] Step S1034: When storing the Mth row (the second row in this embodiment) of multiple row buffers, store the current row data of the intermediate image to be processed into the Nth row (the first row in this embodiment) buffer.

[0052] In step S1034, when storing data into the Mth row buffer of multiple row buffers, the current row data of the intermediate image to be processed is stored into the Nth row buffer. Thus, the row data of multiple row buffers are processed in parallel, reflecting a pipelined processing approach. In other words, in the FPGA, after adjusting the brightness and threshold of pixel values, the row data is buffered using Line_Buffer IP. To accelerate the smoothing of the boundaries of different exposure areas, multiple Line_Buffers are used for row buffering. For example, when the second row data (the current row data) arrives, the first row data (the target row data) is read from the first Line_Buffer and stored in the second Line_Buffer, while the second row data is stored in the first Line_Buffer, and so on. This method keeps the data unchanged when accessing data in non-boundary areas, while in boundary areas, it allows for real-time weighted substitution operations when reading pixel values ​​from the Line_Buffer, enabling smoothing of transition areas. The FPGA-accelerated regional exposure image fusion method provided by this invention can perform regional exposure image fusion processing in real time, and its fusion speed is higher than that of a PC while maintaining a high image frame rate.

[0053] From the above appendix Figure 1 As can be seen from the example of the image fusion method based on field-programmable gate arrays, since this technical solution is based on field-programmable gate arrays and performs serial-to-parallel conversion on the original image at the beginning of processing, and after reading the cached target images, the pipeline mechanism of the field-programmable gate array can be used to fuse the image data in these target images corresponding to different exposure areas of the original image. Therefore, compared with the existing technology of image fusion based on PC, the technical solution of this application has a faster processing speed and higher real-time performance, and is particularly suitable for scenarios with high real-time requirements for image processing.

[0054] Please see the appendix Figure 2 and Figure 3 This application provides an image fusion device based on a field-programmable gate array (FPGA), which includes a serial-to-parallel conversion module 201, a preprocessing buffer module 202, and a fusion processing module 203 disposed on the FPGA, as detailed below:

[0055] The serial-to-parallel conversion module 201 is used to perform serial-to-parallel conversion on the original image to obtain parallel data to be processed; for example, it can be done using XILINX IOSERDES serial-to-parallel conversion IP.

[0056] The preprocessing cache module 202 is used to preprocess and cache the parallel data to be processed to obtain the target image.

[0057] The fusion processing module 203 is used to read the cached target processing image and use a field-programmable gate array to fuse the image data in the target processing image corresponding to different exposure areas of the original image to obtain a fused image.

[0058] Optional, attached Figure 2 The example preprocessing cache module 202 may include Figure 3 The pre-computation module in the example device can be calculated using a computation module located within the FPGA. For example, the calculated average value of the parallel data to be processed corresponds to different regions. The computation module can use existing adders, dividers, etc., on the FPGA. The cache control module can use the FPGA's own control module to implement cache read and write operations for subsequent cache modules. For example, it can convert the data into the format required by subsequent cache modules or arrange read and write operations. Of course, those skilled in the art can also program the code based on the FPGA according to requirements. The image fusion device based on the field-programmable gate array also includes a cache module (such as a DDR3 module) located outside the field-programmable gate array. The pre-computation module performs computation processing, while the cache control module and the cache module perform cache processing.

[0059] Optionally, the above appendix Figure 2 In the example device, when the target image has average pixel values ​​corresponding to different exposure areas in the original image, the pre-computation module obtains the average pixel values ​​based on the pipeline characteristics of the field-programmable gate array; in another example, when caching is performed based on the caching module, the caching control module converts the data to be cached into the format required by the caching module and stores it based on the caching module.

[0060] Optionally, the above Figure 2 The example fusion module 203 may include a brightness adjustment module, a threshold adjustment module, and a boundary brightness smoothing module. The brightness adjustment module can use multipliers, dividers, etc., in an FPGA to implement the calculation of the fusion method and achieve brightness adjustment. The threshold adjustment module can use comparators, etc., in an FPGA to implement the calculation of the fusion method. The boundary brightness smoothing module can use storage areas (multi-line buffers), adders, etc., in an FPGA to implement the calculation of the fusion method. Specifically, when forming an intermediate image to be processed after threshold adjustment and performing boundary brightness smoothing, the brightness of the target image corresponding to different exposure areas in the original image is adjusted to be consistent based on the brightness adjustment module, threshold adjustment is performed based on the threshold adjustment module, and boundary brightness smoothing is performed based on the boundary brightness smoothing module.

[0061] As can be seen from the description of the above technical solution, since the technical solution is based on field-programmable gate arrays and performs serial-to-parallel conversion on the original image at the beginning of processing, and after reading the cached target processing image, the pipeline mechanism of the field-programmable gate array can be used to fuse the image data in these target processing images corresponding to different exposure areas of the original image. Therefore, compared with the existing technology that implements image fusion based on the PC, the technical solution of this application has a faster processing speed and thus higher real-time performance, and is particularly suitable for scenarios with high real-time requirements for image processing.

[0062] Figure 4 This is a schematic diagram of the structure of a device provided in one embodiment of this application. For example... Figure 4 As shown, the device 4 in this embodiment mainly includes: a processor 40, a memory 41, and a computer program 42 stored in the memory 41 and executable on the processor 40, such as a program for an image fusion method based on a field-programmable gate array (FPGA). When the processor 40 executes the computer program 42, it implements the steps described in the above embodiment of the image fusion method based on an FPGA, for example... Figure 1 The steps S101 to S103 are shown. Alternatively, when the processor 40 executes the computer program 42, it implements the functions of each module / unit in the above-described device embodiments, for example... Figure 2 The functions of the serial-to-parallel conversion module 201, the preprocessing cache module 202, and the fusion processing module 203 are shown.

[0063] For example, the computer program 42 of the image fusion method based on a field-programmable gate array (FPGA) mainly includes: performing serial-to-parallel conversion on the original image to obtain parallel data to be processed; preprocessing and caching the parallel data to be processed to obtain a target processed image; reading the cached target processed image and using the FPGA to fuse image data corresponding to different exposure areas of the original image in the target processed image to obtain a fused image. The computer program 42 can be divided into one or more modules / units, one or more of which are stored in the memory 41 and executed by the processor 40 to complete this application. One or more modules / units can be a series of computer program instruction segments capable of performing specific functions, which describe the execution process of the computer program 42 in the device 4. For example, computer program 42 can be divided into the functions of serial-to-parallel conversion module 201, preprocessing cache module 202, and fusion processing module 203 (a module in the virtual device). The specific functions of each module are as follows: serial-to-parallel conversion module 201 is used to convert the original image into parallel data to be processed; preprocessing cache module 202 is used to preprocess and cache the parallel data to be processed to obtain the target image; fusion processing module 203 is used to read the cached target image and use a field-programmable gate array to fuse the image data in the target image corresponding to different exposure areas of the original image to obtain a fused image.

[0064] Device 4 may include, but is not limited to, processor 40 and memory 41. Those skilled in the art will understand that... Figure 4 This is merely an example of device 4 and does not constitute a limitation on device 4. It may include more or fewer components than shown, or combine certain components, or different components. For example, a computing device may also include input / output devices, network access devices, buses, etc.

[0065] The processor 40 may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0066] The memory 41 can be an internal storage unit of the device 4, such as a hard disk or RAM of the device 4. The memory 41 can also be an external storage device of the device 4, such as a plug-in hard disk, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card equipped on the device 4. Furthermore, the memory 41 can include both internal and external storage units of the device 4. The memory 41 is used to store computer programs and other programs and data required by the device. The memory 41 can also be used to temporarily store data that has been output or will be output.

[0067] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed. That is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above-described device can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0068] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0069] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0070] In the embodiments provided in this application, it should be understood that the disclosed apparatus / device and method can be implemented in other ways. For example, the apparatus / device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0071] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0072] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0073] If the integrated module / unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a non-transitory computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program for the image fusion method based on a field-programmable gate array (FPGA) can be stored in a computer-readable storage medium. When executed by a processor, the computer program can implement the steps of the various method embodiments described above, namely, performing serial-to-parallel conversion on the original image to obtain parallel data to be processed; preprocessing and caching the parallel data to be processed to obtain a target processed image; reading the cached target processed image; and using an FPGA to fuse image data corresponding to different exposure areas of the original image in the target processed image to obtain a fused image. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. Non-transitory computer-readable media may include: any entity or device capable of carrying computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in non-transitory computer-readable media may be appropriately added to or subtracted from the requirements of legislation and patent practice in different jurisdictions. For example, in some jurisdictions, according to legislation and patent practice, non-transitory computer-readable media do not include electrical carrier signals and telecommunication signals. The above embodiments are only used to illustrate the technical solutions of this application and are not intended to limit it. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application. The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of this application. It should be understood that the above description is only a specific embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this invention.

Claims

1. A method of image fusion based on field programmable logic gate array, characterized in that, The method comprises: serial-parallel conversion is performed on the original image to obtain parallel to-be-processed data; the parallel to-be-processed data is preprocessed and cached to obtain a target processing image; the cached target processing image is read, and image data corresponding to different exposure regions of the original image in the target processing image is fused by using the field programmable logic gate array to obtain a fused image; wherein, the step of fusing the image data corresponding to the different exposure regions of the original image in the target processing image by using the field programmable logic gate array to obtain the fused image comprises: the brightness of the regions corresponding to the different exposure regions of the original image in the target processing image is adjusted to be consistent to obtain an intermediate to-be-processed image; adjacent regions between the regions corresponding to the different exposure regions of the original image in the intermediate to-be-processed image are subjected to boundary brightness smoothing processing to obtain the fused image.

2. The image fusion method based on field programmable logic gate array according to claim 1, wherein, The step of preprocessing and caching the parallel to-be-processed data to obtain a target processing image comprises: for the parallel to-be-processed data, average pixel values corresponding to different exposure regions of the original image are calculated based on the pipeline characteristics of the field programmable logic gate array to obtain the target processing image.

3. The image fusion method based on field programmable logic gate array according to claim 1, wherein, The step of forming the intermediate to-be-processed image further comprises a threshold adjustment step, and the threshold adjustment step comprises: comparing each pixel value in the brightness-adjusted image with a preset pixel threshold value; if there is a pixel point with a pixel value higher than the preset pixel threshold value, the pixel value of the pixel point with the pixel value higher than the preset pixel threshold value is replaced with the preset pixel threshold value to obtain the intermediate to-be-processed image.

4. The image fusion method based on field programmable logic gate array according to claim 1, wherein, The step of adjusting the brightness of the regions corresponding to the different exposure regions of the original image in the target processing image to be consistent to obtain an intermediate to-be-processed image comprises: taking one of the different exposure regions of the original image as a reference region, taking the average pixel value of the reference region as a reference pixel value, and calculating the proportion of the average pixel value of a non-reference region corresponding to the different exposure regions relative to the reference pixel value; adjusting the brightness of the non-reference region according to the brightness of the reference region and the proportion, so that the brightness of the regions corresponding to the different exposure regions of the original image in the target processing image is consistent, and the intermediate to-be-processed image is obtained.

5. The image fusion method based on field programmable logic gate array according to claim 1, wherein, The step of performing boundary brightness smoothing processing on adjacent regions between the regions corresponding to the different exposure regions of the original image in the intermediate to-be-processed image to obtain the fused image comprises: reading target row data of the intermediate to-be-processed image from an Nth row buffer of a plurality of row buffers, the target row data of the intermediate to-be-processed image being one row of data immediately preceding current row data, and the N being an integer greater than or equal to 1; performing the boundary brightness smoothing processing on the target row data of the intermediate to-be-processed image; storing the data subjected to the boundary brightness smoothing processing into an Mth row buffer of the plurality of row buffers, the Mth row buffer being a row buffer subsequent to the Nth row buffer, and the M being an integer greater than or equal to 1; The current row data of the intermediate to-be-processed image is stored in the Mth row buffer of the plurality of row buffers.

6. The image fusion method based on field programmable logic gate array according to claim 5, wherein, The boundary brightness smoothing processing on the target row data of the intermediate to-be-processed image comprises: defining a transition region with a boundary line of the adjacent region as a center line; replacing pixel values of a region on one side of the center line of the transition region with a weighted sum of the original pixel values and corresponding pixel values of a region on the other side of the center line of the transition region to obtain the fusion image.

7. The image fusion method based on field programmable logic gate array according to claim 6, wherein, replacing pixel values of a region on one side of the center line of the transition region with a weighted sum of the original pixel values and corresponding pixel values of a region on the other side of the center line of the transition region to obtain the fusion image, comprises: two pixel points on a line perpendicular to the center line and on both sides of the center line in the transition region and corresponding to different exposure regions M 1 and M 2, the pixel value of the pixel point M 1 is IMG ( p 1, q 1), and the pixel value of the pixel point M 2 is IMG ( p 2, q 2). The pixel value of the pixel point (1) is calculated according to the following formula M 1 IMG ( p 1 q 1 ) and the pixel value of the pixel point (2) is calculated according to the following formula M 2 IMG ( p 2 q 2 ) ； ; said MASK ( p 1, q 1) is a pixel value of the template image corresponding to the pixel point M 1, said MASK ( p 2, q 2) is a pixel value of the template image corresponding to the pixel point M 2.

8. The image fusion method based on field programmable logic gate array according to claim 7, wherein, The setting of the template image comprises: defining an initial template image consistent with the size of the original image; assigning different initial template pixel values to regions of the initial template image corresponding to different exposure regions of the original image, and the initial template pixel values of regions corresponding to the same exposure region are the same; performing multiple mean filtering processing on the initial template image to which the initial template pixel values are assigned to obtain a to-be-selected template image; selecting a region from the to-be-selected template image that is consistent with the size of the transition region and corresponds to a region with the center line of the transition region as a center line to obtain the template image.

9. An image fusion device based on a field programmable logic gate array suitable for use in the method according to any one of claims 1 to 8, characterized in that The image fusion device based on the field programmable logic gate array comprises a serial-parallel conversion module, a preprocessing cache module and a fusion processing module arranged on the field programmable logic gate array, wherein: The serial-parallel conversion module is configured to perform serial-parallel conversion on the original image to obtain the parallel to-be-processed data. The preprocessing cache module is configured to pre-process and cache the parallel to-be-processed data to obtain a target processing image. The fusion processing module is configured to read the cached target processing image, fuse image data corresponding to different exposure regions of the original image in the target processing image by using the field programmable logic gate array, and obtain a fusion image.

10. The FPGA-based image fusion apparatus according to claim 9, wherein, The preprocessing cache module comprises a pre-computation module, a cache control module arranged on the field programmable logic gate array, and a cache module arranged outside the field programmable logic gate array, wherein the pre-computation module is configured to perform computation processing, and the cache control module and the cache module are configured to perform cache processing.

11. The FPGA-based image fusion device of claim 10, wherein, When the target processing image has average pixel values corresponding to different exposure regions of the original image, the pre-computation module is configured to obtain the average pixel values based on the pipeline characteristics of the field programmable logic gate array and output the average pixel values to the fusion processing module. When the cache control module and the cache module perform cache processing, the cache control module is configured to convert data to be cached into a format required by the cache module, and the cache module is configured to perform storage.

12. The FPGA-based image fusion apparatus according to any one of claims 9 to 11, wherein The fusion processing module comprises a brightness adjustment module, a threshold adjustment module and a boundary brightness smoothing processing module, wherein: When the intermediate to-be-processed image adjusted by the threshold value and the boundary brightness smoothing processing are formed, the brightness of the target processing image corresponding to different exposure regions in the original image is adjusted to be consistent based on the brightness adjustment module, the threshold value adjustment is performed based on the threshold value adjustment module, and the boundary brightness smoothing processing is performed based on the boundary brightness smoothing processing module.

13. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, The computer program is executed by the processor to implement the steps of the method according to any one of claims 1 to 8.

14. A computer-readable storage medium, the computer-readable storage medium storing a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method according to any one of claims 1 to 8.

Images