A real-time depth of field synthesis algorithm and system based on FPGA architecture

By implementing parallel processing on the FPGA architecture and calculating the focus coefficient using standard deviation and mean filtering, the problem of real-time depth of field synthesis in the existing technology requires a high-performance calculator, and the low-cost and lightweight real-time depth of field synthesis capability is achieved.

CN113892126BActive Publication Date: 2024-11-15SHENZHEN SUNSIGHT TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN201980096049.2
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2019-05-31
Publication Date
2024-11-15
Estimated Expiration
2039-05-31

AI Technical Summary

Technical Problem

Existing real-time depth of field synthesis algorithms require high-performance calculators to implement, resulting in large size, high cost, and inability to achieve lightweight, and difficult to implement real-time processing on low-configured computers.

Method used

The real-time depth of field synthesis algorithm based on FPGA architecture is adopted, and the parallel processing capability of FPGA is used to calculate the focus coefficient of the image through standard deviation and mean filtering, and the image data is processed in parallel to achieve real-time depth of field synthesis.

Benefits of technology

It reduces equipment costs, improves real-time processing capabilities, and realizes real-time depth of field synthesis on low-configured devices, meeting the needs of lightweight and miniaturization.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113892126B_ABST
    Figure CN113892126B_ABST
Patent Text Reader

Abstract

The present invention discloses a real-time depth of field synthesis algorithm and system based on FPGA architecture, which specifically includes the following steps: acquiring a new image; FPGA performs standard deviation calculation on the above image, uses n*n filter kernel window to calculate the standard deviation of the image, and performs filtering processing; n is an integer; uses n*n filter kernel window to calculate the average value of the standard deviation of the image, performs average filtering, and obtains the focus coefficient of the image; n is an integer; FPGA compares the focus coefficient of the new image and the previous synthesized image, controls the iteration of the new and old image data, updates the pixel data of the synthesized image and maintains the focus coefficient of the iterated image; FPGA repeats the above steps based on the parallel processing capability until the program terminates. The present invention utilizes the parallel processing capability of FPGA to optimize the algorithm that originally requires a high-performance processor into FPGA, thereby improving the real-time processing capability and reducing product cost.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of depth of field synthesis, and in particular to a real-time depth of field synthesis algorithm and system based on FPGA architecture. Background Art

[0002] Depth of field synthesis technology is a technology based on digital image processing. It uses the same image sensor to capture corresponding images at different depths of field and then fuses multiple images based on the focus characteristics of each image.

[0003] At present, there are many algorithms for implementing real-time depth of field synthesis, but they all have a common implementation process, which are:

[0004] 1. Calculate the focus coefficient of each pixel in an image.

[0005] 2. Compare the focus coefficient of each pixel with the focus coefficient of the pixel at the same position in the history. If the focus coefficient of the point is higher than the historical value, it means that the pixel is a potential point in the focal plane.

[0006] The color information of the pixel needs to be stored as the information of the final synthesized image.

[0007] 3. Capture a new image and repeat steps 1 and 2 until all the depths of field to be synthesized are captured and calculated, forming a super depth of field image.

[0008] Since the resolution of each image is relatively large, such algorithms require a computer with strong computing power as a computing support. If the computer configuration is not sufficient, the real-time effect cannot be achieved, and only offline depth of field synthesis can be achieved. In addition, since such algorithms rely on computers to complete, the volume cannot be miniaturized and lightweight. Summary of the invention

[0009] The present invention proposes a real-time depth of field synthesis algorithm and system based on FPGA architecture, which utilizes the parallel processing capability of FPGA to optimize the algorithm that originally requires a high-performance processor into FPGA, thereby improving the real-time processing capability and reducing product costs.

[0010] The technical solution of the present invention is achieved in this way:

[0011] A real-time depth of field synthesis algorithm based on FPGA architecture specifically includes the following steps:

[0012] S1, collect new images;

[0013] S2, FPGA calculates the standard deviation of the above image, uses n*n filter kernel window to calculate the standard deviation of the image, and performs filtering processing; n is an integer;

[0014] S3, using an n*n filter kernel window to calculate the average value of the image standard deviation, perform average filtering, and obtain the focus coefficient of the image; n is an integer;

[0015] S4, FPGA compares the focus coefficients of the new image and the previous synthesized image, controls the iteration of the new and old image data, updates the pixel data of the synthesized image and maintains the focus coefficient of the image after iteration; if the focus coefficient of a certain pixel point of the new image is greater than the focus coefficient of the previous synthesized image, the pixel point of the new image is replaced with the pixel point of the previous synthesized image, otherwise it is not replaced; and the larger value of the focus coefficient is updated to the focus coefficient of the image after iteration;

[0016] S5, FPGA repeats the above steps S1-S4 based on the parallel processing capability until the program terminates.

[0017] Preferably, in step S1, a new image is captured by a camera or a photosensitive component.

[0018] Preferably, step S2 specifically includes the following steps:

[0019] The standard deviation of the image is calculated using an n*n filter kernel window. The formula for the standard deviation is:

[0020]

[0021] N represents the number of pixels in the window n*n, x i Represents the value of each pixel in the window. represents the average value of the window, S N Represents the standard deviation of the pixel where this window is located.

[0022] Preferably, step S3 specifically includes the following steps:

[0023] The n*n filter kernel window is used to calculate the average value of the image, and average filtering is performed to obtain the focus coefficient of the image; the calculation formula for the average value is:

[0024]

[0025] n represents the area of ​​the window n*n, x i Represents each value in the window, Represents the average value of the window.

[0026] A real-time depth of field synthesis system based on FPGA architecture, comprising:

[0027] The image acquisition module is used to acquire new image data, convert the acquired image into video stream data, and transmit it to the image standard deviation filtering module;

[0028] An image standard deviation filtering module is used to perform standard deviation filtering on the received video stream data, generate new video stream data and image standard deviation data and transmit them to the image mean filtering module;

[0029] The image mean filter module is used to perform mean filtering on the received new image standard deviation data to obtain the image focus coefficient, and then align the corresponding video stream data and transmit it to the image synthesis module;

[0030] The image synthesis module reads the image data and image focus coefficient after the previous iteration from the DDR controller module, compares the new image standard deviation data with the image focus coefficient after the previous iteration, controls the iteration of the new and old image data, updates the pixel data of the synthesized image and updates it to the image focus coefficient after the iteration; if the new image focus coefficient of a certain pixel point is greater than the focus coefficient of the previous synthesized image, the pixel point of the new image is replaced with the pixel point of the previous synthesized image, otherwise it is not replaced, and the larger value of the focus coefficient is updated to the image focus coefficient after the iteration;

[0031] The DDR controller module is used to receive instructions from the image synthesis module, retrieve data from the DDR and hold data.

[0032] Preferably, the image standard deviation filtering module uses the formula for standard deviation filtering as follows:

[0033]

[0034] N represents the number of pixels in the window n*n, x i Represents the value of each pixel in the window. represents the average value of the window, S N Represents the standard deviation value of the pixel where this window is located;

[0035] The image mean filtering module performs mean filtering using the formula:

[0036]

[0037] n represents the area of ​​the window n*n, x i Represents each value in the window, Represents the average value of the window.

[0038] Preferably, it also includes

[0039] The image display / transmission module is used to display or transmit the image synthesized by the image synthesis module.

[0040] Preferably, it also includes

[0041] Parameter control module: used to transmit different instructions to each module, used to control the switch and window size parameters of different filter modules.

[0042] The beneficial effects of the present invention are: utilizing the parallel processing capability of FPGA, optimizing the algorithm that originally requires a high-performance processor into FPGA, improving the real-time processing capability, and reducing product costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0044] Figure 1 A flowchart of a real-time depth of field synthesis algorithm based on FPGA architecture of the present invention;

[0045] Figure 2 The present invention is a principle block diagram of a real-time depth of field synthesis system based on FPGA architecture. DETAILED DESCRIPTION

[0046] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0047] like Figure 1 As shown, the present invention proposes a real-time depth of field synthesis algorithm based on FPGA architecture, which specifically includes the following steps:

[0048] S1, capturing a new image; in step S1, a camera or a photosensitive component captures a new image.

[0049] S2, FPGA calculates the standard deviation of the above image, uses n*n filter kernel window to calculate the standard deviation of the image, and performs filtering processing; n is an integer; the collected image is transmitted to the FPGA device, and the standard deviation of the image is calculated in the FPGA. In image processing, an n*n processing window is generally used, and this window is used to filter the image.

[0050] The standard deviation of the image is calculated using an n*n filter kernel window. The formula for the standard deviation is:

[0051]

[0052] N represents the number of pixels in the window n*n, x i Represents the value of each pixel in the window. represents the average value of the window, S N Represents the standard deviation of the pixel in this window. After the above filtering, the whole image will be average filtered again.

[0053] S3, using n*n filter kernel window to calculate the average value of the image, perform average filtering, and obtain the focus coefficient of the image; n is an integer; receiving the image data from the previous step, in this step, the standard deviation of the image is average filtered, the advantage of this processing is that it can smooth the excess of the standard deviation of the image, avoid noise interference and boundary jaggedness during the synthesis process. The mean filter in FPGA also uses an n*n filter window to calculate the average value of all values ​​in this window.

[0054] The n*n filter kernel window is used to calculate the average value of the image, and average filtering is performed to obtain the focus coefficient of the image; the calculation formula for the average value is:

[0055]

[0056] n represents the area of ​​the window n*n, x i Represents each value in the window, Represents the average value of the window.

[0057] Each pixel pair will have a focus coefficient. If the focus coefficient of a certain point is also large, it means that the point is also close to a clear image in the entire image series, which is the image data required for depth of field synthesis.

[0058] The amount of calculation in the above two steps will increase in geometric proportion as the filter window becomes larger. The larger the window, the longer the calculation time. Generally, when the window reaches 31*31, it can meet the depth of field synthesis in most cases. If the image resolution is relatively large, most computers cannot achieve real-time processing. However, on the FPGA architecture, all image data can be processed in parallel, so real-time processing can be achieved.

[0059] S4, FPGA compares the focus coefficients of the new image and the previous synthesized image, controls the iteration of the new and old image data, updates the pixel data of the synthesized image and maintains the focus coefficient of the image after the iteration; if the focus coefficient of a certain pixel point in the new image is greater than the focus coefficient of the previous synthesized image, the pixel point of the new image will replace the pixel point of the previous synthesized image, otherwise it will not be replaced; and the larger value of the focus coefficient will be updated to the focus coefficient of the image after the iteration.

[0060] S5, FPGA repeats the above steps S1-S4 based on the parallel processing capability until the program terminates.

[0061] like Figure 2 As shown, the present invention also proposes a real-time depth of field synthesis system based on FPGA architecture, including

[0062] The image acquisition module is used to acquire new image data, convert the acquired image into video stream data (data stream 1), and transmit it to the image standard deviation filtering module;

[0063] The image standard deviation filter module is used to perform standard deviation filtering on the received video stream data, generate new video stream data (data stream 1) and image standard deviation data (data stream 2) and transmit them to the image mean filter module; this calculation module is a part of the image focus function. This module is mainly composed of an N*N standard deviation filter kernel, and the window size of the filter kernel can be controlled by the parameter control module.

[0064] The image mean filter module is used to perform mean filtering on the received new image standard deviation data (data stream 2) to obtain the image focus coefficient, and then align the corresponding video stream data (data stream 1) and transmit it to the image synthesis module; the main function of this module is to achieve mean blurring of the image standard deviation, thereby reducing noise interference and image edge jaggedness during subsequent image synthesis. This module is also part of the image focus function calculation. The module will receive the image standard deviation data (data stream 2) sent by the image standard deviation filter module, perform mean filtering on the image standard deviation data, and then generate a new image focus coefficient data (data stream 3) for use by the image synthesis module. This module is mainly composed of an N*N mean filter kernel, and the window size of the filter kernel can be controlled by the parameter control module.

[0065] The image synthesis module reads the image data and image focus coefficient after the previous iteration from the DDR controller module, compares the new image standard deviation data with the image focus coefficient after the previous iteration, controls the iteration of the new and old image data, updates the pixel data of the synthesized image and updates the image focus coefficient to the iteration; if the new image focus coefficient of a certain pixel point is greater than the focus coefficient of the previous synthesized image, the pixel point of the new image will replace the pixel point of the previous synthesized image, otherwise it will not be replaced. And the larger value of the focus coefficient is updated to the image focus coefficient after the iteration. This module will read the previously synthesized image data (data stream 4) and the previously saved image focus coefficient (data stream 5) from the DDR controller. Then compare the new image focus coefficient (data stream 3) and the previous image focus coefficient (data stream 5) point by point. If the new image focus coefficient of a certain pixel point is greater than the previous image focus coefficient, it means that the new image is closer to a clear image at that point. Its video stream data (data stream 1) and image focus coefficient (data 3) need to replace the previously synthesized image data (data stream 4) and the previous focus coefficient (data stream 5). On the contrary, the new image at that point is far away from the clear image. The image data will not be replaced at that point, and the original synthesized image data will be retained. Finally, the module will generate new image video stream data (data stream 6) and new image focus coefficient (data stream 7) and send them to the DDR control module. The new video stream data is sent to the image display / transmission module for display or output of depth of field synthesis.

[0066] The DDR controller module is used to receive instructions from the image synthesis module, retrieve data from the DDR and hold data.

[0067] Preferably, the image standard deviation filtering module uses the formula for standard deviation filtering as follows:

[0068]

[0069] N represents the number of pixels in the window n*n, x i Represents the value of each pixel in the window. represents the average value of the window, S N Represents the standard deviation value of the pixel where this window is located;

[0070] The image mean filtering module performs mean filtering using the formula:

[0071]

[0072] n represents the area of ​​the window n*n, x i Represents each value in the window, Represents the average value of the window.

[0073] The image display / transmission module is used to display or transmit the image synthesized by the image synthesis module.

[0074] Parameter control module: used to transmit different instructions to each module, used to control the switch and window size parameters of different filter modules.

[0075] The implementation framework of the present invention is composed of 7 modules, namely, a DDR controller module, an image acquisition module, an image standard deviation filter module, an image mean filter module, an image synthesis module, an image display / transmission module and a parameter control module. In this framework, each module can work independently and in parallel, so that the parallelism of the depth of field synthesis algorithm is optimized to the maximum extent, and the depth of field synthesis of images with different resolutions can realize real-time depth of field synthesis.

[0076] The present invention uses a specific practical algorithm to express the superiority of FPGA for depth of field synthesis. Other different algorithms can also realize depth of field synthesis of different algorithms by modifying the existing implementation module. Using FPGA architecture to realize real-time depth of field synthesis can maximize the parallel algorithm and greatly shorten the calculation time of depth of field synthesis, thereby meeting the demand for real-time depth of field synthesis of images at different resolutions. Using FPGA structure can also be used as an acceleration card for different devices to meet the demand for real-time depth of field synthesis of different devices.

[0077] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A real-time depth of field synthesis algorithm based on FPGA architecture, It is characterized in that The specific steps include: S1, collect new images; S2, FPGA calculates the standard deviation of the above image, uses n*n filter kernel window to calculate the standard deviation of the image, and performs filtering processing; n is an integer; S3, using an n*n filter kernel window to calculate the average value of the image standard deviation, perform average filtering, and obtain the focus coefficient of the image; n is an integer; S4, FPGA compares the focus coefficient of the new image and the previous synthesized image, controls the iteration of the new and old image data, updates the pixel data of the synthesized image and maintains the focus coefficient of the iterated image; If the focus coefficient of the new image of a certain pixel is greater than the focus coefficient of the previous synthesized image, the pixel of the new image will be replaced by the pixel of the previous synthesized image, otherwise it will not be replaced; and the larger value of the focus coefficient will be updated to the focus coefficient of the iterated image; S5, FPGA repeats the above steps S1-S4 based on the parallel processing capability until the program terminates.

2. The real-time depth of field synthesis algorithm based on FPGA architecture according to claim 1, It is characterized in that In step S1, a new image is captured by a camera or a photosensitive component.

3. The real-time depth of field synthesis algorithm based on FPGA architecture according to claim 1, It is characterized in that Step S2 specifically includes the following steps: The standard deviation of the image is calculated using an n*n filter kernel window. The formula for the standard deviation is: N represents the number of pixels in the window n*n, x i Represents the value of each pixel in the window. represents the average value of the window, S N Represents the standard deviation of the pixel where this window is located.

4. The real-time depth of field synthesis algorithm based on FPGA architecture according to claim 1, It is characterized in that Step S3 specifically includes the following steps: The n*n filter kernel window is used to calculate the average value of the image, and average filtering is performed to obtain the focus coefficient of the image; the calculation formula for the average value is: n represents the area of ​​the window n*n, x i Represents each value in the window, Represents the average value of the window.

5. A real-time depth of field synthesis system based on FPGA architecture, It is characterized in that include The image acquisition module is used to acquire new image data, convert the acquired image into video stream data, and transmit it to the image standard deviation filtering module; An image standard deviation filtering module is used to perform standard deviation filtering on the received video stream data, generate new video stream data and image standard deviation data and transmit them to the image mean filtering module; The image mean filter module is used to perform mean filtering on the received new image standard deviation data to obtain the image focus coefficient, and then align the corresponding video stream data and transmit it to the image synthesis module; The image synthesis module reads the image data and image focus coefficient after the previous iteration from the DDR controller module, compares the new image standard deviation data with the image focus coefficient after the previous iteration, controls the iteration of the new and old image data, updates the pixel data of the synthesized image and updates the image focus coefficient to the iteration; If the focus coefficient of the new image of a certain pixel is greater than the focus coefficient of the previous synthesized image, the pixel of the new image will be replaced by the pixel of the previous synthesized image, otherwise it will not be replaced, and the larger value of the focus coefficient will be updated to the focus coefficient of the iterated image; The DDR controller module is used to receive instructions from the image synthesis module, retrieve data from the DDR and store data.

6. The real-time depth of field synthesis system based on FPGA architecture according to claim 5, It is characterized in that The image standard deviation filtering module performs standard deviation filtering using the formula: N represents the number of pixels in the window n*n, x i Represents the value of each pixel in the window. represents the average value of the window, S N Represents the standard deviation value of the pixel where this window is located; The image mean filtering module performs mean filtering using the formula: n represents the area of ​​the window n*n, x i Represents each value in the window, Represents the average value of the window.

7. The real-time depth of field synthesis system based on FPGA architecture according to claim 5, It is characterized in that Also includes The image display / transmission module is used to display or transmit the image synthesized by the image synthesis module.

8. The real-time depth of field synthesis system based on FPGA architecture according to claim 5, It is characterized in that Also includes Parameter control module: used to transmit different instructions to each module, used to control the switch and window size parameters of different filter modules.