An image deblurring method and system based on FPGA multi-image and a storage medium

Abstract

The application discloses a kind of based on FPGA multi-image image deblurring method, system and storage medium, belong to image data processing technical field.The method includes: the image obtained is carried out deblurring convolution calculation, and deblurring convolution kernel is iterated;The picture obtained after deblurring convolution calculation of each iteration is evaluated, if evaluation is more optimal result, then using this deblurring convolution kernel replaces existing deblurring convolution kernel;After deblurring convolution kernel iteration is completed, obtain optimized deblurring convolution kernel;Using optimized deblurring convolution kernel is carried out deblurring convolution operation to the image obtained under the same scene;Wherein, multiple different functions of FPGA image are provided in FPGA, and each image is used to realize several steps in the above method.The application utilizes the multi-image function of FPGA, increases the iteration process of deblurring operator, solves the problem that single fixed operator is not applicable to different scenes, greatly reduces the demand of algorithm design on FPGA computing resources.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image data processing, in particular to an image deblurring method and system based on FPGA multi-image and a storage medium. BACKGROUND

[0002] In the image acquisition process, due to camera shaking, inaccurate focusing, noise and other reasons, the obtained image will appear blurred. Deblurring image processing is a classic problem in the field of image processing. Common methods include traditional inverse filtering in the frequency domain, Wiener filtering algorithm, traditional difference restoration algorithm in the spatial domain, least squares algorithm, and new deblurring algorithms based on deep learning. However, the above methods require a large amount of computing resources, and are generally implemented on a more powerful processor in the backend after image acquisition. For mobile devices, the common deblurring method is to collect a certain number of image results in advance, offline debugging and optimization, select the coefficient parameters that meet the conditions, and then update them to the hardware system; another type is to collect multiple mode images (such as long exposure and short exposure) by the mobile device, and then perform deblurring processing on the blurred image based on the information of the clear image obtained.

[0003] The above deblurring scheme for mobile image devices has the following problems. The first type of scheme collects image data in advance, and the optimal parameters are calculated offline. The device needs a lot of human intervention, and the efficiency is low. After offline parameter acquisition, hardware update often needs to be recompiled, and the timeliness is poor. The second type of scheme collects multiple mode image information (such as different exposure time images and different sensor images), which increases the design complexity and design cost of the mobile device. SUMMARY

[0004] The image deblurring method, system and storage medium based on FPGA multi-image proposed by the present application can at least solve one of the above technical problems.

[0005] To achieve the above purpose, the present application proposes the following technical scheme:

[0006] An image deblurring method based on FPGA multi-image, comprising:

[0007] obtaining an image, performing deblurring convolution calculation on the image, and iterating the deblurring convolution kernel;

[0008] evaluating the picture obtained after each iteration of deblurring convolution calculation, if the difference in brightness of the picture before and after deblurring convolution calculation is within a set range, and the contrast of the image after deblurring convolution calculation is enhanced, then using the deblurring convolution kernel of this time to replace the existing deblurring convolution kernel;

[0009] after the iteration of the deblurring convolution kernel is completed, an optimized deblurring convolution kernel is obtained;

[0010] using the optimized deblurring kernel to perform deblurring convolution operation on the image acquired under the same scene;

[0011] The FPGA is internally provided with a plurality of FPGA images with different functions, and each image is used to implement a plurality of steps in the above method.

[0012] Further, the FPGA is internally provided with two FPGA images with different functions: a first image and a second image.

[0013] The first image is used to iterate the deblurring kernel to obtain an optimized deblurring kernel.

[0014] The second image is used to perform deblurring convolution operation on the image acquired under the same scene according to the optimized deblurring kernel.

[0015] Further, the deblurring convolution calculation is a deblurring convolution calculation based on a LoG operator, comprising:

[0016] The convolution kernel parameter in the LoG operator traversing in a set interval;

[0017] each of the traversing process , has a set LoG operator, according to the set LoG operator to perform convolution calculation on the image, to obtain the edge information of the image;

[0018] superimpose the image edge information and the input image to obtain the image after deblurring convolution calculation.

[0019] Further, the image obtained after each iteration of deblurring convolution calculation is evaluated, comprising:

[0020] The evaluation index is the brightness M and the contrast V of any image S, which is calculated by the following formula:

[0021]

[0022]

[0023] Where N is the number of pixels on the image, S i is the pixel value of the current pixel point;

[0024] The evaluation index corresponding to the existing deblurring kernel is M inter and V inter ; The evaluation index corresponding to the image before deconvolution calculation is M0 and V0;

[0025] For the image obtained after each iteration of deblurring convolution calculation , calculate its brightness and contrast ;

[0026] If and , the current deblurring kernel is used to replace the existing deblurring kernel;

[0027] Wherein, Thr is the set luminance difference range.

[0028] On the other hand, the application also proposes a FPGA multi-image based image deblurring system, comprising:

[0029] FPGA, the FPGA is provided with two different function FPGA images: the first image and the second image, and a non-volatile memory;

[0030] The first image is used to obtain an image, perform deblurring convolution calculation on the image, and iterate the deblurring convolution kernel; evaluate the picture obtained after each iteration of deblurring convolution calculation, if the luminance difference between the picture before and after deblurring convolution calculation is within the set range, and the contrast of the image after deblurring convolution calculation is enhanced, then use the current deblurring kernel to replace the existing deblurring kernel; after completing all iterations, obtain the optimized deblurring kernel;

[0031] The non-volatile memory is used to store the parameters of the first image, the second image and the optimized deblurring kernel;

[0032] The second image is used to read the parameters of the optimized deblurring kernel, and perform deblurring convolution operation on the image obtained under the same scene according to the parameters.

[0033] Further, the first image comprises:

[0034] The parameter iteration calculation module is used to perform deblurring convolution calculation on the image, and iterate the deblurring convolution kernel;

[0035] The intermediate operator temporarily saves the optimized deblurring kernel, and the initial value is the existing deblurring kernel;

[0036] The quality monitoring module evaluates the picture obtained after each iteration of deblurring convolution calculation, judges whether the luminance difference between the picture before and after deblurring convolution calculation is within the set range, and whether the contrast of the image before and after deblurring convolution calculation is enhanced; if not, the intermediate operator remains unchanged; if yes, the intermediate operator is updated to the current deblurring kernel;

[0037] After the iteration is completed, the parameters of the deblurring kernel of the intermediate operator are saved to the non-volatile memory.

[0038] Further, the second mirror image comprises:

[0039] An image acquisition control module is configured to control an external device to acquire a picture and input the picture to the parameter application module.

[0040] The parameter application module reads the parameters of the deblurring convolution kernel in the non-volatile memory, and performs deblurring operation on the image through deblurring convolution calculation.

[0041] An image output module is configured to output the deblurred image.

[0042] Further, the deblurring convolution calculation is deblurring convolution calculation based on a LoG operator.

[0043] Further, after the system is started, if the current picture acquisition scene does not change and the first mirror image has obtained the optimized deblurring convolution kernel, the second mirror image directly performs deblurring convolution operation on the image.

[0044] If the current picture acquisition scene changes, the first mirror image first iterates the deblurring convolution kernel to obtain the parameters of the optimized deblurring convolution kernel, and then the second mirror image reads the parameters of the optimized deblurring convolution kernel and performs deblurring convolution operation on the image acquired under the same scene according to the parameters.

[0045] In another aspect, the application further provides a computer readable storage medium, wherein the computer readable storage medium stores a computer program, and the computer program is executed by a processor to make the processor execute the above-mentioned image deblurring method based on FPGA multi-mirror image.

[0046] In a mobile image acquisition device, due to the limitation of device computing resources, image deblurring generally uses a fixed operator to perform convolution calculation on the image, but the deblurring effect of the fixed operator cannot meet the application requirements of different scenes at the same time, and the deblurring effect may decrease after the scene is changed. The application increases the deblurring operator iteration and updating function for the current scene on the basis of the application of the deblurring operator, can optimize the operator parameters for different scenes, and realizes the ability of compatible multi-scene application. In addition, the application divides the operator iteration function and the operator application function into time-sharing operation according to the multi-mirror image function of the FPGA, and further reduces the requirement of the algorithm on the hardware computing resources. BRIEF DESCRIPTION OF DRAWINGS

[0047] Figure 1 is a schematic diagram of the image deblurring system based on FPGA multi-mirror image of the application;

[0048] Figure 2 is a schematic diagram of the storage content of the non-volatile memory in the embodiment of the application;

[0049] Figure 3 This is a flow chart of switching of the FPGA in an embodiment of the present invention. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments.

[0051] This embodiment first proposes an image deblurring method based on FPGA multiple mirrors. In this method, multiple FPGA mirrors with different functions are provided in the FPGA. In this embodiment, two FPGA mirrors are provided in the FPGA: a first mirror and a second mirror. It is understandable that more mirrors can be designed according to different requirements of the deblurring algorithm, and each mirror is used to implement several steps in the deblurring method.

[0052] like Figure 1 As shown, the first image provides an iterative optimization function for the deblurring operator, which is mainly composed of an initial operator, a parameter iteration calculation module, an intermediate operator, a quality monitoring module, and an image acquisition control module. The initial operator stores the initial value of the deblurring convolution kernel. Depending on the selected deblurring algorithm, the convolution kernel has different orders and values. The parameter iteration calculation module is used to repeatedly calculate the parameters of the deblurring convolution kernel. The parameters are mainly the values ​​of the convolution kernel elements. The parameter iteration method is selected according to different algorithms and can be set to parameter equal step traversal or adjusted according to the calculation results. The intermediate operator is the next iterative calculation convolution kernel obtained based on the calculation result of the current input convolution kernel. The quality monitoring module is used to evaluate the effect of image deblurring under current conditions. This solution comprehensively considers image brightness and contrast. The image brightness before and after the convolution calculation is within a similar range, and the result of increased contrast is retained. The image acquisition control module controls the device to collect images as input during the iterative calculation process. After the calculation is completed, the optimized operator is written to the non-volatile memory.

[0053] The second image provides the application function of the deblurring operator and is mainly composed of an image acquisition control module, a parameter application module, and an image output module. The image acquisition control module is similar to the image acquisition control module in the first image. The parameter application module reads the optimization operator obtained from the first image from non-volatile memory and convolves it with the input image. The image output module transmits the resulting image according to a specific protocol.

[0054] In this solution, the first image uses the LoG operator to perform convolution calculation on the input image, extract the edge information of the image, and then superimpose the convolution result with the original image to obtain the enhanced and deblurred image. The calculation of the LoG operator is as follows:

[0055]

[0056] x, y are pixel coordinate positions, is the parameter of LoG operator. Different LoG operators can extract edges of images at different scales. The parameter iterative calculation module of the first mirror image calculates the values in the set interval traversal. For a certain LoG operator under a certain parameter, the parameter iterative calculation module first generates a 5x5 order discrete normalized convolution kernel , such as = 0.5, there is:

[0057]

[0058] Convolution kernel and input image I are convolved to obtain image edge information , and then it is superimposed with the input image to obtain the current deblurred image :

[0059]

[0060] The quality monitoring module of the first mirror image receives the image , calculates the image brightness and contrast , wherein:

[0061]

[0062]

[0063] Where N is the number of pixels on the image, S i is the pixel value of the current pixel point.

[0064] Compare , with the original input image corresponding indicators and and the intermediate operator corresponding indicators , , when and , update the intermediate operator and its corresponding indicators to , , , otherwise, the intermediate operator and the corresponding index are kept unchanged (the initial value of the intermediate operator is set as the initial operator). Wherein, Thr is the set luminance difference range. When the parameter iterative calculation module completes the value traversal, the first mirror saves the intermediate operator to the corresponding area of the non-volatile memory.

[0065] In the embodiment, the second mirror parameter application module is based on the optimized LoG operator, and the corresponding convolution kernel is h. The edge information of the actual input image is extracted and superimposed with the original image to obtain the deblurred image S, that is:

[0066]

[0067] The embodiment also proposes a FPGA multi-mirror-based image deblurring system, which corresponds to the FPGA multi-mirror-based image deblurring method described above, as shown in Figure 1 The system includes an FPGA, which is provided with two FPGA mirrors with different functions: a first mirror and a second mirror, and a non-volatile memory.

[0068] The first mirror is used to obtain an image, perform deblurring convolution calculation on the image, and iterate the deblurring convolution kernel. The image obtained after each iteration of deblurring convolution calculation is evaluated. If the luminance difference between the image before and after deblurring convolution calculation is within the set range, and the contrast of the image after deblurring convolution calculation is enhanced, the deblurring convolution kernel of this time is used to replace the existing deblurring convolution kernel. After all iterations are completed, the optimized deblurring convolution kernel is obtained.

[0069] The non-volatile memory is used to store the parameters of the first mirror, the second mirror, and the optimized deblurring convolution kernel. The non-volatile memory also stores a startup information table, a backup startup information table, and transfer parameters between mirrors (optimized deblurring operators for specific scenarios).

[0070] The second mirror is used to read the parameters of the optimized deblurring convolution kernel and perform deblurring convolution operation on the image obtained under the same scene according to the parameters.

[0071] The non-volatile memory stores two or more mirrors, as shown in Figure 2 The mirror startup information table is stored in the starting position (Addr0) of the non-volatile memory at the same time, which is used to guide the FPGA to load and start the mirror from a specific address of the non-volatile memory. The mirror startup information table contains device feature information and startup address information. The device feature information is used for FPGA matching and identification. When the device feature information matches, the FPGA will continue to read the subsequent content and interpret it as a jump address, and then load and start the mirror from the jump address.

[0072] ​The FPGA switching different image process is shown in Figure 3 If the current running image is the first image and the parameter iterative calculation is completed, the starting address information in the starting information table in the head region of the non-volatile memory is updated to the address (Addr2) of the second image, and after the updating is completed, the FPGA is controlled to restart and load the image. If the current running image is the second image and the optimized deblurring operator needs to be updated due to the change of the device running environment, the starting address information in the starting information table in the head region of the non-volatile memory is updated to the address (Addr1) of the first image, and after the updating is completed, the FPGA is controlled to restart and load the image. If there are multiple images, the image switching mode is similar to the above method, the starting address information in the starting information table in the head region of the non-volatile memory is updated according to the image starting process, and the FPGA is controlled to restart.

[0073] The format of the standby starting information table is the same as that of the starting detailed table, and is placed at the end (Addr4) of the non-volatile memory. Since the starting information table in the head region is frequently updated, if an error occurs or the device is powered off during the updating, the FPGA will load the starting information from the standby starting information table at the end of the non-volatile memory to start normally.

[0074] When the device faces a new scene, the user makes the device load the first image by sending a command, after the loading, the device controls the collection according to the image collection module, and performs the iterative calculation of the LoG operator based on the collection result according to the calculation process of the first image described above, when the iterative calculation is completed, the device exits the operator iterative process and writes the optimized operator into the corresponding region (Addr3) of the non-volatile memory; after the operator iterative calculation is completed and the corresponding region is written, the device loads the second image (normal working image) according to the image switching process described above, at this time, the user can normally configure the device to work; after the user starts the image collection, the second image reads the deblurring operator from the corresponding region (Addr3) of the non-volatile memory, and performs the image calculation according to the calculation process of the second image described above; the calculation result is output and displayed by the output module according to a specific protocol.

[0075] The image deblurring system based on the FPGA multiple images of the embodiment uses the dual image function of the FPGA, increases the deblurring operator iterative process, and solves the problem of the adaptability decline of the single fixed operator facing different scenes; at the same time, the dual image is used to time-multiplex the operator iterative module and the parameter application module, and the demand of the algorithm design for the FPGA calculation resource is greatly reduced.

[0076] The embodiment further includes a computer readable storage medium, and the computer readable storage medium stores a computer program, and the computer program is executed by a processor to make the processor execute the image deblurring method based on the FPGA multiple images.

[0077] Any combination of the technical features in the above embodiments can be made, and for the sake of brevity, not all possible combinations are described above. However, any combination of features described above is deemed to be within the scope of the present disclosure, as long as such combinations are not mutually contradictory.

[0078] The above embodiments are merely used to illustrate the technical solutions of the present disclosure, rather than limit the present disclosure; even though the present disclosure has been described in detail with reference to the foregoing embodiments, those ordinarily skilled in the art should understand that they can still make modifications to the technical solutions described in the foregoing embodiments, or equivalently replace some technical features therein; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present disclosure.

Claims

1. An FPGA multi-image-based image deblurring method, characterized in that, The method comprises the following steps: obtaining an image, performing deblurring convolution calculation on the image, and iterating a deblurring kernel; evaluating the image obtained after each iteration of the deblurring convolution calculation, and if the difference in brightness between the image before and after the deblurring convolution calculation is within a set range and the contrast of the image after the deblurring convolution calculation is enhanced, using the deblurring kernel of this iteration to replace the existing deblurring kernel; after the iteration of the deblurring kernel is completed, obtaining an optimized deblurring kernel; performing deblurring convolution operation on the image obtained under the same scene using the optimized deblurring kernel. The FPGA is internally provided with a plurality of FPGA images with different functions, and each image is used to implement a plurality of steps in the above method.

2. The FPGA-based multi-image based image deblurring method of claim 1, wherein, The FPGA is internally provided with two FPGA images with different functions: a first image and a second image. The first image is used to iterate the deblurring kernel and obtain an optimized deblurring kernel. The second image is used to perform deblurring convolution operation on the image obtained under the same scene according to the optimized deblurring kernel.

3. The FPGA-based multi-image based image deblurring method of claim 1, wherein, The deblurring convolution calculation is a deblurring convolution calculation based on a LoG operator, which comprises the following steps: Convolution kernel parameters in log operators Traverse within a set interval; each of the traversing processes , having a set LoG operator, and performing convolution calculation on the image according to the set LoG operator to obtain edge information of the image; superimposing the edge information of the image on the input image to obtain the image after the deblurring convolution calculation.

4. The FPGA-based multi-image based image deblurring method of claim 3, wherein, The evaluation of the image obtained after each iteration of the deblurring convolution calculation comprises the following steps: The evaluation index is the brightness M and the contrast V of any image S, which is calculated by the following formula: where N is the number of pixels on the image, S i is the pixel value of the current pixel point; The existing evaluation index corresponding to the deblurring convolution kernel is M inter and V inter ; the evaluation index corresponding to the image before the deconvolution calculation is M0 and V0; the image resulting from the deblurring convolution calculation after each iteration , calculating its luminance and contrast ; If and , then the current deblurring kernel is used instead of the existing deblurring kernel. wherein Thr is the set range of the difference in brightness.

5. An FPGA multi-image based image deblurring system, characterized in that, The method comprises the following steps: The FPGA is internally provided with two FPGA images with different functions: a first image and a second image, and a non-volatile memory. The first image is used to obtain an image, perform deblurring convolution calculation on the image, and iterate a deblurring kernel. The evaluation of the image obtained after each iteration of the deblurring convolution calculation comprises the following steps: The evaluation index is the brightness M and the contrast V of any image S, which is calculated by the following formula: wherein Thr is the set range of the difference in brightness. The method comprises the following steps:

6. The FPGA-based multi-image based image deblurring system of claim 5, wherein, The non-volatile memory is used to store the parameters of the first image, the second image, and the optimized deblurring kernel. The second image is used to read the parameters of the optimized deblurring kernel and perform deblurring convolution operation on the image obtained under the same scene according to the parameters. The first image comprises the following steps: The parameter iteration calculation module is used to perform deblurring convolution calculation on the image and iterate a deblurring kernel. The intermediate operator temporarily saves the optimized deblurring kernel, and the initial value thereof is the existing deblurring kernel. The quality monitoring module evaluates the image obtained after each iteration of the deblurring convolution calculation, judges whether the difference in brightness between the image before and after the deblurring convolution calculation is within a set range and the contrast of the image before and after the deblurring convolution calculation is enhanced; if not, the intermediate operator remains unchanged; if yes, the intermediate operator is updated to the deblurring kernel of this iteration. After the iteration is completed, the parameters of the deblurring kernel of the intermediate operator are saved to the non-volatile memory.

7. The FPGA-based multi-image based image deblurring system of claim 5, wherein, The second mirror image comprises: An image acquisition control module, configured to control an external device to acquire a picture and input the picture to the parameter application module; The parameter application module reads parameters of a deblurring convolution kernel in a non-volatile memory, and performs a deblurring operation on the image through deblurring convolution calculation. An image output module, configured to output the deblurred image.

8. The FPGA-based multi-image based image deblurring system of claim 5, wherein, The deblurring convolution calculation is based on a LoG operator.

9. The FPGA-based multi-image based image deblurring system of claim 5, wherein, After the system is started, if the current picture acquisition scene does not change and the first mirror image has obtained an optimized deblurring convolution kernel, the second mirror image directly performs a deblurring convolution operation on the image. If the current picture acquisition scene changes, the first mirror image first iterates the deblurring convolution kernel to obtain parameters of the optimized deblurring convolution kernel, and then the second mirror image reads the parameters of the optimized deblurring convolution kernel and performs a deblurring convolution operation on the image acquired under the same scene according to the parameters.

10. A computer-readable storage medium having stored therein a computer program, characterized in that, The computer program, when executed by a processor, causes the processor to perform the FPGA multi-mirror image-based image deblurring method according to any one of claims 1 to 4.

Images