A multi-parameter image local blur autonomous identification and restoration method

Through the multi-parameter image local blur autonomous recognition and recovery method, combined with grid division and deep learning network, the problems of low detection accuracy and poor recovery effect in image local blur recovery are solved, fast and efficient image recovery is achieved, and optical is enhanced Load imaging capability.

CN115511728BActive Publication Date: 2024-11-26BEIJING RES INST OF SPATIAL MECHANICAL & ELECTRICAL TECH
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202211058811.8
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-08-31
Publication Date
2024-11-26
Estimated Expiration
2042-08-31

AI Technical Summary

Technical Problem

The prior art has low detection accuracy in image local blur recovery, especially poor recovery effect at the edge, lacks fast and effective methods, making it difficult to deal with blur detection and recovery of ultra-large-scale pixel images.

Method used

Multi-parameter image local blur autonomous recognition and recovery method is used to calculate the blur of the image to mesh, determine the blur area, and use the deep learning network of the U-net architecture to restore the image, optimize the network structure to adapt to different degree of blur enter.

Benefits of technology

It realizes rapid detection and recovery of blurred areas of large-scale images, with good recovery effect, speed of 1 frame/second, avoids ringing effect, and significantly enhances the ability of optical loads to image in spatial environments.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN115511728B_ABST
    Figure CN115511728B_ABST
Patent Text Reader

Abstract

A method for autonomously identifying and restoring local blur of a multi-parameter image, which calculates the blurriness of a collected blurry image and selects the required scale for grid division according to the calculation result; performs grid division at different scales and calculates the blurriness of grid blocks of each group of blurry images after grid division at various scales; determines the blurry area according to the blurriness of the grid blocks of each group of blurry images; performs restoration calculation on the blurry image in the determined blurry area and outputs a restored clear image.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention relates to a method for autonomously identifying and restoring local blur of a multi-parameter image, and belongs to the technical field of aerospace optical remote sensing. Background Art

[0002] The development of large-scale pixel focal plane detectors can effectively enhance the performance of optical payloads, break through the bottleneck of the space-bandwidth product of traditional optical payloads, and improve the upper limit of the information detection, transmission and processing capabilities of imaging systems. However, the development of large-scale pixel focal plane detectors depends on the development and integration of ultra-small pixel imaging devices. The increase in the overall scale and the reduction in pixel size will be suitable for optical systems with small F numbers. However, the back intercept of small F-number optical systems is usually small, and large-size focal plane detectors are prone to defocus. At the same time, due to the limited level of large-scale integration technology of ultra-small pixels, the insufficient flatness of the device itself will also cause local blurring of the system image. In order to solve the image blur caused by system defocus and improve the system assembly tolerance, it is necessary to identify and restore the blurred area of ​​the system image through an algorithm. The existing image blur detection and restoration algorithm detects regular areas. When applied to the detection of actual irregular areas, the detection accuracy is low. In image restoration, the restoration effect of the restored image at the edge is poor, and there is a lack of fast and effective detection and restoration methods for blur detection and restoration of ultra-large-scale pixel images.

[0003] In 2020, Lin Chen et al. proposed a new image blind deblurring algorithm in "Blind Image Deblurring Algorithm Based on Attention Mechanism" to solve the problem of blurred image restoration. The algorithm is based on the attention mechanism and proposes a new network architecture - multi-scale attention network. The network is based on an asymmetric decoder-encoder structure. The encoding side uses residual dense blocks to improve the network's ability to extract and express input images. The decoding side introduces an attention mechanism to realize the multi-scale structure of the image pyramid. Finally, the network is reversely optimized through the loss function of the multi-scale mean square error. The blurred image restoration in real scenes has sharper edges and clearer details. However, for locally blurred images, there will be obvious ringing effects at the edges, which will affect the image restoration effect. Summary of the invention

[0004] The technical problem solved by the present invention is: in view of the problem that the restoration effect of locally blurred images in the current prior art is mediocre, a multi-parameter image local blurred autonomous identification and restoration method is proposed.

[0005] The present invention solves the above technical problems by the following technical solutions:

[0006] A multi-parameter image local blur autonomous identification and restoration method, comprising:

[0007] Calculate the blurriness of the collected blurry image, and select the required scale of grid division according to the calculation result;

[0008] Perform grid division at different scales, and perform blur calculation on grid blocks of each group of fuzzy images after grid division at various scales;

[0009] Determine the blur area according to the blur degree of the grid blocks of each group of blur images;

[0010] The blurred image in the determined blurred area is restored and the restored clear image is output.

[0011] The collected fuzzy image is divided into N×N sub-image blocks, the fuzziness of each sub-image block is calculated, and the required scale of grid division is determined according to the calculation results.

[0012] The method for determining the scale required for the grid division is:

[0013] Performing discrete cosine transform on each sub-image block to obtain frequency domain signal distribution of each sub-image;

[0014] Sum the sub-image frequency domain signals and sort them from small to large according to the summation results;

[0015] The grid scale is divided according to the sorted results. The grid scale of the first 70% sub-images is set to n×n, and the grid scale of the last 30% sub-images is set to m×m, where m=3n;

[0016] The smaller the sum result is, the clearer the corresponding sub-image is.

[0017] The method for calculating the blurriness of the grid blocks of each group of blurred images is specifically as follows:

[0018] Determine the operator matrix of the current grid block image;

[0019] Calculate the relative blur of each grid block image according to the operator matrix;

[0020] The relative blur results are weighted to determine the blur of each grid block image.

[0021] The operator matrix selects the Laplace operator, specifically:

[0022]

[0023] Where f is the image to be processed, Laplacian represents the Laplace function processing of image f, x is the horizontal direction of the image, and y is the vertical direction of the image.

[0024] When performing weighted calculation on the relative blur result, the corresponding weighting coefficient is determined according to the division scale of the selected block image, and the weighted calculation of the block image is performed.

[0025] The specific steps for calculating the fuzzy area are:

[0026] The m-block grid blur of each group of blurred images is defined as the sample set D = x 1 ,x 2 ,...,x m , and preset the number of classifications k and the maximum number of iterations N;

[0027] Randomly select k samples from the sample set and determine the initial k centroid vectors μ 1 ,μ 2 ,...,μ k ;

[0028] Determine the initial cluster partition C based on k centroid vectors t , calculate the selected sample x i The distance from each centroid vector is used to divide the clusters into C using the minimum distance value marked t Update, and determine whether the centroid vector has changed according to the updated cluster division. If there is a change, recalculate the initial cluster division according to the current centroid vector and perform iterative update. Otherwise, perform corrosion operation according to the updated cluster division to remove the surrounding discrete fuzzy points, and then perform expansion operation on the result after corrosion operation to fill the gaps in the local fuzzy area and obtain the local fuzzy area.

[0029] The method for determining the fuzzy area based on cluster division is as follows:

[0030] Determine the initial cluster partition C t :

[0031] Calculate the selected sample x i With each centroid vector μ j (j=1,2,...k) distance:

[0032] Mark the minimum distance value d ij , determine the category λ i It is used to indicate the degree of blur and is adjusted according to the required precision. The higher the precision, the more likely it is to determine the category λ. i The more;

[0033] Partition C into clusters t To update: j=1,2,...,k;

[0034] C j Recalculate the new centroid for all sample points in:

[0035] Judge the updated k centroid vectors.

[0036] The specific steps of restoring the blurred image in the determined blurred area are as follows:

[0037] Establish a blurred image restoration network architecture;

[0038] A blurred image database is established, and 80% of the data in the database is used as a training set and 20% of the data is used as a test set. Random division is performed during the training process, and images with different degrees of blur are placed in the blurred image restoration network architecture for training. The clear images obtained after training are used as restored images.

[0039] The blurred image restoration network uses the U-net architecture as the core network architecture, extracts features from the input blurred image after training, and restores the blurred image by random partitioning.

[0040] The steps of establishing the fuzzy image database are:

[0041] The remote sensing data of satellite earth observation is sliced, and the initial random wavefront data is generated using the zernike function. The corresponding point spread function is calculated according to the Fresnel diffraction formula. The sliced ​​image is convolved with the random point spread function to obtain all blurred images.

[0042] The advantages of the present invention compared with the prior art are:

[0043] The present invention provides a multi-parameter image local blur autonomous identification and restoration method, which introduces multi-scale grid division in traditional image blur recognition, and can realize rapid detection of local blur in large-scale images according to different image sizes. At the same time, deep learning is used in the image restoration algorithm, and the network structure is optimized according to inputs of different blur levels. Ultimately, the billion-pixel image blur detection and restoration speed reaches 1 frame per second, and the restored image has no obvious ringing effect, the restoration effect is good, and the ability of optical payloads to image in space environments is greatly enhanced. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 A schematic diagram of the local blur detection results provided by the invention;

[0045] Figure 2 The core architecture diagram of the image restoration network provided for the invention;

[0046] Figure 3 Schematic diagram of the local blurred image and restoration result provided by the invention;

[0047] Figure 4 A flow chart of the method for autonomous identification and restoration of local blurred images provided by the invention; DETAILED DESCRIPTION

[0048] A multi-parameter image local blur autonomous identification and restoration method, such as Figure 4 As shown in the figure, in order to solve the local blur problem of images caused by the high spatial bandwidth product optical load of the micro-pixel large-scale integrated focal plane detector, a local blur detection algorithm is introduced to optimize the restoration network. The determined blur area is combined with the blurry image and sent to the network for restoration. The restored image has no obvious ringing effect and a good restoration result is achieved. The specific steps are as follows:

[0049] Calculate the blurriness of the collected blurry image, and select the required scale of grid division according to the calculation result;

[0050] Perform grid division at different scales, and perform blur calculation on grid blocks of each group of fuzzy images after grid division at various scales;

[0051] Determine the blur area according to the blur degree of the grid blocks of each group of blur images;

[0052] The blurred image in the determined blurred area is restored and the restored clear image is output.

[0053] The collected blurred image is divided into N×N sub-image blocks, the blurriness of each sub-image block is calculated, and the required scale of grid division is determined according to the calculation results;

[0054] The method to determine the scale required for grid division is:

[0055] Performing discrete cosine transform on each sub-image block to obtain frequency domain signal distribution of each sub-image;

[0056] Sum the sub-image frequency domain signals and sort them from small to large according to the summation results;

[0057] The grid scale is divided according to the sorted results. The grid scale of the first 70% sub-images is set to n×n, and the grid scale of the last 30% sub-images is set to m×m, where m=3n;

[0058] The smaller the sum result is, the clearer the corresponding sub-image is;

[0059] The specific method for calculating the blurriness of the grid blocks of each group of blurred images is as follows:

[0060] Determine the operator matrix of the current grid block image;

[0061] Calculate the relative blur of each grid block image according to the operator matrix;

[0062] Perform weighted calculation on the relative ambiguity results to determine the ambiguity of each grid block image;

[0063] The operator matrix selects the Laplace operator, specifically:

[0064]

[0065] In the formula, f is the image to be processed, Laplacian represents the Laplace function processing of image f, x is the horizontal direction of the image, and y is the vertical direction of the image;

[0066] When performing weighted calculation on the relative fuzziness result, the corresponding weighting coefficient is determined according to the division scale of the selected block image, and the weighted calculation of the block image is performed;

[0067] The specific steps for calculating the fuzzy area are:

[0068] The m-block grid blur of each group of blurred images is defined as the sample set D = x 1 ,x 2 ,...,x m , and preset the number of classifications k and the maximum number of iterations N;

[0069] Randomly select k samples from the sample set and determine the initial k centroid vectors μ 1 ,μ 2 ,...,μ k ;

[0070] Determine the initial cluster partition C based on k centroid vectors t , calculate the selected sample x i The distance from each centroid vector is used to divide the clusters into C using the minimum distance value marked t Update, determine whether the centroid vector has changed according to the updated cluster division, if there is a change, recalculate the initial cluster division according to the current centroid vector and perform iterative update, otherwise perform corrosion operation according to the updated cluster division, remove the surrounding discrete fuzzy points, and then perform expansion operation on the result after corrosion operation to fill the gaps in the local fuzzy area and obtain the local fuzzy area;

[0071] The method for determining the fuzzy area based on cluster division is as follows:

[0072] Determine the initial cluster partition C t :

[0073] Calculate the selected sample x i With each centroid vector μ j (j=1,2,...k) distance:

[0074] Mark the minimum distance value d ij , determine the category λ i It is used to indicate the degree of blur and is adjusted according to the required precision. The higher the precision, the more likely it is to determine the category λ. i The more;

[0075] Partition C into clusters t To update: j=1,2,...,k;

[0076] C j Recalculate the new centroid for all sample points in:

[0077] Judge the updated k centroid vectors;

[0078] The specific steps of restoring the blurred image in the determined blurred area are as follows:

[0079] Establish a blurred image restoration network architecture;

[0080] Establish a blurred image database, use 80% of the data in the database as a training set and 20% of the data as a test set. Randomly divide the data during the training process, put images with different blur levels into the blurred image restoration network architecture for training, and use the clear images obtained after training as the restored images.

[0081] The blurred image restoration network uses the U-net architecture as the core network architecture. After training, it extracts features from the input blurred image and restores the blurred image through random partitioning.

[0082] The steps to establish the fuzzy image database are:

[0083] The GF-2 earth observation remote sensing data are sliced, and the first 10 Zernike coefficients are assigned random initial values ​​of 0 to 1. The Zernike function is used to generate initial random wavefront data. The corresponding point spread function is calculated according to the Fresnel diffraction formula. The sliced ​​image is convolved with the random point spread function to obtain all blurred images.

[0084] The following is further described in conjunction with the accompanying drawings and specific embodiments:

[0085] In the current embodiment, the local blurred image is identified and restored, the optical system aperture is set to 300mm, the field of view angle is 2°×2°, the focal length f is 500mm, the F number is 1.67, the spectrum is set to 0.4μm~0.7μm, and image simulation is performed to obtain a clear image. The focal depth of the system is about 5μm, and the focal plane is set to a defocused state of 10μm to obtain a blurred image. The clear image and the blurred image are spliced ​​to obtain a locally blurred image. The specific steps include:

[0086] The input blurred image is divided into three different scales, and the three scale blurred images are used as input to calculate the blurriness using the Laplacian operator, specifically:

[0087] For digital images, taking scale as an example, the Laplacian operator can be simplified to: g(i,j) = 4f(i,j)-f(i+1,j)-f(i-1,j)-f(i,j+1)-f(i,j-1) The operator can be expressed in a matrix as:

[0088]

[0089] Perform weighted calculation on the calculated Laplacian matrix. Taking scale as an example, the weighted matrix is:

[0090]

[0091] The calculated blurs of the three scales of 3×3, 9×9, and 27×27 are weighted, and the weight coefficients are 0.25, 0.5, and 0.25 respectively, to obtain the blur degrees within different scale ranges;

[0092] Input the blur value of each sub-image block and define it as the sample set D = x 1 ,x 2 ,...,x 3 , and set the number of classifications k to 2 and the maximum number of iterations N to 50;

[0093] Randomly select k samples from the dataset D as the initial k centroid vectors: μ 1 ,μ 2 ,...,μ k ;

[0094] For n = 1, 2, ..., N:

[0095] (a) Initialize the cluster partition C as

[0096] (b) For i = 1, 2, ... m, calculate the sample x i and each centroid vector μ j (j=1,2,...k) distance: x i The smallest one is marked as d ij The corresponding category λ i . Update at this time

[0097] (c) For j = 1, 2, ..., k, for C j Recalculate the new centroid for all sample points in

[0098] (d) If all k centroid vectors remain unchanged, then:

[0099] Output cluster partition C = C 1 ,C 2 ,...C k ,After eroding and dilating the data, the fuzzy region is obtained.

[0100] like Figure 1 As shown, specifically including:

[0101] (a) is the image with the keyboard in focus;

[0102] (b) is a 3×3 scale blur image;

[0103] (c) is the 9×9 scale blur image;

[0104] (d) is the blurred image of 27×27 scale;

[0105] (e) is the blurred image after weighted average

[0106] (f) is the Kmeans partition result;

[0107] (g) is the final detection result after corrosion expansion.

[0108] Input the blurred area R and blurred image I into the training network, and both are sent to the network for calculation;

[0109] The diagram of the training network structure is as follows Figure 2 As shown;

[0110] The blurred image in the determined blurred area is restored by training the network, and the output is as follows: Figure 3 (a) is a locally blurred image; (b) is a restored image.

[0111] The current embodiment introduces multi-scale grid division in traditional image blur recognition, which can realize rapid detection of local blur in large-scale images according to the size of the image. At the same time, deep learning is used in the image restoration algorithm to optimize the network structure according to the input of different blur levels, and finally realizes billion-pixel image blur detection and restoration at a speed of 1 frame per second, and the restored image has no obvious ringing effect, the restoration effect is good, and the ability of optical payloads to image in space environments is greatly enhanced.

[0112] Although the present invention has been disclosed as above in the form of a preferred embodiment, it is not intended to limit the present invention. Any person skilled in the art may make possible changes and modifications to the technical solution of the present invention by using the methods and technical contents disclosed above without departing from the spirit and scope of the present invention. Therefore, any simple modifications, equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solution of the present invention shall fall within the protection scope of the technical solution of the present invention.

[0113] The contents not described in detail in the specification of the present invention belong to the common knowledge of those skilled in the art.

Claims

1. A multi-parameter image local blur autonomous identification and restoration method, characterized in that include: Calculate the blurriness of the collected blurry image, and select the required scale of grid division according to the calculation result; Perform grid division at different scales, and perform blur calculation on grid blocks of each group of fuzzy images after grid division at various scales; Determine the blur area according to the blur degree of the grid blocks of each group of blur images; Perform restoration calculation on the blurred image of the determined blurred area and output a restored clear image; The collected blurred image is divided into N×N sub-image blocks, the blurriness of each sub-image block is calculated, and the required scale of grid division is determined according to the calculation results; The method for determining the scale required for the grid division is: Performing discrete cosine transform on each sub-image block to obtain frequency domain signal distribution of each sub-image; Sum the sub-image frequency domain signals and sort them from small to large according to the summation results; The grid scale is divided according to the sorted results. The grid scale of the first 70% sub-images is set to n×n, and the grid scale of the last 30% sub-images is set to m×m, where m=3n; The smaller the sum result is, the clearer the corresponding sub-image is; The method for calculating the blurriness of the grid blocks of each group of blurred images is specifically as follows: Determine the operator matrix of the current grid block image; Calculate the relative blur of each grid block image according to the operator matrix; The relative blur results are weighted to determine the blur of each grid block image.

2. The method for autonomous identification and restoration of local blur in a multi-parameter image according to claim 1, characterized in that: The operator matrix selects the Laplace operator, specifically: Where f is the image to be processed, Laplacian represents the Laplace function processing of image f, x is the horizontal direction of the image, and y is the vertical direction of the image.

3. The method for autonomous identification and restoration of local blur in a multi-parameter image according to claim 2, characterized in that: When performing weighted calculation on the relative blur result, the corresponding weighting coefficient is determined according to the division scale of the selected block image, and the weighted calculation of the block image is performed.

4. The method for autonomous identification and restoration of local blur in a multi-parameter image according to claim 3, characterized in that: The specific steps for calculating the fuzzy area are: The m-block grid blur of each group of blurred images is defined as the sample set D = x1, x2, ..., x m , and preset the number of classifications k and the maximum number of iterations N; Randomly select k samples from the sample set and determine the initial k centroid vectors μ1, μ2, ..., μ k ; Determine the initial cluster partition C based on k centroid vectors t , calculate the selected sample x i The distance from each centroid vector is used to divide the clusters into C using the minimum distance value marked t Update, and determine whether the centroid vector has changed according to the updated cluster division. If there is a change, recalculate the initial cluster division according to the current centroid vector and perform iterative update. Otherwise, perform corrosion operation according to the updated cluster division to remove the surrounding discrete fuzzy points, and then perform expansion operation on the result after corrosion operation to fill the gaps in the local fuzzy area and obtain the local fuzzy area.

5. The method for autonomous identification and restoration of local blur in multi-parameter images according to claim 4, characterized in that: The method for determining the fuzzy area based on cluster division is as follows: Determine the initial cluster partition C t : Calculate the selected sample x i With each centroid vector μ j (j=1,2,...k) distance: Mark the minimum distance value d ij , determine the category λ i It is used to indicate the degree of blur and is adjusted according to the required precision. The higher the precision, the more likely it is to determine the category λ. i The more; Partition C into clusters t To update: C j Recalculate the new centroid for all sample points in: ; Judge the updated k centroid vectors.

6. The method for autonomous identification and restoration of local blur in multi-parameter images according to claim 5, characterized in that: The specific steps of restoring the blurred image in the determined blurred area are as follows: Establish a blurred image restoration network architecture; A blurred image database is established, and 80% of the data in the database is used as a training set and 20% of the data is used as a test set. Random division is performed during the training process, and images with different degrees of blur are placed in the blurred image restoration network architecture for training. The clear images obtained after training are used as restored images.

7. The method for autonomous identification and restoration of local blur in multi-parameter images according to claim 6, characterized in that: The blurred image restoration network uses the U-net architecture as the core network architecture, extracts features from the input blurred image after training, and restores the blurred image by random partitioning.

8. The method for autonomous identification and restoration of local blur in multi-parameter images according to claim 7, characterized in that: The steps of establishing the fuzzy image database are: The remote sensing data of satellite earth observation is sliced, and the initial random wavefront data is generated using the zernike function. The corresponding point spread function is calculated according to the Fresnel diffraction formula. The sliced ​​image is convolved with the random point spread function to obtain all blurred images.