An image restoration method, system and computer program product

Abstract

The application discloses an image restoration method, system and computer program product, and relates to the technical field of semiconductor manufacturing. The image restoration method comprises the following steps: performing a real-time processing procedure of image restoration through an FPGA, that is, completing the real-time processing procedure including operator preprocessing, frequency domain convolution operation and data synchronous operation under effective flag control through the FPGA, so as to reduce the delay of the image restoration algorithm, realize the synchronous flow of a target image and an operation result or, avoid invalid operation caused by asynchronous data, improve the image restoration efficiency, and the real-time processing procedure does not depend on a software running environment and has high reliability.

Description

Technical Field

[0001] This application belongs to the field of semiconductor manufacturing technology, and in particular relates to an image restoration method, system and computer program product. Background Technology

[0002] As chip manufacturing precision increases, the demand for image quality in optical defect detection is also rising. However, during operation, imaging equipment is affected by factors such as optical diffraction and lens aberrations, often resulting in blurred chip images and unclear edge details, leading to false positives and false negatives in subsequent defect detection. Therefore, restoring the acquired chip images has become a crucial step in improving the reliability of defect detection.

[0003] Currently, deconvolution algorithms are typically run in the Graphics Processing Unit (GPU). This involves acquiring and utilizing the point spread function (PSF) of the imaging device to construct a degradation model, and then using this model to compensate for lost image information during the imaging process, thus achieving image restoration. However, GPUs have high hardware costs, and their operating speed depends on software drivers and the system's runtime environment, making them susceptible to interference from other tasks in the system. Consequently, the reliability of image restoration using GPUs is relatively poor. Summary of the Invention

[0004] This application provides an image restoration method, system, and computer program product that can improve the reliability and efficiency of image restoration.

[0005] A first aspect of this application provides an image restoration method applied to an FPGA, comprising: Receive and buffer target images sent by imaging devices; Based on the operational image size of the image restoration, the pre-cached initial forward operator and initial backward operator are preprocessed to obtain the target forward operator and target backward operator that match the operational image size in the frequency domain dimension. The initial forward operator and the initial backward operator are determined based on the point spread function image of the imaging device. The target forward operator and the target image are subjected to frequency domain convolution operation to obtain a first convolution result, and a first valid flag of the first convolution result is generated. Based on the first valid flag, the cached target image is read, and an error correction factor is determined according to the target image and the first convolution result. The target backward operator and the error correction factor are subjected to frequency domain convolution operation to obtain a second convolution result, and a second valid flag of the second convolution result is generated. Based on the second valid flag, the cached target image is read, and the restored image is determined according to the target image and the second convolution result.

[0006] A second aspect of this application provides an image restoration apparatus applied to an FPGA, comprising: The first receiving module is used to receive and buffer the target image sent by the imaging device; The operator preprocessing module is used to preprocess the pre-cached initial forward operator and initial backward operator according to the size of the operational image of the image restoration, so as to obtain the target forward operator and target backward operator that match the size of the operational image in the frequency domain dimension. The initial forward operator and the initial backward operator are determined based on the point spread function image of the imaging device. The first convolution module is used to perform frequency domain convolution operation on the target forward operator and the target image to obtain a first convolution result and generate a first valid flag of the first convolution result; The error correction module is used to read the cached target image based on the first valid flag, and determine the error correction factor based on the target image and the first convolution result. The second convolution module is used to perform frequency domain convolution operation on the target backward operator and the error correction factor to obtain a second convolution result, and generate a second valid flag of the second convolution result; The restoration module is used to read the cached target image based on the second valid flag, and determine the restored image based on the target image and the second convolution result.

[0007] A third aspect of this application provides an image restoration system, the system comprising a host computer and an FPGA, including: The host computer is used to receive the point spread function image sent by the imaging device, and determine the initial forward operator and the initial backward operator based on the point spread function image; and send the initial forward operator and the initial backward operator to the FPGA for caching after preprocessing; The FPGA is used to: receive and cache the target image sent by the imaging device; preprocess the pre-cached initial forward operator and the initial backward operator according to the operational image size of the target image to obtain a target forward operator and a target backward operator that match the operational image size in the frequency domain dimension; invoke the target forward operator to perform a frequency domain convolution operation on the target forward operator and the target image to obtain a first convolution result and generate a first valid flag of the first convolution result; based on the first valid flag, read the cached target image and determine an error correction factor according to the target image and the first convolution result; perform a frequency domain convolution operation on the target backward operator and the error correction factor to obtain a second convolution result and generate a second valid flag of the second convolution result; based on the second valid flag, read the cached target image and determine the restored image according to the target image and the second convolution result.

[0008] A fifth aspect of the embodiments of this application provides an electronic device, the device comprising: a memory and a program or instructions stored in the memory and executable on a processor, wherein when the program or instructions are executed by the processor, they implement an image restoration method as provided in any aspect of the embodiments of this application described above.

[0009] A sixth aspect of the embodiments of this application provides a readable storage medium on which a program or instructions are stored, and when the program or instructions are executed by a processor, they implement an image restoration method as provided in any of the embodiments of this application described above.

[0010] A seventh aspect of the embodiments of this application provides a computer program product in which instructions, when executed by a processor of an electronic device, cause the electronic device to perform an image restoration method as provided in any of the embodiments of this application described above.

[0011] The technical solution provided in this application has at least the following beneficial effects: In the image restoration method provided in this application embodiment, the real-time processing of image restoration is performed by an FPGA. That is, the real-time processing including operator preprocessing, frequency domain convolution operation and data synchronization operation under effective flag control is completed by the FPGA. By using the hardware pipeline architecture and timing control logic of the FPGA, the latency of the image restoration algorithm is reduced, and the synchronous flow of the target image and the operation result is realized. Invalid operation caused by data asynchrony is avoided, the image restoration efficiency is improved, and the real-time processing does not depend on the software running environment, so the reliability is high. Attached Figure Description

[0012] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 This is a flowchart illustrating an image restoration method provided in one embodiment of this application; Figure 2 This is a schematic diagram of a point spread function image acquired by an imaging device provided in one embodiment of this application; Figure 3 This is a schematic diagram of the FPGA operator preprocessing flow provided in one embodiment of this application; Figure 4 This is a schematic diagram illustrating the principle of image restoration provided in one embodiment of this application; Figure 5 This is a schematic diagram illustrating the principle of image processing provided in one embodiment of this application; Figure 6a This is a schematic diagram of a target image provided in one embodiment of this application; Figure 6b This is a schematic diagram of a restored image provided in one embodiment of this application; Figure 7 This is a schematic diagram of the structure of an image restoration system provided in one embodiment of this application; Figure 8 This is a schematic diagram of the structure of an image restoration device provided in one embodiment of this application; Figure 9 This is a schematic diagram of an image restoration device provided in one embodiment of this application. Detailed Implementation

[0014] The features and exemplary embodiments of various aspects of this application will be described in detail below. To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain this application and not to limit it. For those skilled in the art, this application can be implemented without some of these specific details. The following description of the embodiments is merely to provide a better understanding of this application by illustrating examples.

[0015] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes said element.

[0016] It should be noted that the acquisition, storage, use, and processing of data in the technical solution of this application all comply with the relevant provisions of national laws and regulations. In the embodiments of this application, certain existing industry solutions such as software, components, and models may be mentioned. These should be considered exemplary, intended only to illustrate the feasibility of implementing the technical solution of this application, and do not imply that the applicant has already used or necessarily used such solutions.

[0017] First, the terms and concepts involved in one or more embodiments of this application will be explained.

[0018] A Field-Programmable Gate Array (FPGA) is a programmable logic chip that contains a large number of configurable logic units, memory, and interconnect resources, and allows for custom circuit functions through a hardware description language.

[0019] Imaging equipment refers to hardware devices that acquire images.

[0020] The target image refers to the image to be processed that is captured in real time by the imaging device and is either blurred or degraded.

[0021] The computational image size refers to the uniform size required for the target image and the operator when the FPGA performs frequency domain convolution operations.

[0022] A point spread function (PSF) image is an image used to describe the imaging effect of an imaging device on an ideal point light source (usually manifested as a bright spot with blurred edges), reflecting the blurring characteristics.

[0023] The initial forward operator is the matrix obtained after normalizing the point spread function image, which describes the degradation process of the image.

[0024] The initial backward operator refers to the backward operator generated by the host computer based on the point spread function image.

[0025] A valid flag is a single-bit hardware signal inside the FPGA used to identify the validity of the convolution result.

[0026] Image restoration refers to the technology that reverses the image degradation process through mathematical algorithms and hardware.

[0027] As chip manufacturing precision increases, the demand for image quality in optical defect detection is also rising. However, during operation, imaging equipment is affected by factors such as optical diffraction and lens aberrations, often resulting in blurred chip images and unclear edge details, leading to false positives and false negatives in subsequent defect detection. Therefore, image restoration has become a crucial step in improving the reliability of defect detection.

[0028] Currently, deconvolution algorithms are generally run in the graphics processing unit (GPU). By acquiring and utilizing the point spread function (PSF) of the imaging device to construct a degradation model, and then using the degradation model to compensate for the image information lost during the imaging process, image restoration can be achieved.

[0029] However, the software-based, GPU hardware-accelerated approach has certain limitations, as follows: First, GPUs are expensive and difficult to adapt to the needs of large-scale production lines (such as wafer defect detection).

[0030] Secondly, the computing efficiency of GPUs depends on the software driver version and the system operating environment. In multi-task concurrent scenarios in the detection process (such as simultaneously executing tasks such as image acquisition, preprocessing, and defect identification), fluctuations in computing power allocation are likely to occur, resulting in poor stability of image restoration.

[0031] Finally, image restoration, as a preprocessing step before detection, does not require excessive resources or time. However, software-based methods with GPU hardware acceleration have complex software processing logic and high computational resource requirements.

[0032] To address the aforementioned technical problems, this application provides an image restoration method and a computer program product. In the image restoration method provided in this application embodiment, the FPGA first receives and buffers the target image acquired in real-time by the imaging device, and calls the pre-buffered initial forward operator and initial backward operator. Through preprocessing to adapt the computational image size to the target image, the target forward operator and target backward operator are obtained. Then, through frequency domain convolution operation, error correction under effective flag control, and secondary convolution, the restored image is obtained. That is, real-time image restoration of the target image is achieved based on the FPGA's hardware parallel architecture, ensuring data synchronization and improving the accuracy and efficiency of image restoration.

[0033] For example, the image restoration method provided in this application can be applied to the production line of a semiconductor manufacturing company to perform real-time deblurring processing on images acquired from wafer surfaces and chip circuits. In practical applications, wafer images captured by imaging devices (such as high-resolution industrial cameras) are often blurred due to optical system diffraction, mechanical vibration, etc. The image restoration method of this application can predetermine the initial forward and backward operators for image restoration. Then, based on the computational image size of the target image, the FPGA preprocesses the initial forward and backward operators to obtain target forward and backward operators adapted to the computational image size. Finally, through frequency domain convolution, error correction under effective flag control, and secondary convolution, the restored image is output and stored in a data storage device. Operators can then perform various mathematical analyses or quality assessments, such as defect size measurement, yield statistics, and process parameter optimization.

[0034] It should be noted that the application scenarios described in the above embodiments of this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. Those skilled in the art will understand that with the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems. The image restoration method provided by the embodiments of this application can be applied to various application scenarios that require image deblurring and image restoration.

[0035] The image restoration method provided in the embodiments of this application is described below. In practical applications, the executing entity of the image restoration method in the embodiments of this application can be a terminal device, such as a desktop computer, a laptop computer, etc., or a remote device similar to a server. Of course, the executing entity in the embodiments of this application can also be a software entity, such as a client or software program installed on a terminal device. The specific type of executing entity corresponding to the technical solution provided in the embodiments of this application is not strictly limited here, and can be flexibly selected according to the actual application scenario and actual needs.

[0036] The following describes specific embodiments of the image restoration method, apparatus, electronic device, storage medium, and computer program product provided in this application. First, an image restoration method is described.

[0037] Figure 1 This is a schematic flowchart illustrating an image restoration method provided in an embodiment of this application. Figure 1 As shown, the method involves a real-time image restoration process performed by an FPGA, including steps S100 to S105.

[0038] S100: The FPGA receives and buffers the target image sent by the imaging device.

[0039] It should be noted that the FPGA needs to receive and buffer the target image sent by the imaging device to achieve real-time image restoration in subsequent steps. The efficiency of this image restoration is also related to the initial forward and backward operators that match the degradation characteristics of the imaging device. Due to the limited hardware resources of the FPGA, if the task of determining the initial forward and backward operators were delegated to the FPGA, calculations such as matrix transformations and frequency domain filtering would consume significant resources, reducing the efficiency of image buffering and image restoration. Therefore, to enable the FPGA to perform real-time image restoration calculations in subsequent steps, avoid the FPGA consuming additional hardware resources due to handling complex operator calculations, and ensure accurate matching between the operators and the degradation characteristics of the imaging device, this step delegates the task of determining the initial forward and backward operators to the preprocessing module, as follows: The imaging device images its point source (a light source in the imaging device that can generate ideal point signals) to obtain a point spread function image. The imaging device then sends this point spread function image to the preprocessing module. The preprocessing module receives the point spread function image and performs preprocessing on it (such as cropping the point spread function image based on the completeness of the point spread function information) to obtain the point spread function matrix. The point spread function matrix is ​​then normalized to obtain the initial forward operator (FP). The process of determining the Wiener-Butterworth initial inverse operator includes: flipping the point spread function image horizontally and vertically (time-domain flipping), and obtaining the frequency domain representation DFT(f) through a Fast Fourier Transform; determining the frequency domain inverse operator through a Fourier domain Wiener filter and a Fourier domain Butterworth filter; and then obtaining the initial inverse operator by performing an inverse Fourier transform and normalization on the frequency domain inverse operator.

[0040] Figure 2 This is a schematic diagram of a point spread function image acquired by an imaging device provided in an embodiment of this application.

[0041] The Fourier domain Wiener filter, the Fourier domain Butterworth filter, and the calculation formula for determining the Wiener-Butterworth filter are as follows: In equations (1), (2) and (3), For the Fourier domain Wiener filter; DFT(f) is the frequency domain result of the point spread function after time-domain flipping and discrete Fourier transform, reflecting the frequency domain characteristics of the PSF; It means The complex conjugate of is used to implement back projection operations in the frequency domain; It means The modulus square (energy spectrum) is used to describe the energy distribution of the PSF in the frequency domain; This is the regularization parameter, and its size can be set according to actual needs (e.g., Set to 0.05), used in At the same time, it is necessary to avoid the denominator approaching 0, which could lead to uncontrolled filter gain, and to balance image restoration sharpness with noise suppression. It refers to the Fourier domain Butterworth filter; This refers to the filter transition band parameter, which can be set according to actual needs; , This refers to the spatial frequency coordinates in the frequency domain; , It refers to the cutoff frequency, which is determined by the full width at half maximum (FWHM) of the PSF and is used to control the suppression threshold of high-frequency noise. This is the filter order, which can be set according to actual needs. It is used to control the steepness of the transition band between the passband and the stopband. The higher the order, the steeper the transition. For Wiener-Butterworth filters, by and Multiplying in the frequency domain yields a result that combines the adaptive noise suppression capability of Wiener filtering with the smooth frequency characteristics of Butterworth filtering. Perform an inverse Fourier transform to transform to the time domain, and obtain the initial backward operator through normalization.

[0042] Furthermore, the point spread function image reflects the optical characteristics of the imaging device. This application does not limit the specific method of preprocessing the point spread function image. It can be set according to actual needs. For example, in order to save FPGA computing resources, the brightest pixel in the point spread function image can be used as the center point to crop the point spread function image. It is only necessary to ensure that the cropped point spread function image includes complete point spread function information (which can be N*N, where N can be in the range of 9 to 19).

[0043] S101: The FPGA preprocesses the pre-cached initial forward operator and the initial backward operator according to the size of the operational image for image restoration, so as to obtain the target forward operator and the target backward operator that match the size of the operational image in the frequency domain dimension. The initial forward operator and the initial backward operator are determined based on the point spread function image of the imaging device.

[0044] In one or more embodiments of this application, in order to enable the operator to directly perform frequency domain convolution operations with the target image in the frequency domain in subsequent steps (the matrix sizes involved in the frequency domain convolution are consistent), and to adapt to the hardware parallel processing logic of the FPGA, reducing computational redundancy or accuracy loss caused by size mismatch, in this step, the FPGA needs to preprocess the initial forward operator and the initial backward operator based on the size of the operation image to restore the image, so as to achieve dimensional alignment between the operator and the target image.

[0045] Specifically, the FPGA reads the initial forward and backward operators from the cache, where both are N*N in size. Based on the FPGA's computing resources, the computational image size (M×M) is determined. This size can be set according to actual needs; to facilitate image restoration operations on the FPGA, M can be an integer power of 2, such as 512 or 1024. According to the computational image size, the initial forward and backward operators are padded with a padded dimension of MN to expand the padded operator size to M*M. By performing Fourier transforms and complex conjugation on the padded initial forward and backward operators, the target forward and backward operators are obtained and stored in the FPGA's on-chip cache for subsequent frequency domain convolution operations.

[0046] It should be noted that the target image is read according to the size of the computational image; that is, the real-time image processing begins when the cached image data meets the computational image size. This application does not limit the specific method of filling the initial forward and backward operators; it can be set according to actual needs, such as zero-filling. To ensure that the position of the valid data in the operators remains unchanged after filling, in one or more embodiments of this application, the FPGA can fill the right and bottom sides of the operators, including: synchronously filling the last column and last row of the pre-cached initial forward and backward operators according to the computational image size of the target image; and synchronously performing frequency domain transformation and conjugate processing on the filled initial forward and backward operators to obtain the target forward and target backward operators. In this embodiment, interference with the effective data of the operator can be avoided, ensuring the accuracy of the frequency domain characteristics. The padding method is compatible with the row-first storage architecture of FPGA. The padding process does not require adjusting the storage order of the original data. Only zero values ​​(or null values) need to be added at the end of each row and a few rows of each column, reducing the hardware overhead of data rearrangement. The conjugate processing of the backward operator is the mathematical basis for realizing back projection in the frequency domain, ensuring that the error correction factor in subsequent steps can accurately offset the effect of image degradation.

[0047] Figure 3 This is a schematic diagram illustrating the FPGA operator preprocessing flow provided in an embodiment of this application. Figure 3As shown, the operator preprocessing process includes steps S300 to S303, as follows: S300: Determine the size of the operational image for image restoration, the size of the initial forward operator, and the size of the initial backward operator.

[0048] S301: Fill the initial forward and backward operators to the size of the computation image.

[0049] S302: Perform frequency domain transformation and conjugation on the padded initial forward operator and initial backward operator to obtain the target forward operator and target backward operator.

[0050] S303: Cache target forward operator and target backward operator.

[0051] It should be noted that the Richardson-Lucy deconvolution algorithm (Richardson-Lucy deconvolution algorithm) is used to achieve this. The Lucy Deconvolution Algorithm (RLD) uses the following formula to perform image restoration: In equation (4), The restored image for the k-th iteration; The restored image for the (k+1)th iteration is based on Optimized restored image; The target image acquired by the imaging device; Forward projection operator (in this application, it is the initial forward operator after FPGA preprocessing); This refers to the backward projection operator (in this application, the initial backward operator after FPGA preprocessing). Because the initial backward operator in this application is optimized using both Wiener and Butterworth filters, the image restoration process of the target image only requires one iteration. Therefore, the calculation formula for image restoration of the target image using the FPGA can be simplified as follows: In equation (5), For the target image; To restore the image; firstly, through Simulate the degradation process of the target image; secondly, through comparison with... The ratio of degradation error is used to quantify the error and obtain the error correction factor; finally, based on... Convolution with the error correction factor maps the error information into a correction value, which is used to adjust the error information. The image was corrected to obtain the restored image.

[0052] This application divides the FPGA-executed image restoration process into two stages: the operator preprocessing stage mentioned above and the image processing stage from steps S104 to S106. Since the size of the output result of the convolution operation is inconsistent with the input operator / target image, in order to ensure the validity of the two convolution operations, this application divides the image processing stage into a first processing sub-stage (used to execute the process of determining the error correction factor based on the target forward operator and the target image in steps S104 to S105) and a second processing sub-stage (used to determine the process of determining the restored image based on the error correction factor, the target backward operator, and the target image in steps S106 to S105).

[0053] Figure 4 This is a schematic diagram illustrating the principle of image restoration provided in an embodiment of this application. Figure 4 As shown, the FPGA obtains the target forward operator CONJ [FFT (FP)] and the target backward operator CONJ [FFT (BP)] through operator preprocessing 400. When the target image is received, it is distributed to the first processing sub-stage 401 and the second processing sub-stage 402. In the first processing sub-stage, CONJ [FFT (FP)] is obtained through FP request and used to perform the error correction factor determination process, and the obtained error correction factor is cached. In the second processing sub-stage, CONJ [FFT (BP)] is obtained through BP request and used to perform the target image correction process, thereby obtaining the restored image.

[0054] S102: The FPGA calls the target forward operator, performs frequency domain convolution operation on the target forward operator and the target image to obtain a first convolution result, and generates a first valid flag of the first convolution result.

[0055] S103: Based on the first valid flag, the FPGA reads the cached target image and determines the error correction factor according to the target image and the first convolution result.

[0056] In one or more embodiments of this application, in order to accurately perform reverse compensation on the target image based on the error correction factor and the target backward operator in subsequent steps, thereby achieving degradation correction of the target image, in steps S104 and S105, after determining the first convolution result of the target forward operator and the target image, this application may use the first valid flag of the first convolution result as the valid flag of the division operation in the error correction factor calculation, to control the reading of the target image and ensure the timing synchronization of the FPGA in image restoration.

[0057] Specifically, the FPGA first reads the target forward operator (frequency domain dimension) and the cached target image (time domain dimension) in parallel from its on-chip memory resources. Then, it performs a Fast Fourier Transform on the target image to convert it into frequency domain data. In the frequency domain, it performs point-to-point multiplication (equivalent to time domain convolution) between the target forward operator and the target image to obtain a frequency domain convolution result. This result is then converted back to the time domain to obtain the first convolution result (a blurred image used to simulate the target image degradation process). Once the computation is complete and the output data is stable, the target image is read based on the first valid flag of the first convolution result. The first convolution result is retrieved, and a division operation is performed with the target image as the dividend and the first convolution result as the divisor. The result of this division operation is the error correction factor.

[0058] The calculation formulas for steps S104 and S105 are as follows: in, This is the error correction factor; The frequency domain data of the target image; Forward operator of target; For the target image, since only one iteration is required, In this application, it is also the target image.

[0059] It should be noted that the target image is sent to the FPGA in real time by the imaging device, while the frequency domain convolution operation requires computation time, and its first convolution result is output with a delay. To ensure the timing synchronization of the error correction factor, a first buffer unit can be independently allocated in the FPGA's storage resources to buffer the target image sent by the imaging device, thereby avoiding interruptions in the frequency domain convolution operation or error correction factor calculation due to unprepared image data. Therefore, in one or more embodiments of this application, this application can read the target image from the first buffer unit based on a first valid flag (indicating that the first convolution result has been generated) while performing the frequency domain convolution operation, ensuring the timing synchronization of the target image and the first convolution result, and avoiding the distortion of the error correction factor caused by asynchronous data. This includes: reading the target image from the first buffer unit based on the first valid flag; performing a dot division operation between the read target image and the first convolution result to determine the error correction factor; and buffering the target image read from the first buffer unit into a third buffer unit.

[0060] In this embodiment, in order to isolate the data streams of the first processing sub-stage and the second processing sub-stage and avoid read / write conflicts, the first cache unit is used to receive and cache the target image sent in real time by the imaging device, and the third cache unit serves as a transitional storage area between the two sub-stages to isolate the data stream sent in real time by the imaging device and the computational data stream of the second processing sub-stage, ensuring that the target image read in subsequent steps is stable and consistent, and avoiding interference from the real-time data stream.

[0061] S104: The FPGA performs frequency domain convolution operation on the target backward operator and the error correction factor to obtain a second convolution result, and generates a second valid flag of the second convolution result.

[0062] S105: The FPGA reads the cached target image based on the second valid flag, and determines the restored image according to the target image and the second convolution result.

[0063] In one or more embodiments of this application, in order to accurately correct the target image and obtain the restored image based on the degradation correction information contained in the error correction factor through a reverse compensation mechanism, in steps S106 and S107, after determining the second convolution result of the target backward operator and the error correction factor, this application may use the second valid flag of the second convolution result as the valid flag of the multiplication operation in the target image correction calculation, so as to control the reading of the target image and ensure the timing synchronization of the FPGA in image restoration.

[0064] Specifically, the FPGA reads the target backward operator (frequency domain dimension) and error correction factor (time domain dimension) from the on-chip cache. It then performs a Fast Fourier Transform on the error correction factor to convert it into frequency domain data. In the frequency domain, it performs point-to-point multiplication (equivalent to time domain convolution) between the target backward operator and the error correction factor to obtain a frequency domain convolution result. This result is then converted back to the time domain to obtain a second convolution result (which is the inverse compensation signal). Once the computation is complete and the output data is stable, the target image is read based on the second valid flag of the second convolution result. The second convolution result is then retrieved, and a multiplication operation is performed to obtain the restored image.

[0065] The calculation formulas for steps S106 and S107 are as follows: in, To restore the image; The frequency domain data is for the error correction factor; For the target backward operator; The target image.

[0066] It should be noted that due to the edge computing effect of convolution processing, the output size of the first processing sub-stage is (M-N+1)×(M-N+1), which is inconsistent with the size of the processed image (M*M). Therefore, in order to ensure the spatial integrity and size adaptability of the error correction factor before the frequency domain convolution operation, in one or more embodiments of this application, the FPGA can cache the error correction factor in the second cache unit according to the position of the error correction factor in the target image, wherein, due to the edge computing effect of convolution processing, the edge position in the target image has no corresponding output data in the error correction factor; and then read the error correction factor from the second cache unit according to the size of the processed image; and perform a frequency domain convolution operation on the target backward operator and the error correction factor read from the second cache unit to obtain the second convolution result. At the same time, in order to ensure that the data timing of the target image read from the third cache unit is synchronized with the second convolution result during the multiplication operation, in one or more embodiments of this application, the FPGA can read the target image from the third cache unit based on the second valid flag, and then perform a dot multiplication operation on the target image read from the third cache unit and the second convolution result to determine the restored image.

[0067] Figure 5 This is a schematic diagram illustrating the principle of image processing provided in an embodiment of this application. Figure 5 As shown, the image processing process includes a first processing sub-stage 401 and a second processing sub-stage 402. The first processing sub-stage is used to process... The second processing sub-stage is used to calculate the error correction factor. The restored image is generated. In the first processing sub-stage, a first buffer unit 500 is set up to synchronize the input timing of the division operation between the target image and the first convolution result. The error correction factor is cached in a second buffer unit 501. The error correction factor read from the second buffer unit has a size of M*M and is used... This indicates that, in the second processing sub-stage, a third buffer unit 502 is set up to achieve input timing synchronization of the multiplication operation between the target image and the second convolution result.

[0068] In the aforementioned image restoration method, a pipeline architecture of "operator preprocessing, frequency domain operation, error compensation, and restored image generation" is adopted. Combined with a hierarchical caching design and single-iteration optimization of the first, second, and third cache units, efficient and accurate restoration of blurred images is achieved. Specifically, the host computer amplifies the floating-point operators to adapt to the integer operation logic of the FPGA, avoiding the high resource consumption of floating-point processing. The FPGA first pads and performs FFT transformation on the operators to ensure they match the size of the processed image. Then, it simulates image degradation through frequency domain convolution, quantizes the error correction factor, and uses the optimized target backward operator to complete reverse compensation, outputting a clear image in a single iteration. Simultaneously, the first cache unit ensures stable reception and temporary storage of the target image, the second cache unit retains the spatial location and size adaptation of the error correction factor, and the third cache unit realizes the reuse and isolation of the target image. Combined with the timing control of effective flags, data conflicts and asynchronous distortion are avoided.

[0069] In step S100, this application does not limit the specific implementation form of the preprocessing module, which can be set according to actual needs. It can take the form of hardware and / or software with data receiving and processing capabilities, such as a host computer. That is, in one or more embodiments of this application, the FPGA is also used to receive and buffer the initial forward operator and initial backward operator sent by the host computer. The initial forward operator and initial backward operator are determined by the host computer based on the point spread function image before the FPGA is powered on.

[0070] In other words, in this application, the host computer performs the operator operation process. The host computer needs to receive the dot spread function image sent by the imaging device, and before the FPGA is powered on, it determines the initial forward operator and the initial backward operator based on the dot spread function image, and then sends the initial forward operator and the initial backward operator to the FPGA, so as to reduce the resource consumption of the FPGA.

[0071] In one or more embodiments of this application, in order to enable the FPGA to directly call the initial forward and initial backward operators in subsequent steps to perform the real-time image restoration process and reduce the latency of FPGA real-time image processing, the host computer can preprocess the determined initial forward and initial backward operators and send them to the FPGA for caching.

[0072] Specifically, the host computer can preprocess the initial forward operator and the initial backward operator (such as format conversion to adapt to the FPGA's integer operation logic), and then send the initial forward operator and the initial backward operator to the FPGA through a data transmission interface (such as Ethernet, PCIe, etc.) so that the FPGA can cache the initial forward operator and the initial backward operator in RAM resources, such as on-chip block random access memory (BRAM).

[0073] It should be noted that, due to the inherent limitations of FPGA hardware, it is not suitable for directly processing floating-point data (requiring additional floating-point arithmetic units, which introduces complexity and high latency). Therefore, to ensure the numerical accuracy of the initial forward and backward operators and improve computational efficiency, in one or more embodiments of this application, the host computer can also perform a floating-point to fixed-point data format conversion on the initial forward and backward operators. This application does not limit the specific method of this format conversion. For example, the numerical accuracy of the operators can be ensured by amplifying to the power of 2, so that the FPGA operation only needs to perform a shift operation to restore the data (the shift operation can be directly implemented through wiring logic, without the need for multipliers or dividers, resulting in low latency). Taking a 4096x amplification as an example, 4096 is 2 to the power of 12. Amplifying based on this factor is equivalent to shifting the binary decimal point of the floating-point data 12 bits to the right, thus ensuring 12 bits of effective precision, meeting the precision requirements of image restoration.

[0074] In addition, after obtaining the restored image in step S107, the restored image can be uploaded to the host computer for display or for downstream tasks such as defect detection.

[0075] Figure 6a A schematic diagram of the target image provided in the embodiments of this application; Figure 6b This is a schematic diagram of the restored image provided in an embodiment of this application. Based on Figure 6a and Figure 6b It can be observed that the image restoration method provided in this application can ensure the image restoration accuracy with just one iteration, thereby avoiding the resource consumption of multiple iterations.

[0076] Based on image restoration methods, this application also provides specific embodiments of an image restoration system.

[0077] like Figure 7 As shown in the figure, an image restoration system 700 provided in this application embodiment includes a host computer 700 and an FPGA 701. The host computer includes an initial operator calculation unit and an image display and detection unit; the FPGA is used to perform operator preprocessing on the initial operator (convolution kernel FFT calculation unit), cache the target operator and the first / second convolution results (convolution kernel FFT caching unit), perform convolution processing in the image restoration process (PSF deconvolution unit), and cache the target image data (image data cache). The solid lines in the figure are used to represent the data flow of the real-time processing of image restoration, and the dashed lines in the figure are used to represent the data flow of the operator calculation and preprocessing processes of image restoration.

[0078] The initial operator calculation unit is used to receive the point spread function image sent by the imaging device, and determine the initial forward operator and the initial backward operator based on the point spread function image; and send the preprocessed initial forward operator and the initial backward operator to the FPGA for buffering. The image display and detection unit is used to receive the restored image returned by the FPGA, and to display and detect (such as defect detection) based on the restored image.

[0079] The FPGA is used to: receive and buffer target images sent by the imaging device; preprocess the pre-buffered initial forward and backward operators according to the size of the operational image to obtain target forward and backward operators that match the operational image size in the frequency domain; invoke the target forward operator, perform frequency domain convolution operation on the target forward operator and the target image to obtain a first convolution result, and generate a first valid flag for the first convolution result; based on the first valid flag, read the buffered target image, and determine the error correction factor according to the target image and the first convolution result; perform frequency domain convolution operation on the target backward operator and the error correction factor to obtain a second convolution result, and generate a second valid flag for the second convolution result; based on the second valid flag, read the buffered target image, and determine the restored image according to the target image and the second convolution result.

[0080] Based on image restoration methods, this application also provides specific embodiments of an image restoration apparatus.

[0081] like Figure 8 As shown in the embodiment of this application, an image restoration device 800 is provided. The device 800 is applied to an FPGA and includes a first receiving module 801, an operator preprocessing module 802, a first convolution module 803, an error correction module 804, a second convolution module 805, and a restoration module 806.

[0082] The first receiving module 801 is used to receive and buffer the target image sent by the imaging device; The operator preprocessing module 802 is used to preprocess the pre-cached initial forward operator and initial backward operator according to the size of the operational image of the image restoration, so as to obtain the target forward operator and target backward operator that match the size of the operational image in the frequency domain dimension. The initial forward operator and the initial backward operator are determined based on the point spread function image of the imaging device. The first convolution module 803 is used to perform frequency domain convolution operation on the target forward operator and the target image to obtain a first convolution result and generate a first valid flag of the first convolution result; The error correction module 804 is used to read the cached target image based on the first valid flag, and determine the error correction factor based on the target image and the first convolution result. The second convolution module 805 is used to perform frequency domain convolution operation on the target backward operator and the error correction factor to obtain a second convolution result, and generate a second valid flag of the second convolution result; The restoration module 806 is used to read the cached target image based on the second valid flag, and determine the restored image based on the target image and the second convolution result.

[0083] In some embodiments, the operator preprocessing module is specifically used to: simultaneously fill the last column and last row of the pre-cached initial forward operator and the initial backward operator according to the computational image size of the target image; and simultaneously perform frequency domain transformation and conjugate processing on the filled initial forward operator and the initial backward operator to obtain the target forward operator and the target backward operator.

[0084] In some embodiments, the error correction module described above is specifically used for: the FPGA including a first cache unit, the first cache unit being used to cache the target image sent by the imaging device; reading the target image from the first cache unit based on the first valid flag; performing a dot division operation between the target image read from the first cache unit and the first convolution result to determine the error correction factor; and caching the target image read from the first cache unit into a third cache unit.

[0085] In some embodiments, the second convolution module is specifically configured to: cache the error correction factor in a second cache unit according to the position of the error correction factor in the target image; read the error correction factor from the second cache unit according to the size of the operation image; and perform frequency domain convolution operation on the target backward operator and the error correction factor read from the second cache unit to obtain a second convolution result.

[0086] In some embodiments, the restoration module is specifically used to: read the target image from the third cache unit based on the second valid flag; and perform a dot product operation between the target image read from the third cache unit and the second convolution result to determine the restored image.

[0087] In some embodiments, the above-described apparatus further includes an operator operation module, specifically configured to: receive and cache the initial forward operator and the initial backward operator sent by a host computer before preprocessing the pre-cached initial forward operator and the initial backward operator according to the size of the operational image restored by the image to obtain the target forward operator and the target backward operator that match the size of the operational image in the frequency domain dimension, wherein the initial forward operator and the initial backward operator are determined by the host computer based on the point spread function image before the FPGA is powered on.

[0088] In some embodiments, the operator operation module described above can also be used to: obtain the initial forward operator and the initial backward operator by the host computer after determining them based on the point spread function image, and then perform a format conversion from floating-point data to fixed-point data.

[0089] Based on an image restoration method, this application also provides a specific embodiment of an image restoration device.

[0090] Figure 9 A schematic diagram of the hardware structure of an image restoration device provided in an embodiment of this application is shown.

[0091] The image restoration device may include a processor 901 and a memory 902 storing computer program instructions.

[0092] Specifically, the processor 901 may include a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement the embodiments of this application.

[0093] Memory 902 may include mass storage for data or instructions. For example, and not limitingly, memory 902 may include a hard disk drive (HDD), floppy disk drive, flash memory, optical disk, magneto-optical disk, magnetic tape, or Universal Serial Bus (USB) drive, or a combination of two or more of these. Where appropriate, memory 902 may include removable or non-removable (or fixed) media. Where appropriate, memory 902 may be internal or external to the integrated gateway disaster recovery device. In a particular embodiment, memory 902 is non-volatile solid-state memory.

[0094] The processor 901 reads and executes computer program instructions stored in the memory 902 to implement any of the image restoration methods in the above embodiments.

[0095] In one example, the electronic device may also include a communication interface 903 and a bus 910. Wherein, as... Figure 9 As shown, the processor 901, memory 902, and communication interface 903 are connected through bus 910 and complete communication with each other.

[0096] The communication interface 903 is mainly used to realize communication between various modules, devices, units and / or equipment in the embodiments of this application.

[0097] Bus 910 includes hardware, software, or both, that couples the components of the electronic device together. For example, and not limitingly, the bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), HyperTransport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an Infinite Bandwidth Interconnect, a Low Pin Count (LPC) bus, a memory bus, a Microchannel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local (VLB) bus, or other suitable buses, or combinations of two or more of these. Where appropriate, bus 910 may include one or more buses. Although specific buses are described and illustrated in embodiments of this application, any suitable bus or interconnect is contemplated herein.

[0098] In addition, in conjunction with the image restoration method described in the above embodiments, this application embodiment can provide a computer storage medium for implementation. The computer storage medium stores computer program instructions; when these computer program instructions are executed by a processor, they implement any of the image restoration methods described in the above embodiments.

[0099] In addition, in conjunction with the image restoration method in the above embodiments, this application embodiment can provide a computer program product for implementation. When the instructions in the computer program product are executed by the processor of an electronic device, the electronic device performs an image restoration method as provided in any aspect of the above embodiments of this application.

[0100] It should be clarified that this application is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of this application is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of this application.

[0101] The functional blocks shown in the above-described structural diagram can be implemented as hardware, software, firmware, or a combination thereof. When implemented in hardware, they can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this application are programs or code segments used to perform the required tasks. Programs or code segments can be stored on a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried on a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, fiber optic media, radio frequency (RF) links, etc. Code segments can be downloaded via computer networks such as the Internet, intranets, etc.

[0102] It should also be noted that the exemplary embodiments mentioned in this application describe methods or systems based on a series of steps or apparatus. However, this application is not limited to the order of the above steps; that is, the steps can be performed in the order mentioned in the embodiments, or in a different order, or several steps can be performed simultaneously.

[0103] The aspects of this disclosure have been described above with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block in the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that these instructions, executable via the processor of the computer or other programmable data processing apparatus, enable the implementation of the functions / actions specified in one or more blocks of the flowchart illustrations and / or block diagrams. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field-programmable logic circuit. It is also understood that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can also be implemented by special-purpose hardware performing the specified functions or actions, or can be implemented by a combination of special-purpose hardware and computer instructions.

[0104] The above description is merely a specific implementation of this application. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, modules, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. It should be understood that the protection scope of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the protection scope of this application.

Claims

1. An image restoration method, characterized in that, Applied to FPGA, the method includes: Receive and buffer target images sent by imaging devices; Based on the operational image size for image restoration, pre-cached initial forward and backward operators are preprocessed to obtain target forward and backward operators that match the operational image size in the frequency domain. The initial forward operator is obtained by preprocessing the point spread function image of the imaging device to obtain a point spread function matrix, and then normalizing the point spread function matrix. The initial backward operator is obtained by performing a time-domain flip on the point spread function image, performing a fast Fourier transform to obtain a frequency domain representation, and then determining the frequency domain inverse operator through a Fourier domain Wiener filter and a Fourier domain Butterworth filter, and finally performing an inverse Fourier transform and normalization on the frequency domain inverse operator. The target forward operator and the target image are subjected to frequency domain convolution operation to obtain a first convolution result, and a first valid flag of the first convolution result is generated. Based on the first valid flag, the cached target image is read, and the target image is divided by the first convolution result to determine the error correction factor. The target backward operator and the error correction factor are subjected to frequency domain convolution operation to obtain a second convolution result, and a second valid flag of the second convolution result is generated. Based on the second valid flag, the cached target image is read, and the target image and the second convolution result are multiplied by a dot product to determine the restored image.

2. The method as described in claim 1, characterized in that, Based on the computational image size of the target image, the pre-cached initial forward operator and initial backward operator are preprocessed to obtain target forward operator and target backward operator that match the computational image size in the frequency domain dimension, including: Based on the operational image size of the target image, the last column and last row of the pre-cached initial forward operator and the initial backward operator are synchronously filled; The initial forward operator and the initial backward operator after filling are simultaneously subjected to frequency domain transformation and conjugation processing to obtain the target forward operator and the target backward operator.

3. The method as described in claim 1, characterized in that, The FPGA includes a first cache unit, which is used to cache the target image sent by the imaging device; Based on the first valid flag, the cached target image is read, and a dot division operation is performed between the target image and the first convolution result to determine the error correction factor, including: Based on the first valid flag, the target image is read from the first cache unit; The target image read from the first cache unit is divided by the first convolution result to determine the error correction factor, and the target image read from the first cache unit is cached in the third cache unit.

4. The method as described in claim 1, characterized in that, Performing a frequency domain convolution operation on the target backward operator and the error correction factor yields a second convolution result, including: The error correction factor is cached in the second cache unit according to its position in the target image; The error correction factor is read from the second cache unit according to the calculated image size; The target backward operator is subjected to frequency domain convolution with the error correction factor read from the second cache unit to obtain the second convolution result.

5. The method as described in claim 3, characterized in that, Based on the second valid flag, the cached target image is read, including: The target image is read from the third cache unit based on the second valid flag.

6. The method according to claim 1, characterized in that, Before preprocessing the pre-cached initial forward and backward operators according to the size of the operational image after image reconstruction to obtain the target forward and target backward operators that match the operational image size in the frequency domain dimension, the method further includes: The system receives and buffers the initial forward operator and the initial backward operator sent by the host computer, wherein the initial forward operator and the initial backward operator are determined by the host computer based on the point spread function image before the FPGA is powered on.

7. The method as described in claim 6, characterized in that, The initial forward operator and the initial backward operator are obtained by the host computer after determining them based on the point spread function image, and then performing a format conversion from floating-point data to fixed-point data.

8. An image restoration system, characterized in that, The system includes a host computer and an FPGA, including: The host computer is used to: receive a point spread function image sent by the imaging device; preprocess the point spread function image of the imaging device to obtain a point spread function matrix, and normalize the point spread function matrix to obtain an initial forward operator; perform a time-domain flip on the point spread function image and then perform a fast Fourier transform to obtain a frequency domain representation, and then determine the frequency domain inverse operator through a Fourier domain Wiener filter and a Fourier domain Butterworth filter, and perform an inverse Fourier transform and normalization on the frequency domain inverse operator to obtain an initial backward operator; and send the preprocessed initial forward operator and the initial backward operator to the FPGA for buffering. The FPGA is used to: receive and cache the target image sent by the imaging device; preprocess the pre-cached initial forward operator and the initial backward operator according to the operational image size for image restoration, to obtain a target forward operator and a target backward operator that match the operational image size in the frequency domain dimension; invoke the target forward operator, perform a frequency domain convolution operation on the target forward operator and the target image to obtain a first convolution result, and generate a first valid flag for the first convolution result; based on the first valid flag, read the cached target image, and perform a dot division operation on the target image and the first convolution result to determine an error correction factor; perform a frequency domain convolution operation on the target backward operator and the error correction factor to obtain a second convolution result, and generate a second valid flag for the second convolution result; based on the second valid flag, read the cached target image, and perform a dot multiplication operation on the target image and the second convolution result to determine the restored image.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer program instructions, which, when executed by a processor, implement the image restoration method as described in any one of claims 1-7.

10. A computer program product, characterized in that, When the instructions in the computer program product are executed by the processor of the electronic device, the electronic device performs the image restoration method as described in any one of claims 1-7.

Images