Method for determining an image restoration model, image restoration method and device

Abstract

The present disclosure relates to the technical field of image processing, and particularly relates to a method for determining an image restoration model, an image restoration method and device, which comprises: acquiring a reference multi-focus image and an initial model, wherein the reference multi-focus image comprises multiple reference images with different focus positions; determining full-focus label data corresponding to the reference multi-focus image according to the multiple reference images with different focus positions; determining an input image by using a point spread function according to the full-focus label data; and training the initial model based on the input image and the full-focus label data to obtain an image restoration model. The technical scheme of the embodiment of the present disclosure improves the image restoration quality and realism, and accelerates the restoration speed.

Description

Technical Field

[0001] This disclosure relates to the field of image processing technology, and more specifically, to a method for determining an image restoration model, an image restoration method and apparatus, a computer-readable storage medium, and an electronic device. Background Technology

[0002] By adding a phase mask to the aperture plane of the optical imaging system for wavefront encoding, the PSF of the optical system becomes insensitive to defocus. Then, digital image processing technology is used to process and restore the encoded image to obtain a clear image with a large depth of field.

[0003] However, existing digital image processing technologies have poor image restoration quality and weak realism when decoding encoded blurred images. The image restoration time is long, the restoration results cannot be displayed in real time, and the computational memory usage is large.

[0004] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of this disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0005] The purpose of this disclosure is to provide a method for determining an image restoration model, an image restoration method and apparatus, a computer-readable storage medium and an electronic device, thereby improving the quality and realism of image restoration to at least a certain extent and accelerating the restoration speed.

[0006] According to a first aspect of this disclosure, a method for determining an image restoration model is provided, comprising: acquiring a reference multi-focus image and an initial model, wherein the reference multi-focus image includes multiple reference images with different focus positions; determining full-focus label data corresponding to the reference multi-focus image based on the multiple reference images with different focus positions; determining an input image based on the full-focus label data using a point spread function; and training the initial model based on the input image and the full-focus label data to obtain an image restoration model.

[0007] According to a second aspect of this disclosure, an image restoration method is provided, comprising: acquiring an initial actual captured image; acquiring an image restoration model, wherein the image restoration model is determined according to the image restoration model determination method described above; and using the image restoration model to perform image restoration on the initial actual captured image to obtain a target large depth-of-field image.

[0008] According to a third aspect of this disclosure, an apparatus for determining an image restoration model is provided, comprising: a first acquisition module for acquiring a reference multi-focus image and an initial model, wherein the reference multi-focus image includes multiple reference images with different focus positions; a first determination module for determining full-focus label data corresponding to the reference multi-focus image based on the multiple reference images with different focus positions; a second determination module for determining an input image based on the full-focus label data using a point spread function; and a model training module for training the initial model based on the input image and the full-focus label data to obtain an image restoration model.

[0009] According to a fourth aspect of this disclosure, an image restoration apparatus is provided, comprising: a second acquisition module for acquiring an initial actual captured image; a third acquisition module for acquiring an image restoration model; and an image restoration module for using the image restoration model to perform image restoration on the initial actual captured image to obtain a target large depth-of-field image; wherein the image restoration model is determined according to the method for determining the image restoration model described in any of the preceding claims.

[0010] According to a fifth aspect of this disclosure, a computer-readable medium is provided having a computer program stored thereon, which, when executed by a processor, implements the method described above.

[0011] According to a sixth aspect of this disclosure, an electronic device is provided, characterized in that it includes: one or more processors; and a memory for storing one or more programs, which, when executed by the one or more processors, cause the one or more processors to implement the method described above.

[0012] This disclosure provides an embodiment of an image restoration model determination method, which involves acquiring a reference multi-focus image and an initial model. The reference multi-focus image includes multiple reference images with different focus positions. Full-focus label data corresponding to the reference multi-focus image is determined based on the multiple reference images with different focus positions. An input image is determined using a point spread function based on the full-focus label data. The initial model is trained based on the input image and the full-focus label data to obtain an image restoration model. Compared to existing technologies, this image restoration model improves the speed of image restoration. Furthermore, in determining the model, the full-focus label data corresponding to the reference multi-focus image is determined using multiple reference images with different focus positions, and the input image is determined using a point spread function based on the full-focus label data. The neural network has good noise resistance, improving the image restoration quality and realism. Moreover, the deep learning algorithm in the model has fast inference speed, low memory usage, low heat generation, and a smooth shooting process without lag, greatly improving the user's shooting experience.

[0013] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0014] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure. It is obvious that the drawings described below are merely some embodiments of this disclosure, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort. In the drawings:

[0015] Figure 1 A schematic diagram of an exemplary system architecture to which embodiments of the present disclosure may be applied is shown;

[0016] Figure 2 A flowchart illustrating a method for determining an image restoration model in an exemplary embodiment of the present disclosure is shown schematically.

[0017] Figure 3 This schematic diagram illustrates the structure of a generator according to an exemplary embodiment of the present disclosure;

[0018] Figure 4 This schematic diagram illustrates the structure of a discriminator in an exemplary embodiment of the present disclosure;

[0019] Figure 5 This schematic diagram illustrates the structure of a residual module in an exemplary embodiment of the present disclosure;

[0020] Figure 6 The diagram schematically illustrates the data flow of a method for determining an image restoration model in an exemplary embodiment of the present disclosure.

[0021] Figure 7 A flowchart illustrating an image restoration method according to an exemplary embodiment of the present disclosure is shown schematically.

[0022] Figure 8 This schematic diagram illustrates a data flow chart of a mobile terminal in an exemplary embodiment of the present disclosure;

[0023] Figure 9 This schematic diagram illustrates the composition of the apparatus for determining the image restoration model in an exemplary embodiment of the present disclosure.

[0024] Figure 10 This schematic diagram illustrates the composition of an image restoration apparatus in an exemplary embodiment of the present disclosure.

[0025] Figure 11 A schematic diagram of an electronic device to which embodiments of the present disclosure may be applied is shown. Detailed Implementation

[0026] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, they are provided so that this disclosure will be more comprehensive and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0027] Furthermore, the accompanying drawings are merely illustrative of this disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted. Some block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0028] In related technologies, a phase mask is added to the aperture plane of an optical imaging system for wavefront encoding, making the optical system's PSF insensitive to defocus. Digital image processing technology is then used to process and restore the encoded image to obtain a clear image with a large depth of field. However, when traditional digital image processing technology decodes the blurred encoded image, on the one hand, the image restoration algorithm is easily affected by noise, resulting in poor image restoration quality and weak realism; on the other hand, traditional image processing methods involve a large amount of computation, resulting in a long image restoration time, inability to display restoration results in real time, and a large amount of computing memory usage. They are usually processed on a PC and are difficult to deploy directly to lightweight mobile phone models.

[0029] Figure 1 A schematic diagram of the system architecture is shown. System architecture 100 may include a terminal 110 and a server 120. The terminal 110 may be a smartphone, tablet, desktop computer, laptop, or other terminal device. The server 120 generally refers to the backend system providing image restoration-related services in this exemplary embodiment, and may be a single server or a cluster of multiple servers. The terminal 110 and server 120 can be connected via wired or wireless communication links for data exchange.

[0030] In one implementation, the image restoration model determination method described above can be executed by server 120. For example, after a user takes an image using terminal 110 or selects a reference multi-focus image from the album of terminal 110, terminal 110 uploads the reference multi-focus image to server 120, and server 120 uses the reference multi-focus image to train an image restoration model and returns the image restoration model to terminal 110.

[0031] The image restoration method described above can be executed by the terminal 110. For example, after a user takes a picture using the terminal 110 or selects an initial actual picture from the album of the terminal 110 and loads the image restoration model described above, the terminal 110 performs image restoration on the picture and outputs a target image with a large depth of field.

[0032] Exemplary embodiments of this disclosure also provide an electronic device for performing the above-described method for determining the image restoration model and the above-described image restoration method. The electronic device may be the terminal 110 or the server 120. Generally, the electronic device may include a processor and a memory, the memory for storing executable instructions of the processor, and the processor configured to perform the above-described method for determining the image restoration model and the above-described image restoration method by executing the executable instructions.

[0033] The following is combined with Figure 2 The method for determining the image restoration model in this exemplary embodiment will be described. Figure 2 An exemplary flow illustrating the method for determining this image restoration model is shown, which may include:

[0034] Step S210: Obtain a reference multi-focus image and an initial model, wherein the reference multi-focus image includes multiple reference images with different focus positions;

[0035] Step S220: Determine the full-focus label data corresponding to the reference multi-focus image based on the multiple reference images with different focus positions;

[0036] Step S230: Determine the input image using the point spread function based on the full-focus label data;

[0037] Step S240: Train the initial model based on the input image and the full-focus label data to obtain the image restoration model.

[0038] Based on the above method, compared with existing technologies, on the one hand, an image restoration model is provided, which can improve the speed of image restoration compared with existing image restoration methods. Furthermore, when determining the model, multiple reference images with different focus positions are used to determine the full-focus label data corresponding to the reference multi-focus images; and the input image is determined using the point spread function based on the full-focus label data. The neural network has good noise resistance, improving the image restoration quality and realism. Moreover, the deep learning algorithm in the model has a fast inference speed, low memory usage, low heat generation, and a smooth shooting process without lag, which can greatly improve the user's shooting experience.

[0039] The following is about Figure 2 Each step in the process will be explained in detail.

[0040] refer to Figure 2 In step S210, a reference multi-focus image and an initial model are obtained, wherein the reference multi-focus image includes multiple reference images with different focus positions.

[0041] In one exemplary embodiment of this disclosure, the aforementioned reference multi-focus image may include multiple reference images at different focus positions. When acquiring the aforementioned reference multi-focus image, a preset focus range can be determined first, and then multiple reference images at different focus positions can be acquired at preset intervals. The aforementioned preset focus range may be 300μm, 500μm, etc., or can be customized according to user needs. Similarly, the aforementioned preset interval may be 30μm, 50μm, etc., or can be customized according to user needs. No specific limitation is made in this exemplary embodiment.

[0042] For example, if the preset focus range is 300μm and the preset interval is 30μm, then the above reference multi-focus image includes 11 reference images at different focus positions.

[0043] In this example implementation, the initial model can be a GAN model or other network structures, and no specific limitation is made in this example implementation.

[0044] In step S220, the full-focus label data corresponding to the reference multi-focus image is determined based on the multiple reference images with different focus positions.

[0045] In this example implementation, after obtaining the above-mentioned reference multi-focus image, multiple reference images in the above-mentioned reference multi-focus image can be fused using the Image Stack Fusion technique to obtain a clear image with a large depth of field (all-in-focus) as the above-mentioned all-focus label data.

[0046] In step S230, the input image is determined using the point spread function based on the fully focused label data.

[0047] In this example embodiment, the image restoration model is applied to a mobile terminal including a mobile phone optical system with a phase mask. Therefore, after obtaining the above-mentioned full-focus tag data, the PSF (Point Spread Function) of the mobile phone optical system with a phase mask can be convolved using the above-mentioned full-focus tag data to obtain intermediate images corresponding to multiple depths, wherein the depth of the above-mentioned intermediate images is the same as the depth of the above-mentioned reference images.

[0048] After obtaining the aforementioned intermediate images, the average value of multiple intermediate images can be calculated, and the 3D PSF information can be fused into the 2D image to obtain a final input image. In another implementation, a weighted average can be performed on multiple intermediate images to fuse the 3D PSF information into the 2D image, ultimately obtaining a final input image. The weight settings can be customized according to user needs, and are not specifically limited in this example implementation summary.

[0049] In one example implementation of this exemplary embodiment, after opening the input image, the input image and its corresponding full-focus label data can be preprocessed by cropping and registration. Both the full-focus label data and the input image can be segmented into multiple images, then matched, and the matched images are input into the initial model for training.

[0050] In step S240, the initial model is trained based on the input image and the full-focus label data to obtain the image restoration model.

[0051] In this example implementation, refer to Figure 3 As shown, the initial model described above can be a GAN model, which can include a generator and a discriminator. For the generator, refer to... Figure 4 As shown, the U-Net architecture can be used to fuse shallow and deep features, and nine residual modules can be used to enhance the feature transformation capability of the neural network. For the discriminator, PatchGAN with a receptive field of 70 is used, which helps to restore high-frequency details of the image and makes the generated image more realistic.

[0052] The generator may include a first convolution module 302, a second convolution module 312, a first downsampling module 303, a second downsampling module 304, a third downsampling module 305, a fourth downsampling module 306, a first upsampling module 308, a second upsampling module 309, a third upsampling module 310, a fourth upsampling module 311, and nine residual modules 307. The input image 301 is input to the first convolution module 302, which transforms the 3-channel image (HW3) into a 64-channel image (HW64). The image then passes through the first downsampling module 303, the second downsampling module 304, the third downsampling module 305, and the fourth downsampling module 306 in sequence to obtain an image with 512 channels (H / 16.W / 16.512), where the length and width are both one-sixteenth of the input image 301. That is, the image is downsampled by the first downsampling module 303, the second downsampling module 304, the third downsampling module 305, and the fourth downsampling module 306 in sequence. Then, after passing through the above nine residual modules 307, as well as the above first upsampling module 308, second upsampling module 309, third upsampling module 310, and fourth upsampling module 311, a 64-channel image (HW64) is obtained, and then after passing through the second convolution module 312, the output image 313 (HW3) is obtained.

[0053] The first convolution module 302 is connected to the second convolution module 312, the first downsampling module 303 is connected to the fourth upsampling module 311, the second downsampling module 304 is connected to the third upsampling module, the third downsampling module 305 is connected to the second upsampling module 309, and the fourth downsampling module 306 is connected to the first upsampling module 308.

[0054] In this example implementation, the upsampling factor is the same as the downsampling factor, and the specific sampling factor can be customized according to needs. In this example implementation, no specific limitation is made.

[0055] Refer to this example implementation Figure 4 As shown, the output image 313 can be input into the discriminator to obtain a discrimination result 406, and then the generator can be updated based on the discrimination result 406. Specifically, the output image 313 passes through the fifth downsampling module 401, the sixth downsampling module 402, and the seventh downsampling module 403 to obtain an image with a length and width of one-eighth of the input image 301 and 512 channels (H / 8.W / 8.512). Then, it passes through the third convolution module 404 and the fourth convolution module 405 to obtain the image to be discriminated. Finally, each pixel of the image to be discriminated is discriminated to obtain the discrimination result 406, which updates the output image 313. The model is stopped from being updated when the output image 313 converges to be the same as the fully focused label data.

[0056] Reference Figure 5 As shown, a residual module 307 includes two convolution modules, namely a fifth convolution module 501 and a sixth convolution module 502. The input of the fifth convolution module 501 and the output of the sixth convolution module 502 are fused and then input to the next residual module 307 or the aforementioned first upsampling module 308.

[0057] In this example implementation, when training the initial model, the minimum mean squared error loss, generative adversarial loss, and perceptual loss between the output image and the fully focused label data can be calculated. The parameters of the generative adversarial network are adjusted based on the minimum mean squared error loss, generative adversarial loss, and perceptual loss to obtain the image restoration model. That is, the loss function used includes three parts: the generative adversarial function, the minimum mean squared error loss, and the perceptual loss function. Specifically, the loss function can be:

[0058] L total =λ1L GAN (G,D)+λ2L MSE (G)+λ3L Perceptual (G)

[0059] L GAN (G,D)=E x,y～pdata(x,y) [logD(x,y)]+E x～pdata(x) [log(1-D(x,G(x)))]

[0060] L MSE (G)=E x,y～pdata(x,y) [||yG(x)||2]

[0061] L Perceptual (G)=E x,y～pdata(x,y) [||VGG(y)-VGG(G(x))||2]

[0062] Where: L total L is the overall loss function; GAN For the adversarial loss function; L MSE L is the minimum mean square error loss function; perceptual λ1, λ2, λ3 are the perceptual loss function; λ1, λ2, λ3 are the penalty coefficients corresponding to different loss function terms; G is the generator; D is the discriminator; x is the input image; y is the fully focused label data (ground-truth); VGG stands for "VGG Convolutional Neural Network".

[0063] In this example implementation, the minimum mean squared error loss (MSE Loss) is used as the basic loss function, and GAN Loss is used to improve the realism of the image and enrich the details of the recovered image. In order to better match the visual effect, perceptual loss is used to optimize the neural network.

[0064] In this example implementation, the cropped and calibrated image pairs are used as the input and ground truth of the initial model, respectively. That is, the segmented input image and the corresponding segmented fully focused label data are used as the input and ground truth of the initial model. The input image is processed by a generator with a U-Net structure to obtain the output image. The output image and the ground truth are respectively fed into the discriminator. The discriminator judges the generator's output as Fake and the ground truth as True. Through the adversarial interaction between the generator and the discriminator, the parameters of the initial model are optimized with the goal of minimizing the loss function between the ground truth and the output value. Finally, the optimized model parameters are saved.

[0065] In this example implementation, refer to Figure 6 As shown, taking the preset focus range of 300μm and the preset interval of 30μm as an example, and the reference multi-focus image 601 including 11 reference images at different focus positions, the method for determining the image restoration model is described in detail. The 11 reference images at different focus positions are fused using the Image StackFusion module 602 to obtain full-focus label data 603. Then, the full-focus label data 603 is passed through the PSF convolution module 604, and the PSF of the mobile phone optical system with a phase mask is used to obtain intermediate images 605 corresponding to 11 depths. The 11 intermediate images are then passed through the averaging fusion module 606 to obtain the input image 607. Then, the full-focus label data and the input are cropped and paired 608 and input into the initial model 609 to train the initial model.

[0066] In this example implementation, when training the initial model, multiple reference images with different focus positions can be used to train the initial model so that the obtained image restoration model has higher accuracy.

[0067] In one disclosed example implementation, after obtaining the aforementioned image restoration model, the image obtained by the model can be evaluated. Specifically, the peak signal-to-noise ratio (PSNR), structural similarity, and perceptual loss metric between the output image and the fully focused label data can be calculated. Specifically, the PSNR can first be calculated, and then:

[0068]

[0069] Where x is the output image predicted by the neural network; y is the clear image of the target (ground-truth); m and n represent the height and width of the image, respectively.

[0070] Next, structural similarity (SSIM) is calculated:

[0071]

[0072] Where, μ x ,μ y Let σ be the mean of x and y, respectively. x ,σ y Let σ be the standard deviation of x and y, respectively. xy Let x be the covariance between x and y; C1 and C2 are constants.

[0073] Finally, the perceptual loss metric (LPIPS) is calculated:

[0074]

[0075] Where, " represents the perceptual distance between two images (the smaller the value, the more similar the two images); 1 represents the number of layers in the neural network (which can be AlexNet, VGG, etc.), and H l W l These represent the height and width of the feature map obtained after the image passes through the first layer of the neural network, respectively; w l This represents the weight coefficients of the first layer; These represent the feature maps obtained after image x and image x0 pass through the first layer of the neural network, respectively.

[0076] The peak signal-to-noise ratio, structural similarity, and perceptual loss metrics are used as evaluation results. After obtaining these evaluation results, it is determined whether they meet preset conditions. If they do not meet the preset conditions, the image restoration model is updated; if they do meet the conditions, training of the image restoration model is stopped. The values ​​of each parameter in the preset conditions can be customized by the user according to different scenarios and needs, and are not specifically limited in this example implementation.

[0077] In summary, this exemplary embodiment, compared to the prior art, provides an image restoration model that improves the speed of image restoration compared to existing image restoration methods. Furthermore, when determining the model, multiple reference images with different focus positions are used to determine the full-focus label data corresponding to the reference multi-focus images; and the input image is determined using the point spread function based on the full-focus label data. The neural network has good noise resistance, improving the image restoration quality and realism. Furthermore, the deep learning algorithm in the model has fast inference speed, low memory usage, low heat generation, and a smooth shooting process without lag, greatly improving the user's shooting experience. The minimum mean squared error loss (MSE Loss) is used as the basic loss function, and GAN Loss is used to further improve the realism of the image and enrich the details of the restored image. To better match the visual effect, perceptual loss is also used to optimize the neural network.

[0078] This disclosure also provides an image restoration method, referring to... Figure 7 As shown, image restoration methods may include:

[0079] Step S710: Acquire the initial actual captured image;

[0080] Step S720: Obtain the image restoration model;

[0081] Step S730: Use an image restoration model to restore the initial actual captured image to obtain a target large depth-of-field image.

[0082] The image restoration model described above is determined based on the image restoration model method described above.

[0083] Based on the above methods, compared with existing technologies, the images restored using image restoration models have richer details and a more realistic feel, such as mitigating artifacts caused by image contour edges. Furthermore, neural networks have better noise resistance; noise introduced during actual mobile phone shooting often interferes with traditional image restoration processes, leading to a decrease in image quality. Deep learning-based algorithms, however, can often maintain image quality consistent with noise-free conditions. Finally, traditional deconvolution algorithms for image restoration are typically iterative and slow, while deep learning algorithms offer faster inference speeds, lower memory usage, less heat generation, and smoother, more lag-free shooting, significantly improving the user experience when taking photos.

[0084] The steps described above are explained in detail below.

[0085] In step S710, the initial actual captured image is acquired;

[0086] In step S720, the image restoration model is obtained;

[0087] In one example embodiment of this disclosure, the initial actual captured image may be captured by a mobile terminal, and the camera module in the mobile terminal includes a phase mask. The initial actual captured image may also be a historical image stored in the mobile terminal, and the historical image is also captured by a camera module equipped with a phase mask when it is acquired.

[0088] In this example embodiment, the image restoration method described above can be executed by a mobile terminal. After acquiring the initial actual captured image, image signal processing can be performed on the initial actual captured image to obtain RGB or YUV data that can be used for the image restoration model. Specifically, refer to... Figure 8 As shown, the ISP processor 802 first processes the initial captured image to capture image statistics that can be used to determine one or more control parameters of the Camera Sensor 801, including images in YUV or RGB format. The Camera Sensor 801 may include a color filter array (such as a Bayer filter) and corresponding photosensitive units. The Camera Sensor can acquire the light intensity and wavelength information captured by each photosensitive unit and provide a set of initial captured images that can be processed by the ISP processor. After processing the initial captured image data, the ISP processor obtains images in YUV or RGB format and sends them to the CPU 805. Specifically, when processing the initial captured images, the ISP processor 802 processes the initial captured images pixel by pixel with a bit depth of 16 bits per image pixel.

[0089] In this example embodiment, the network architecture 803 and network parameters 804 of the above-mentioned image restoration model can be obtained. Then, when the above-mentioned image restoration operation is started, the network architecture and the network parameters 804 are used to obtain the image restored by the image restoration model and display it on the display 806. Then, the image is passed through the encoder 807 and stored in the memory 808.

[0090] In one exemplary embodiment of this disclosure, the image restoration model can be a lightweight model, or a large model can be compressed and distilled to obtain a lightweight model structure, saving computation time and computation parameters, saving memory, improving shooting speed, making the shooting process smoother, and further enhancing the user shooting experience.

[0091] In step S730, the initial actual captured image is restored using an image restoration model to obtain a target large depth-of-field image.

[0092] After obtaining the image restoration model and the initial actual captured image, the initial actual captured image can be input into the image restoration model to obtain the target large depth-of-field image, wherein the sharpness of the initial actual captured image is less than the sharpness of the target large depth-of-field image.

[0093] It should be noted that the above image restoration method is not limited to mobile terminals, but can also be applied to actual fluorescence microscopes, using larger neural network models to ensure better performance, pursue greater depth of field exploration, capture neuronal images of various cells, pathological slide images, and promote the development of the biomedical field.

[0094] It should be noted that the above figures are merely illustrative of the processes included in the method according to exemplary embodiments of this disclosure, and are not intended to be limiting. It is readily understood that the processes shown in the above figures do not indicate or limit the temporal order of these processes. Furthermore, it is readily understood that these processes may be executed synchronously or asynchronously, for example, in multiple modules.

[0095] Further reference Figure 9 As shown, this example embodiment also provides an image restoration model determination device 900, including a first acquisition module 910, a first determination module 920, a second determination module 930, and a model training module 940. Wherein:

[0096] The first acquisition module 910 can be used to acquire a reference multi-focus image and an initial model, wherein the reference multi-focus image includes multiple reference images with different focus positions. Specifically, a preset focus range can be acquired, and multiple reference images can be acquired at preset intervals.

[0097] The first determining module 920 can be used to determine the full-focus label data corresponding to the reference multi-focus image based on multiple reference images with different focus positions. Specifically, the full-focus label data is obtained by stacking and fusing multiple reference images with different focus positions.

[0098] The second determining module 930 can be used to determine the input image based on the full-focus label data using a point spread function. Specifically, it uses the full-focus label data to convolve the point spread function to obtain the intermediate image; and obtains the input image based on multiple intermediate images. The average value of the multiple intermediate images is used as the input image.

[0099] The model training module 940 can be used to train the initial model based on the input image and the full-focus label data to obtain an image restoration model. Specifically, the input image is input into the generative adversarial network to obtain an output image; the minimum mean squared error loss, generative adversarial loss, and perceptual loss between the output image and the full-focus label data are calculated; and the parameters of the generative adversarial network are adjusted according to the minimum mean squared error loss, generative adversarial loss, and perceptual loss to obtain the image restoration model.

[0100] Further reference Figure 10 As shown, this example embodiment also provides an image restoration device 1000, including a second acquisition module 1010, a third acquisition module 1020, and an image restoration module 1030. Wherein,

[0101] The second acquisition module 1010 can be used to acquire the initial actual captured image, and the third acquisition module 1020 can be used to acquire the image restoration model; the image restoration module 1030 can be used to use the image restoration model to perform image restoration on the initial actual captured image to obtain the target large depth-of-field image.

[0102] In one example implementation, the third acquisition module 1020 acquires the network architecture and network parameters of the image restoration model; in response to the image restoration operation, it uses the network architecture and the network parameters to obtain the image restoration model.

[0103] In one example implementation, the image restoration module 1030 inputs the actual captured image into the image restoration model to obtain the target large depth-of-field model.

[0104] The specific details of each module in the above-mentioned device have been described in detail in the method section of the implementation. For any undisclosed details, please refer to the implementation content of the method section, and therefore will not be repeated here.

[0105] The following is based on Figure 11 Taking the mobile terminal 1100 as an example, the construction of this electronic device will be described by way of example. Those skilled in the art will understand that, apart from components specifically designed for mobile purposes, Figure 11 The structure can also be applied to fixed types of equipment.

[0106] like Figure 11 As shown, the mobile terminal 1100 may specifically include: a processor 1101, a memory 1102, a bus 1103, a mobile communication module 1104, an antenna 1, a wireless communication module 1105, an antenna 2, a display screen 1106, a camera module 1107, an audio module 1108, a power module 1109, and a sensor module 1110.

[0107] Processor 1101 may include one or more processing units. For example, processor 1110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, an encoder, a decoder, a digital signal processor (DSP), a baseband processor, and / or a neural network processing unit (NPU). The method for determining the image restoration model in this exemplary embodiment can be executed by an AP, GPU, or DSP. When the method involves neural network-related processing, it can be executed by an NPU.

[0108] An encoder can encode (i.e., compress) images or videos. For example, it can encode a target image into a specific format to reduce data size for easier storage or transmission. A decoder can decode (i.e., decompress) the encoded data of an image or video to restore the image or video data. For example, it can read the encoded data of a target image, decode it, and restore the target image data, then perform related processing to determine an image restoration model. The mobile terminal 1100 can support one or more encoders and decoders. Thus, the mobile terminal 1100 can process images or videos in various encoding formats, such as JPEG (Joint Photographic Experts Group), PNG (Portable Network Graphics), BMP (Bitmap), and MPEG (Moving Picture Experts Group) 1, MPEG2, H.263, H.264, HEVC (High Efficiency Video Coding), and other video formats.

[0109] The processor 1101 can be connected to the memory 1102 or other components via the bus 1103.

[0110] The memory 1102 can be used to store computer executable program code, which includes instructions. The processor 1101 executes various functional applications and data processing of the mobile terminal 1100 by running the instructions stored in the memory 1102. The memory 202 can also store application data, such as images, videos, and other files.

[0111] The communication function of mobile terminal 1100 can be implemented through mobile communication module 1104, antenna 1, wireless communication module 1105, antenna 2, modem processor, and baseband processor. Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Mobile communication module 1104 can provide 2G, 3G, 4G, and 5G mobile communication solutions for mobile terminal 1100. Wireless communication module 1105 can provide wireless communication solutions such as wireless LAN, Bluetooth, and near-field communication for mobile terminal 1100.

[0112] The display screen 1106 is used to implement display functions, such as displaying the user interface, images, and videos. The camera module 1107 is used to implement shooting functions, such as capturing images and videos. The audio module 1108 is used to implement audio functions, such as playing audio and capturing voice. The power module 1109 is used to implement power management functions, such as charging the battery, powering the device, and monitoring the battery status. The sensor module 1110 may include a depth sensor 11101, a pressure sensor 11102, a gyroscope sensor 11103, a barometric pressure sensor 11104, etc., to implement corresponding sensing and detection functions.

[0113] Those skilled in the art will understand that various aspects of this disclosure can be implemented as systems, methods, or program products. Therefore, various aspects of this disclosure can be specifically implemented in the following forms: entirely in hardware, entirely in software (including firmware, microcode, etc.), or in a combination of hardware and software, collectively referred to herein as “circuit,” “module,” or “system.”

[0114] Exemplary embodiments of this disclosure also provide a computer-readable storage medium having a program product stored thereon capable of implementing the methods described above in this specification. In some possible embodiments, various aspects of this disclosure may also be implemented as a program product including program code that, when the program product is run on a terminal device, causes the terminal device to perform the steps described in the "Exemplary Methods" section of this specification according to various exemplary embodiments of this disclosure.

[0115] It should be noted that the computer-readable medium disclosed herein may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof.

[0116] In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium, which can transmit, propagate, or transfer a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wireline, optical fiber, RF, etc., or any suitable combination thereof.

[0117] Furthermore, program code for performing the operations of this disclosure can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0118] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and embodiments are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the claims.

[0119] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A method for determining an image restoration model, characterized in that, Obtain a reference multi-focus image and an initial model, wherein the reference multi-focus image includes multiple reference images with different focus positions; Determine the full-focus label data corresponding to the reference multi-focus image based on multiple reference images with different focus positions; The input image is determined using the point spread function based on the fully focused label data. An image restoration model is obtained by training the initial model based on the input image and the full-focus label data; wherein, the initial model is a generative adversarial network, and the process of obtaining the image restoration model includes: inputting the input image into the generative adversarial network to obtain an output image; calculating the minimum mean square error loss, generative adversarial loss, and perceptual loss between the output image and the full-focus label data; and adjusting the parameters of the generative adversarial network according to the minimum mean square error loss, generative adversarial loss, and perceptual loss to obtain the image restoration model.

2. The method according to claim 1, characterized in that, Obtaining a reference multi-focus image includes: Obtain a preset focus range and acquire multiple reference images at preset intervals.

3. The method according to claim 1, characterized in that, The step of determining the full-focus label data corresponding to the reference multi-focus image based on multiple reference images with different focus positions includes: The full-focus label data is obtained by stacking and fusing multiple reference images with different focus positions.

4. The method according to claim 3, characterized in that, Determining the input image using the point spread function based on the fully focused label data includes: The intermediate image is obtained by convolving the fully focused label data with the point spread function; The input image is obtained from multiple intermediate images.

5. The method according to claim 4, characterized in that, The step of obtaining the input image based on multiple intermediate images includes: The average value of multiple intermediate images is used as the input image.

6. An image restoration method, characterized in that, include: Acquire the initial actual captured images; An image restoration model is obtained, wherein the image restoration model is determined by the image restoration model determination method according to any one of claims 1-5; The target large depth-of-field image is obtained by using an image restoration model to restore the initial actual captured image.

7. A device for determining an image restoration model, characterized in that, include: The first acquisition module is used to acquire a reference multi-focus image and an initial model, wherein the reference multi-focus image includes multiple reference images with different focus positions; The first determining module is used to determine the full-focus label data corresponding to the reference multi-focus image based on the multiple reference images with different focus positions; The second determining module is used to determine the input image based on the full-focus label data using a point spread function; The model training module is used to train the initial model based on the input image and the full-focus label data to obtain an image restoration model; wherein, the initial model is a generative adversarial network, and obtaining the image restoration model includes: inputting the input image into the generative adversarial network to obtain an output image; calculating the minimum mean square error loss, generative adversarial loss, and perceptual loss between the output image and the full-focus label data; and adjusting the parameters of the generative adversarial network according to the minimum mean square error loss, generative adversarial loss, and perceptual loss to obtain the image restoration model.

8. An image restoration device, characterized in that, include: The second acquisition module is used to acquire the initial actual captured image; The third acquisition module is used to acquire the image restoration model; The image restoration module is used to restore the target large depth-of-field image from the initial actual captured image using an image restoration model; The image restoration model is determined by the method according to any one of claims 1-5.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1 to 6.

10. An electronic device, characterized in that, include: One or more processors; as well as A memory for storing one or more programs, which, when executed by one or more processors, cause the one or more processors to implement the method as described in any one of claims 1 to 6.

Images