Model training method and device and rapid super-depth-of-field imaging method and device

Abstract

The invention discloses a model training method and device and a rapid super-depth-of-field imaging method and device, and belongs to the technical field of computer vision. The model training method comprises the following steps: acquiring a single-frame zoom image sample and a plurality of frames of fixed-focus image samples shot in a preset working distance range; fusing the multiple frames of fixed-focus image samples to generate a full-focus image sample, and matching the single-frame zoom image sample with the full-focus image sample to construct a reference data set for an image recovery task; constructing a multiple-input multiple-output network model suitable for an image restoration task; taking a single-frame zoom image sample in the reference data set as input, taking a full-focus image sample matched with the single-frame zoom image sample as a label image, and training the model; and storing parameters of the model after training is completed. According to the invention, the training of the multiple-input multiple-output network model is realized, the method can be used for an image restoration task to obtain the super-depth-of-field image, and the accuracy and efficiency of super-depth-of-field image generation are improved.

Description

Technical Field

[0001] The present application belongs to the field of computer vision technology, and in particular, relates to a model training method and device and a fast super-depth-of-field imaging method and device. Background Art

[0002] Industrial vision technology plays a vital role in the field of industrial automation, using computer vision technology to complete tasks such as target recognition and quality inspection. In highly automated and complex scenarios, the camera system is required to be able to obtain global and clear images with a large depth of field, which is crucial for subsequent data analysis.

[0003] In the face of the depth of field problem in actual industrial scenes, traditional solutions include using a large depth of field lens or reducing the aperture to expand the depth of field. However, these methods will limit the image brightness while expanding the depth of field, affecting the imaging speed and clarity, and it is difficult to meet the requirements of high speed and high precision. The field of computer vision provides another solution, which is to shoot multiple images with different focal points through multi-focus imaging technology, and then use the image fusion algorithm to generate a full-depth image, which can retain the clarity of the original image to a large extent and achieve depth of field extension within a custom range. Nevertheless, this method is computationally complex and has high data throughput, which poses a challenge to real-time applications. In addition, new devices such as light field cameras can achieve depth of field adjustment in post-processing, but their high cost and complex operation limit their widespread application in the industrial field. Therefore, the development of industrial vision technology needs to increase speed and reduce costs while meeting the requirements of obtaining large depth of field images to meet the needs of highly automated scenarios. Summary of the invention

[0004] The present application aims to solve at least one of the technical problems existing in the related art. To this end, the present application proposes a model training method, device and fast super-depth imaging method and device, which realize the training of a multi-input multi-output network model. The trained multi-input multi-output network model can be used for image restoration tasks to obtain super-depth images, thereby improving the accuracy and efficiency of super-depth image generation.

[0005] In a first aspect, the present application provides a model training method, the method comprising:

[0006] Obtaining single-frame zoom image samples and multiple-frame fixed-focus image samples captured within a preset working distance range;

[0007] Fusing the multiple frames of fixed-focus image samples to generate fully focused image samples, and matching the single-frame zoom image samples with the fully focused image samples to construct a benchmark data set for image restoration tasks;

[0008] Constructing a multi-input multi-output network model suitable for the image restoration task;

[0009] Taking a single-frame zoom image sample in the benchmark data set as input and a fully focused image sample matched with the single-frame zoom image sample as a label map, training the multi-input multi-output network model;

[0010] The parameters of the multi-input multi-output network model are saved when the training is completed.

[0011] In the above technical scheme, single-frame zoom image samples and multi-frame fixed-focus image samples are obtained, and fully focused image samples are generated by fusing multi-frame fixed-focus image samples to construct a benchmark data set. A multi-input multi-output network model is constructed and trained using the benchmark data set. Finally, the trained model parameters are saved, thereby realizing the training of the multi-input multi-output network model. The trained multi-input multi-output network model can be used for image restoration tasks to obtain super-depth-of-field images, thereby improving the accuracy and efficiency of super-depth-of-field image generation.

[0012] According to one embodiment of the present application, the multi-input multi-output network model includes: a multi-input single encoder and a multi-output single decoder;

[0013] The multi-input single encoder is used to input the original image into the network at different sizes, extract features of the original image from different sizes, and generate multi-layer multi-size feature maps;

[0014] The multi-output single decoder is used to perform image reconstruction at each layer to restore the details and texture information of the original image.

[0015] In the above technical scheme, the multi-input multi-output network model is composed of a multi-input single encoder and a multi-output single decoder. The multi-input single encoder is used to receive original images of different sizes, extract features of the original images from different sizes, and generate corresponding multi-layer and multi-size feature maps. The multi-output single decoder is used to perform image reconstruction on each layer of the network to restore the details and texture information of the original image. Through multi-size input and hierarchical reconstruction, the multi-input multi-output network model can capture and utilize image information more comprehensively, thereby improving the accuracy and efficiency of image restoration.

[0016] According to an embodiment of the present application, the single-frame zoom image sample in the benchmark data set is used as input, and the all-focus image sample matched by the single-frame zoom image sample is used as a label map to train the multi-input multi-output network model, including:

[0017] Inputting the single-frame zoom image sample into the multi-input single encoder in different sizes to obtain a multi-layer multi-size feature map;

[0018] Inputting the multi-layer and multi-size feature maps into the multi-output single decoder to obtain an image restoration result;

[0019] The loss between the image restoration result and the label map is calculated, and the multi-input multi-output network model is trained based on the loss.

[0020] In the above technical scheme, a single-frame zoom image sample in a benchmark data set is used as input, and a matching fully focused image sample is used as a label to train a multi-input and multi-output network model. By inputting a single-frame zoom image into the encoder at different sizes, multi-layer and multi-scale features are extracted, and the decoder restores the image details. Finally, the loss between the restored result and the label map is calculated and the model parameters are optimized. The trained model can be used to output a fully focused image based on a single-frame zoom image, thereby improving the accuracy and efficiency of image restoration.

[0021] According to one embodiment of the present application, the multi-input single encoder includes: a first convolution module, a shallow convolution module, a first residual convolution module, a downsampling module and a feature attention module,

[0022] The step of inputting the single-frame zoom image sample into the multi-input single encoder in different sizes to obtain a multi-layer multi-size feature map comprises:

[0023] Inputting the single-frame zoom image sample into the first convolution module and the first residual convolution module in sequence for processing to generate a first-layer feature map;

[0024] Inputting the first layer feature map into the downsampling module to generate a first layer downsampling feature map;

[0025] Reducing the length and width of the single-frame zoom image sample to 1 / 2 of the original length and width to obtain a first reduced image;

[0026] Inputting the first reduced image into a shallow convolution module to generate a second layer input image;

[0027] Inputting the second layer input image and the first layer down-sampled feature map into the feature attention module and then into the first residual convolution module to generate a second layer feature map;

[0028] Inputting the second layer feature map into the downsampling module to generate a second layer downsampling feature map;

[0029] Reducing the length and width of the single-frame zoom image sample to 1 / 4 of the original length and width to obtain a second reduced image;

[0030] Inputting the second reduced image into a shallow convolution module to generate a third layer input image;

[0031] The third layer input image and the second layer down-sampled feature map are input into the feature attention module together and then input into the first residual convolution module to generate a third layer feature map.

[0032] In the above technical scheme, the multi-input single encoder includes a first convolution module, a shallow convolution module, a first residual convolution module, a downsampling module and a feature attention module, which realizes feature extraction of single-frame zoom image samples of different sizes, effectively improves the ability to extract features of single-frame zoom image samples, and provides a more accurate and efficient feature representation for image restoration tasks.

[0033] According to one embodiment of the present application, the multi-output single decoder includes: a second convolution module, a second residual convolution module, an upsampling module and an output convolution module, and the output convolution module includes output convolution modules of the first layer, the second layer and the third layer;

[0034] The step of inputting the multi-layer multi-size feature map into the multi-output single decoder to obtain an image restoration result comprises:

[0035] Inputting the third layer feature map into the second residual convolution module to obtain a third layer intermediate result;

[0036] Inputting the third layer intermediate result into the output convolution module of the third layer to obtain a first convolution result;

[0037] Adding the first convolution result and the second reduced image to obtain a 1 / 4 reduced image restoration result;

[0038] Inputting the third layer intermediate result into the upsampling module to obtain the second layer upsampling feature map;

[0039] The second layer up-sampled feature map and the second layer down-sampled feature map are concatenated according to channels and sequentially input into the second convolution module and the second residual convolution module to obtain a second layer intermediate result;

[0040] Inputting the second layer intermediate result into the output convolution module of the second layer to obtain a second convolution result;

[0041] Adding the second convolution result to the first reduced image to obtain a 1 / 2 reduced image restoration result;

[0042] Input the second layer intermediate result into the upsampling module to obtain the first layer upsampling feature map;

[0043] The first layer up-sampled feature map and the first layer down-sampled feature map are concatenated according to channels and sequentially input into the second convolution module and the second residual convolution module to obtain a first layer intermediate result;

[0044] Inputting the first layer intermediate result into the output convolution module of the first layer to obtain a third convolution result;

[0045] The third convolution result is added to the single-frame zoom image sample to obtain an image restoration result with the same size as the single-frame zoom image sample.

[0046] In the above technical solution, the multi-output single decoder includes a second convolution module, a second residual convolution module, an upsampling module and an output convolution module. The output convolution module includes output convolution modules of the first layer, the second layer and the third layer, thereby realizing efficient processing of multi-layer and multi-size feature maps, thereby improving the accuracy and efficiency of image restoration.

[0047] According to an embodiment of the present application, the calculating the loss between the image restoration result and the label map, and training the multi-input multi-output network model based on the loss, includes:

[0048] Scaling the label image to obtain a 1 / 2 reduced label image and a 1 / 4 reduced label image;

[0049] Calculate the loss between the 1 / 4 reduced image restoration result and the 1 / 4 reduced label map, calculate the loss between the 1 / 2 reduced image restoration result and the 1 / 2 reduced label map, calculate the loss between the image restoration result and the label map, and superimpose the losses as content loss;

[0050] Transforming the 1 / 4 reduced image restoration result, the 1 / 2 reduced image restoration result and the image restoration result into the frequency domain, transforming the 1 / 4 reduced label map, the 1 / 2 reduced label map and the label map into the frequency domain, calculating the distance result between the 1 / 4 reduced image restoration result and the 1 / 4 reduced label map in the frequency domain, calculating the distance result between the 1 / 2 reduced image restoration result and the 1 / 2 reduced label map in the frequency domain, calculating the distance result between the image restoration result and the label map in the frequency domain, and superimposing the distance results as auxiliary losses;

[0051] Based on the content loss and the auxiliary loss, the parameters of the multi-input multi-output network model are optimized.

[0052] In the above technical scheme, the label map is scaled to match the restoration results of different scales, and then the loss between the restoration results of each scale and the corresponding label map is calculated as the content loss, and the image restoration results and the label map are converted to the frequency domain, and their distance results in the frequency domain are calculated as the auxiliary loss. Based on the content loss and auxiliary loss, the network model parameters are optimized, and the training effect of the multi-input and multi-output network model is improved, thereby improving the accuracy and efficiency of image restoration.

[0053] In a second aspect, the present application provides a fast ultra-depth of field imaging method, the method comprising:

[0054] Get a single-frame zoom image;

[0055] Inputting the single-frame zoom image into a fast super-depth-of-field imaging model to obtain a super-depth-of-field image;

[0056] Among them, the fast super depth of field imaging model is obtained by cropping and deploying a multi-input multi-output network model, and the multi-input multi-output network model is obtained by executing the model training method described in the first aspect above.

[0057] In the above technical scheme, a super depth of field image is obtained by acquiring a single-frame zoom image and inputting it into a fast super depth of field imaging model that has been cropped and optimized. The multi-input multi-output network model can be used for image restoration tasks to obtain a super depth of field image. The fast super depth of field imaging model is obtained based on the deployment after cropping the multi-input multi-output network model, which improves the efficiency of super depth of field image generation.

[0058] According to one embodiment of the present application, the method further includes:

[0059] Acquire the multi-input multi-output network model;

[0060] Deleting the output convolution module of the second layer and the output convolution module of the third layer of the multi-output single decoder of the multi-input multi-output network model to obtain a multi-input single-output test model;

[0061] Use the multi-input single-output test model to load the parameters of the multi-input multi-output network model when training is completed to obtain an intermediate model;

[0062] The intermediate model is deployed on acceleration hardware to obtain the fast super-depth-of-field imaging model.

[0063] In the above technical scheme, a multi-input multi-output network model is obtained, some output convolution modules of its multi-output single decoder are cropped to obtain a multi-input single-output test model, the original model training parameters are loaded to obtain an intermediate model, and the intermediate model is deployed on acceleration hardware to obtain a fast super depth of field imaging model. While maintaining the image restoration capability of the fast super depth of field imaging model, the working efficiency of the fast super depth of field imaging model is improved.

[0064] According to one embodiment of the present application, acquiring a single-frame zoom image includes:

[0065] Constructing an optical working condition consistent with a benchmark data set used to train the multi-input multi-output network model, the optical working condition including a preset working distance range;

[0066] A single-frame zoom image within the preset working distance range is acquired.

[0067] In the above technical scheme, by constructing an optical working condition that matches the multi-input and multi-output network model training benchmark data set, including setting a specific working distance range and obtaining a single-frame zoom image under this condition, the consistency between the model application environment and the training environment can be improved, thereby improving the accuracy and efficiency of super-depth of field image generation.

[0068] In a third aspect, the present application provides a model training device, the device comprising:

[0069] A first acquisition module is used to acquire single-frame zoom image samples and multiple-frame fixed-focus image samples captured within a preset working distance range;

[0070] A data set generation module, used to fuse the multiple frames of fixed-focus image samples to generate fully focused image samples, and match the single-frame zoom image samples with the fully focused image samples to construct a benchmark data set for image restoration tasks;

[0071] A construction module, used to construct a multi-input and multi-output network model suitable for the image restoration task;

[0072] A training module, used to train the multi-input multi-output network model using a single-frame zoom image sample in the benchmark data set as an input and a fully focused image sample matched by the single-frame zoom image sample as a label;

[0073] A storage module is used to save the parameters of the multi-input multi-output network model when training is completed.

[0074] In the above technical scheme, the model training device obtains single-frame zoom image samples and multi-frame fixed-focus image samples, generates fully focused image samples by fusing the multi-frame fixed-focus image samples to construct a benchmark data set, constructs a multi-input multi-output network model, and uses the data set for training. Finally, the trained model parameters are saved, thereby realizing the training of the multi-input multi-output network model. The trained multi-input multi-output network model can be used for image restoration tasks to obtain super-depth of field images, thereby improving the accuracy and efficiency of super-depth of field image generation.

[0075] In a fourth aspect, the present application provides a fast ultra-depth-of-field imaging device, the device comprising:

[0076] An optical imaging module, used to obtain a single-frame zoom image;

[0077] A processing module, used for inputting the single-frame zoom image into a fast super-depth-of-field imaging model to obtain a super-depth-of-field image;

[0078] Among them, the fast super depth of field imaging model is obtained by cropping and deploying a multi-input multi-output network model, and the multi-input multi-output network model is obtained by executing the model training method described in the first aspect above.

[0079] In the above technical solution, the fast super-depth of field imaging device obtains a single-frame zoom image and inputs it into a fast super-depth of field imaging model that has been cropped and optimized to obtain a super-depth of field image. The multi-input multi-output network model can be used for image restoration tasks to obtain a super-depth of field image. The fast super-depth of field imaging model is obtained by cropping the multi-input multi-output network model and then deploying it, thereby improving the efficiency of super-depth of field image generation.

[0080] According to one embodiment of the present application, the optical imaging module includes: an optical imaging module and an electronic control module.

[0081] The optical imaging module includes a liquid lens module;

[0082] The electronic control module includes a current or voltage control module, and the current or voltage control module is used to drive the liquid lens module to change the diopter through a current or voltage signal to adjust the working distance of the liquid lens module.

[0083] In the above technical solution, the optical imaging module includes a liquid lens module and a current / voltage control module. The electronic control module accurately controls the refractive power of the liquid lens through current or voltage signals, thereby realizing flexible adjustment of the working distance and improving the adaptability and imaging quality of the fast super-depth of field imaging device.

[0084] In a fifth aspect, the present application provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the processor implements the model training method as described in the first aspect above, or implements the fast super-depth of field imaging method as described in the second aspect above.

[0085] In a sixth aspect, the present application provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the model training method as described in the first aspect above, or implements the fast super-depth of field imaging method as described in the second aspect above.

[0086] In the seventh aspect, the present application provides a chip, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the model training method as described in the first aspect above, or to implement the fast super-depth of field imaging method as described in the second aspect above.

[0087] In an eighth aspect, the present application provides a computer program product, comprising a computer program, which, when executed by a processor, implements the model training method as described in the first aspect above, or implements the fast super-depth of field imaging method as described in the second aspect above.

[0088] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0089] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:

[0090] Figure 1 It is a flowchart of a model training method provided in some embodiments of the present application;

[0091] Figure 2 is a schematic diagram of the structure of a multi-input multi-output network model provided in some embodiments of the present application;

[0092] Figure 3 It is a schematic diagram of a process of training a multi-input multi-output network model provided in some embodiments of the present application;

[0093] Figure 4 is a flowchart of a fast super-depth-of-field imaging method provided by some embodiments of the present application;

[0094] Figure 5 is a schematic diagram of the structure of a model training device provided in some embodiments of the present application;

[0095] Figure 6 is a schematic structural diagram of a fast super-depth-of-field imaging device provided in some embodiments of the present application;

[0096] Figure 7 It is a schematic diagram of the structure of an electronic device provided in some embodiments of the present application.

[0097] Description of reference numerals:

[0098] 200: Multi-input multi-output network model; 201: Multi-input single encoder; 202: Multi-output single decoder;

[0099] 2011: The first convolutional module; 2012: The shallow convolutional module; 2013: The first residual convolutional module;

[0100] 2014: Downsampling module; 2015: Feature attention module; 2021: Second convolution module;

[0101] 2022: second residual convolution module; 2023: upsampling module; 2024: output convolution module;

[0102] 500: model training device; 501: first acquisition module; 502: data set generation module;

[0103] 503: construction module; 504: training module; 505: storage module;

[0104] 600: fast ultra-depth imaging device; 601: optical imaging module; 602: processing module;

[0105] 603: optical imaging module; 604: electronic control module; 605: liquid lens module;

[0106] 606: current or voltage control module; 700: electronic device; 701: processor; 702: memory. DETAILED DESCRIPTION

[0107] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0108] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0109] The model training method, device and fast super-depth-of-field imaging method and device provided in the embodiments of the present application are described in detail below in conjunction with the accompanying drawings through specific embodiments and their application scenarios.

[0110] The model training method provided in the embodiment of the present application may be executed by an electronic device or a functional module or functional entity in the electronic device that can implement the model training method. The electronic devices mentioned in the embodiment of the present application include but are not limited to mobile phones, tablet computers, computers, cameras, and wearable devices. The model training method provided in the embodiment of the present application is described below using an electronic device as an example of the execution subject.

[0111] Figure 1 is a flow chart of the model training method provided in some embodiments of the present application. Figure 1 As shown, the model training method includes: step 110, step 120, step 130, step 140 and step 150.

[0112] Step 110: Acquire a single-frame zoom image sample and multiple-frame fixed-focus image samples captured within a preset working distance range.

[0113] It can be understood that, taking adjusting the target lens to shoot the target object as an example, a single-frame zoom image refers to adjusting the focal length of the target lens during a single exposure process and maintaining the exposure state during this period, thereby fusing the imaging information of the target object at different focal lengths to obtain an image. The single-frame zoom image records the comprehensive imaging information of the target object at multiple focal lengths.

[0114] It can be understood that a multi-frame fixed-focus image is composed of a group of single-frame fixed-focus images with different depths of field, which refers to a group of images obtained by adjusting the focal length of the target lens and taking multiple independent shots of the same target object. Each frame of the fixed-focus image is shot at a specific, fixed focal length.

[0115] It can be understood that the single-frame zoom image sample and the multiple-frame fixed-focus image samples should be acquired under the same optical working condition.

[0116] Optionally, the target lens is a liquid lens, and the adjustment of its focal length is achieved by changing the applied voltage or current, and by changing the distribution or shape of the liquid substance (such as a special optical fluid) inside the lens, thereby changing the lens's diopter, i.e., focal length. The working distance range refers to the adjustable range of the lens focal length. When the target lens is a liquid lens, it refers to the focal length variation range that the target lens can achieve under different voltage or current control.

[0117] It is understandable that the preset working distance range defines the adjustment boundary of the focal length of the target lens. In the process of shooting multiple frames of fixed-focus images, each frame of fixed-focus image produces a specific clear area due to the fixed focal length. The clear area may match the area where the target object is located, so that a specific part of the target object can be clearly imaged; or it may not match, resulting in blurred imaging. By adjusting the focal length of the target lens within the preset working distance range, the clear areas of multiple frames of fixed-focus images can cover the entire area where the target object is located, and then the fully focused image synthesized based on multiple frames of fixed-focus images can achieve clear imaging of the entire target object. This fully focused image can be considered as a super depth of field image.

[0118] It can be understood that a single-frame zoom image combines the imaging of the target object at different focal lengths within a preset working distance range. In the process of acquiring a single-frame zoom image, the target lens performs a long-range zoom, and its focal length variation range covers the focal length used when shooting each fixed-focus image in the multi-frame fixed-focus image.

[0119] It can be understood that the single-frame zoom image sample refers to a single-frame zoom image used to train a multi-input multi-output network model, and the multi-frame fixed-focus image sample refers to a multi-frame fixed-focus image used to train a multi-input multi-output network model.

[0120] Step 120: fuse the multiple frames of fixed-focus image samples to generate fully focused image samples, and match the single-frame zoom image samples with the fully focused image samples to construct a benchmark data set for image restoration tasks.

[0121] The method of fusing multiple frames of fixed-focus image samples to generate fully focused image samples may be a Laplace pyramid method, a wavelet transform method, etc. In some embodiments, fusing multiple frames of fixed-focus image samples to generate fully focused image samples includes: calculating the clarity of the clear area of ​​each image in the multiple frames of fixed-focus image samples, determining the clarity weight of each image accordingly, and fusing the images based on these weights; or, first calculating the transformation matrix between the images in the multiple frames of fixed-focus image samples through key point detection and feature matching, achieving spatial domain alignment and unification of the multiple frames of fixed-focus image samples based on the transformation matrix, and then fusing the multiple frames of fixed-focus image samples.

[0122] The matched single-frame zoom image samples and fully focused image samples are added to the benchmark dataset for the image restoration task. It can be understood that the single-frame zoom image samples usually appear as blurred images, and the image restoration task is to restore the super-depth of field image that is as close to the fully focused image sample as possible from the blurred single-frame zoom image sample.

[0123] Step 130: construct a multi-input multi-output network model suitable for the image restoration task.

[0124] It can be understood that the multi-input multi-output network model can accept various types of image data as input, and adjust the output layer to produce a variety of output results according to different application requirements.

[0125] Optionally, the multi-input multi-output network model adopts a Deblur-Unet network.

[0126] Step 140: Using the single-frame zoom image samples in the benchmark data set as input and the fully focused image samples matched by the single-frame zoom image samples as label images, the multi-input multi-output network model is trained.

[0127] The fully focused image samples are used as labels corresponding to the single-frame zoom image samples, i.e., the expected output of the multi-input multi-output network model, and are compared with the actual output of the multi-input multi-output network model (the image outputted by receiving the single-frame zoom image samples as input) and the loss is calculated, and the multi-input multi-output network model is trained based on the loss. Optionally, the loss is calculated based on the structure similarity index measure (SSIM), the mean absolute error loss (MAE), or the L1 loss, etc.

[0128] Step 150: Save the parameters of the multi-input multi-output network model when training is completed.

[0129] It can be understood that saving the multi-input multi-output network model parameters when training is completed eliminates the need for retraining when the model is subsequently reloaded, which helps to realize the deployment and practical application of the multi-input multi-output network model.

[0130] In the above technical scheme, single-frame zoom image samples and multi-frame fixed-focus image samples are obtained, and fully focused image samples are generated by fusing multi-frame fixed-focus image samples to construct a benchmark data set. A multi-input multi-output network model is constructed and trained using the benchmark data set. Finally, the trained model parameters are saved, thereby realizing the training of the multi-input multi-output network model. The trained multi-input multi-output network model can be used for image restoration tasks to obtain super-depth-of-field images, thereby improving the accuracy and efficiency of super-depth-of-field image generation.

[0131] In one embodiment of the present application, the multi-input multi-output network model includes: a multi-input single encoder and a multi-output single decoder;

[0132] The multi-input single encoder is used to input the original image into the network at different sizes, extract features of the original image from different sizes, and generate multi-layer multi-size feature maps;

[0133] The multi-output single decoder is used to perform image reconstruction at each layer to restore the details and texture information of the original image.

[0134] Figure 2 is a schematic diagram of the structure of a multi-input multi-output network model provided by some embodiments of the present application. Figure 2 As shown, the multi-input multi-output network model 200 includes: a multi-input single encoder 201 and a multi-output single decoder 202.

[0135] The multi-input single encoder is used to receive original images of different sizes (also called scales) as input, and extract features from inputs of different sizes to generate multi-layer multi-size feature maps, where each layer corresponds to a feature map of one size. The multi-layer size feature maps represent the feature information of the image at multiple sizes and are the data basis for subsequent image restoration. Optionally, the multi-input single encoder includes multiple convolutional layers, which can extract features from original images of different sizes layer by layer, and then output multi-layer multi-size feature maps, which may be different in spatial size, and together constitute a comprehensive and in-depth feature representation of the original image.

[0136] The multi-output single decoder is used to reconstruct the image based on the multi-layer and multi-scale feature maps output by the multi-input single encoder. It can perform image reconstruction on different network layers, thereby gradually restoring the details and texture information in the image. Optionally, the multi-input single encoder includes an upsampling layer and a convolution layer. The upsampling layer is used to gradually enlarge the feature map output by the multi-input single encoder, while the convolution layer can further refine and adjust these features so that they eventually match the original image. Through the hierarchical reconstruction method, the multi-output single decoder can gradually optimize the reconstruction quality of the image at different scales and generate high-quality output images.

[0137] In the above technical scheme, the multi-input multi-output network model is composed of a multi-input single encoder and a multi-output single decoder. The multi-input single encoder is used to receive original images of different sizes, extract features of the original images from different sizes, and generate corresponding multi-layer and multi-size feature maps. The multi-output single decoder is used to perform image reconstruction on each layer of the network to restore the details and texture information of the original image. Through multi-size input and hierarchical reconstruction, the multi-input multi-output network model can capture and utilize image information more comprehensively, thereby improving the accuracy and efficiency of image restoration.

[0138] In one embodiment of the present application, the single-frame zoom image sample in the benchmark data set is used as input, and the all-focus image sample matched by the single-frame zoom image sample is used as a label map to train the multi-input multi-output network model, including:

[0139] Inputting the single-frame zoom image sample into the multi-input single encoder in different sizes to obtain a multi-layer multi-size feature map;

[0140] Inputting the multi-layer and multi-size feature maps into the multi-output single decoder to obtain an image restoration result;

[0141] The loss between the image restoration result and the label map is calculated, and the multi-input multi-output network model is trained based on the loss.

[0142] It can be understood that single-frame zoom image samples are selected from the benchmark dataset. These images are obtained by dynamically adjusting the focal length during a single exposure process, and the imaging information at different focal lengths is integrated. The fully focused image samples matching the single-frame zoom image samples are used as label maps. The fully focused image samples can be considered as super-depth of field images, which represent the expected output during the model training process.

[0143] The single-frame zoom image samples are input into the multi-input single encoder at different sizes, which allows the model to capture image features from different sizes. Optionally, the encoder processes these images of different sizes through a series of convolutional layers, extracts the features of the single-frame zoom image samples of different sizes layer by layer, and generates multi-layer multi-size feature maps. The multi-layer multi-size feature maps output by the multi-input single encoder are passed to the multi-output single decoder, and the image is restored based on these feature maps through the multi-output single decoder. Optionally, the upsampling layer in the multi-output single decoder is used to gradually increase the spatial resolution of the feature map and restore the details of the image. The convolutional layer further refines these features so that the image restoration result matches the single-frame zoom image sample.

[0144] The loss between the image restoration result output by the multi-output single decoder and the all-focus image label map is calculated. The loss function can be the structural similarity index (SSIM), the absolute error loss (MAE) or the L1 loss, etc.

[0145] Optionally, based on the calculated loss, the parameters of the network model are adjusted through an optimization algorithm (such as gradient descent) to minimize the loss function. This process involves a back-propagation algorithm to update the weights in the network and bias the refined image features, adjusting the texture and details of the image to match the size and quality of the original image.

[0146] In the above technical scheme, a single-frame zoom image sample in a benchmark data set is used as input, and a matching fully focused image sample is used as a label to train a multi-input and multi-output network model. By inputting a single-frame zoom image into the encoder at different sizes, multi-layer and multi-scale features are extracted, and the decoder restores the image details. Finally, the loss between the restored result and the label map is calculated and the model parameters are optimized. The trained model can be used to output a fully focused image based on a single-frame zoom image, thereby improving the accuracy and efficiency of image restoration.

[0147] In one embodiment of the present application, the multi-input single encoder includes: a first convolution module (Convolution), a shallow convolution module (Small Image Convolution), a first residual convolution module (L_ResBlock), a downsampling module and a feature attention module (Feature Fusion),

[0148] The step of inputting the single-frame zoom image sample into the multi-input single encoder in different sizes to obtain a multi-layer multi-size feature map comprises:

[0149] Inputting the single-frame zoom image sample into the first convolution module and the first residual convolution module in sequence for processing to generate a first-layer feature map;

[0150] Inputting the first layer feature map into the downsampling module to generate a first layer downsampling feature map;

[0151] Reducing the length and width of the single-frame zoom image sample to 1 / 2 of the original length and width to obtain a first reduced image;

[0152] Inputting the first reduced image into a shallow convolution module to generate a second layer input image;

[0153] Inputting the second layer input image and the first layer down-sampled feature map into the feature attention module and then into the first residual convolution module to generate a second layer feature map;

[0154] Inputting the second layer feature map into the downsampling module to generate a second layer downsampling feature map;

[0155] Reducing the length and width of the single-frame zoom image sample to 1 / 4 of the original length and width to obtain a second reduced image;

[0156] Inputting the second reduced image into a shallow convolution module to generate a third layer input image;

[0157] The third layer input image and the second layer down-sampled feature map are input into the feature attention module together and then input into the first residual convolution module to generate a third layer feature map.

[0158] Optionally, an interpolation algorithm is used to scale the single-frame zoom image samples.

[0159] like Figure 2 As shown, the multi-input single encoder 201 includes: a first convolution module 2011, a shallow convolution module 2012, a first residual convolution module 2013, a downsampling module 2014 and a feature attention module 2015.

[0160] It can be understood that the first convolution module 2011 is used to perform preliminary convolution operations on the original single-frame zoom image sample (usually input in the form of a grayscale image, but can also be a color image), such as extracting basic features such as edges and textures. It can be considered that the output of the first convolution module 201 contains the basic feature information of the single-frame zoom image sample.

[0161] It can be understood that the downsampling module 2014 is used to perform downsampling processing on the feature maps of each level. This process reduces the size of the feature map and retains key feature information, thereby generating a downsampled feature map, including:

[0162] The first layer feature map is received to generate a first layer down-sampled feature map, or the second layer feature map is received to generate a second layer down-sampled feature map.

[0163] It can be understood that the shallow convolution module 2012 is used to perform a preliminary convolution operation on the single-frame zoom image sample whose size has been reduced (such as the first reduced image obtained by reducing the length and width of the single-frame zoom image sample to 1 / 2 or the second reduced image obtained by reducing it to 1 / 4) to extract basic features, and the output of the shallow convolution module 2012 is used as the input image of the next level, including: receiving the first reduced image to generate the second layer input image, or receiving the second reduced image to generate the third layer input image. Optionally, the shallow convolution module has fewer convolution layers or smaller convolution kernels to quickly and efficiently extract image features such as edges and textures.

[0164] It can be understood that the first residual convolution module 2013 is used to further extract and enhance the feature map obtained through the preliminary convolution operation through the residual learning mechanism to capture deeper feature information and generate feature maps of various levels, including:

[0165] Receive the output from the first convolution module and generate the first layer feature map.

[0166] Alternatively, receiving the second layer input image processed by the feature attention module 2015 and the first layer down-sampled feature map, generating a second layer feature map,

[0167] Alternatively, receiving the third layer input image processed by the feature attention module 2015 and the second layer down-sampled feature map, and generating a third layer feature map;

[0168] It can be understood that the feature attention module 2015 is used to receive and integrate feature maps from different levels, and perform weighted processing on the fused feature maps through the attention mechanism to highlight the key features in the image, including:

[0169] Receive the second layer input image and the first layer downsampled feature map, generate the second layer feature map,

[0170] Alternatively, a third layer input image and a second layer downsampled feature map are received to generate a third layer feature map.

[0171] Optionally, a single-frame zoom image sample is input into a multi-input multi-output network model in the form of a grayscale image, and a first-layer input image is generated after passing through the first convolution module 2011. At this time, the dimension of each pixel changes from 1 layer to 2 layers, and the first-layer input image is input into the first residual convolution module 2013 to generate a first-layer feature map;

[0172] The first layer feature map is input into the downsampling module 2014 to generate the first layer downsampling feature map. At this time, the dimension of each pixel is changed from 2 layers to 4 layers, and the length and width of the first layer downsampling feature map are respectively reduced to 1 / 2 of the first layer feature map. The length and width of the original single-frame zoom image sample are reduced to 1 / 2 of the original and then input into the shallow convolution module 2012 to generate the second layer input image. The second layer input image and the first layer downsampling feature map are input into the feature attention module 2015 and then into the first residual convolution module 2013 to generate the second layer feature map.

[0173] The second layer feature map is input into the downsampling module 2014 to generate the second layer downsampling feature map. At this time, the dimension of each pixel is changed from 4 layers to 8 layers, and the length and width of the second layer downsampling feature map are reduced to 1 / 2 of the second layer feature map respectively. The length and width of the original single-frame zoom image sample are reduced to 1 / 4 of the original and then input into the shallow convolution module 2012 to generate the third layer input image. The third layer input image and the second layer downsampling feature map are input into the feature attention module 2015 together and then into the first residual convolution module 2013 to generate the third layer feature map.

[0174] In the above technical scheme, the multi-input single encoder includes a first convolution module, a shallow convolution module, a first residual convolution module, a downsampling module and a feature attention module, which realizes feature extraction of single-frame zoom image samples of different sizes, effectively improves the ability to extract features of single-frame zoom image samples, and provides a more accurate and efficient feature representation for image restoration tasks.

[0175] In one embodiment of the present application, the multi-output single decoder includes: a second convolution module (Convolution), a second residual convolution module (L_ResBlock), an upsampling module and an output convolution module (Convolution Output), and the output convolution module includes output convolution modules of the first layer, the second layer and the third layer;

[0176] The step of inputting the multi-layer multi-size feature map into the multi-output single decoder to obtain an image restoration result comprises:

[0177] Inputting the third layer feature map into the second residual convolution module to obtain a third layer intermediate result;

[0178] Inputting the third layer intermediate result into the output convolution module of the third layer to obtain a first convolution result;

[0179] Adding the first convolution result and the second reduced image to obtain a 1 / 4 reduced image restoration result;

[0180] Inputting the third layer intermediate result into the upsampling module to obtain the second layer upsampling feature map;

[0181] The second layer up-sampled feature map and the second layer down-sampled feature map are concatenated according to channels and sequentially input into the second convolution module and the second residual convolution module to obtain a second layer intermediate result;

[0182] Inputting the second layer intermediate result into the output convolution module of the second layer to obtain a second convolution result;

[0183] Adding the second convolution result to the first reduced image to obtain a 1 / 2 reduced image restoration result;

[0184] Input the second layer intermediate result into the upsampling module to obtain the first layer upsampling feature map;

[0185] The first layer up-sampled feature map and the first layer down-sampled feature map are concatenated according to channels and sequentially input into the second convolution module and the second residual convolution module to obtain a first layer intermediate result;

[0186] Inputting the first layer intermediate result into the output convolution module of the first layer to obtain a third convolution result;

[0187] The third convolution result is added to the single-frame zoom image sample to obtain an image restoration result with the same size as the single-frame zoom image sample.

[0188] It is understandable that a feature map can have multiple channels, each channel represents different feature information, and channel-by-channel splicing refers to splicing on the channel dimension of the feature map. Optionally, by adjusting the spatial dimension of the feature map and splicing the feature map on the channel dimension to achieve skip connection, the effective combination of features at different levels can be achieved.

[0189] like Figure 2 As shown, the multi-output single decoder 202 includes: a second convolution module 2021, a second residual convolution module 2022, an upsampling module 2023 and an output convolution module 2024.

[0190] It can be understood that the second convolution module 2021 receives the result of splicing the up-sampled feature map and the down-sampled feature map obtained after processing by the up-sampling module by channel, further refines and enhances the features through the convolution operation, and provides a richer feature representation for the subsequent second residual convolution module, including:

[0191] Receive the result of channel splicing of the second layer up-sampling feature map and the second layer down-sampling feature map, perform convolution operation, and output to the second residual convolution module.

[0192] Alternatively, a result of channel-wise concatenation of the first-layer up-sampling feature map and the first-layer down-sampling feature map is received, a convolution operation is performed, and the result is output to a second residual convolution module.

[0193] It can be understood that the second residual convolution module 2022 is used to further extract and enhance the features of the feature map through the residual learning mechanism, including:

[0194] Receive the third layer feature map and output the third layer intermediate result.

[0195] Alternatively, when the second convolution module 2021 receives the result of channel splicing of the second layer up-sampled feature map and the second layer down-sampled feature map as input, the output of the second convolution module 2021 is received as input, and the second layer intermediate result is output.

[0196] Alternatively, when the second convolution module 2021 receives the result of splicing the first layer up-sampled feature map and the first layer down-sampled feature map by channel as input, the output of the second convolution module 2021 is received as input, and the first layer intermediate result is output;

[0197] It can be understood that the upsampling module 2023 is used to perform upsampling processing on the intermediate results of each level, in which the spatial size is increased, the spatial resolution of the feature map is gradually increased, and the feature information is retained, thereby generating an upsampled feature map, including:

[0198] Receive the intermediate result of the third layer and output the upsampled feature map of the second layer.

[0199] Alternatively, receive the intermediate result of the second layer and output the upsampled feature map of the first layer;

[0200] It can be understood that the output convolution module 2024 includes output convolution modules of the first layer, the second layer, and the third layer, receives the intermediate results of each level from the second residual convolution module, and outputs the convolution results of each level, including:

[0201] The output convolution module of the third layer receives the intermediate result of the third layer and outputs the first convolution result;

[0202] The output convolution module of the second layer receives the intermediate result of the second layer and outputs the second convolution result;

[0203] The output convolution module of the first layer receives the intermediate result of the first layer and outputs the third convolution result.

[0204] Optionally, the third layer feature map is input into the second residual convolution module 2022 to obtain a third layer intermediate result, the third layer intermediate result is input into the third layer output convolution module 2024, and the obtained result is added to the single frame zoom image sample whose length and width are reduced to 1 / 4 of the original, to obtain a 1 / 4 reduced image restoration result;

[0205] The third-layer intermediate result is input into the upsampling module 2023 to generate a second-layer upsampling feature map. At this time, the dimension of each pixel is changed from 8 layers to 4 layers, and the length and width of the second-layer upsampling feature map are increased to twice the third-layer feature map respectively; the second-layer upsampling feature map and the second-layer downsampling feature map are spliced ​​according to channels and then sequentially input into the second convolution module 2021 and the second residual convolution module 2022 to obtain the second-layer intermediate result, and the second-layer intermediate result is input into the output convolution module 2024 of the second layer, and the obtained result is added to the single-frame zoom image sample whose length and width are reduced to 1 / 2 of the original, to obtain a 1 / 2 reduced image restoration result;

[0206] After the second layer intermediate result is input into the upsampling module 2023, the first layer upsampling feature map is generated. At this time, the dimension of each pixel is changed from 4 layers to 2 layers, and the length and width of the first layer upsampling feature map are increased to twice of the second layer feature map respectively; the first layer upsampling feature map and the first layer downsampling feature map are spliced ​​by channel and then input into the second convolution module 2021 and the second residual convolution module 2022 in sequence to obtain the first layer intermediate result, and the first layer intermediate result is input into the output convolution module 2024 of the first layer, and the obtained result is added to the original single-frame zoom image sample to obtain an image restoration result of the same size as the input image.

[0207] In the above technical solution, the multi-output single decoder includes a second convolution module, a second residual convolution module, an upsampling module and an output convolution module. The output convolution module includes output convolution modules of the first layer, the second layer and the third layer, thereby realizing efficient processing of multi-layer and multi-size feature maps, thereby improving the accuracy and efficiency of image restoration.

[0208] Figure 3 is a flow chart of training a multi-input multi-output network model provided by some embodiments of the present application. Figure 3 As shown, in one embodiment of the present application, the calculation of the loss between the image restoration result and the label map, and the training of the multi-input multi-output network model based on the loss include: step 310, step 320, step 330 and step 340.

[0209] Step 310: scaling the label image to obtain a 1 / 2 reduced label image and a 1 / 4 reduced label image;

[0210] Step 320: Calculate the loss between the 1 / 4 reduced image restoration result and the 1 / 4 reduced label map, calculate the loss between the 1 / 2 reduced image restoration result and the 1 / 2 reduced label map, calculate the loss between the image restoration result and the label map, and superimpose the losses as content loss;

[0211] Step 330: transform the 1 / 4 reduced image restoration result, the 1 / 2 reduced image restoration result and the image restoration result into the frequency domain, transform the 1 / 4 reduced label map, the 1 / 2 reduced label map and the label map into the frequency domain, calculate the distance result between the 1 / 4 reduced image restoration result and the 1 / 4 reduced label map in the frequency domain, calculate the distance result between the 1 / 2 reduced image restoration result and the 1 / 2 reduced label map in the frequency domain, calculate the distance result between the image restoration result and the label map in the frequency domain, and superimpose the distance results as the auxiliary loss;

[0212] Step 340: Optimize the parameters of the multi-input multi-output network model based on the content loss and the auxiliary loss.

[0213] Optionally, use an interpolation algorithm to scale the label map.

[0214] It can be understood that the image restoration result refers to the output result obtained by inputting a single-frame zoom image sample in the benchmark dataset into the multi-input multi-output network model, and the label map refers to the fully focused image sample matched by the single-frame zoom image sample, and the fully focused image sample is generated by fusion based on multiple frames of fixed-focus image samples.

[0215] Optionally, the structural similarity index measure (SSIM) loss between the 1 / 4 reduced image restoration result and the 1 / 4 reduced label map, the 1 / 2 reduced image restoration result and the 1 / 2 reduced label map, and the image restoration result and the label map are calculated respectively, and these loss results are superimposed to form content loss.

[0216] Optionally, use Fast Fourier Transformation (FFT) to convert the 1 / 4 reduced image restoration result, the 1 / 2 reduced image restoration result, and the image restoration result into the frequency domain, and convert the 1 / 2 reduced label map, the 1 / 4 reduced label map, and the label map into the frequency domain, and calculate the 1 / 4 reduced image restoration result and the 1 / 4 reduced label map, the 1 / 2 reduced image restoration result and the 1 / 2 reduced label map, and the image restoration result and the label map. Figure 3 The L1 distance (Manhattan distance) of the layer images in the frequency domain is calculated, and these distance results are superimposed as auxiliary losses.

[0217] Based on the content loss and auxiliary loss, the parameters of the multi-input and multi-output network model are adjusted through optimization algorithms (such as gradient descent, etc.), the content loss and auxiliary loss are repeatedly calculated, and the model parameters are continuously optimized and adjusted to make the image restoration result as close to the label image as possible.

[0218] In the above technical scheme, the label map is scaled to match the restoration results of different scales, and then the loss between the restoration results of each scale and the corresponding label map is calculated as the content loss, and the image restoration results and the label map are converted to the frequency domain, and their distance results in the frequency domain are calculated as the auxiliary loss. Based on the content loss and auxiliary loss, the network model parameters are optimized, and the training effect of the multi-input and multi-output network model is improved, thereby improving the accuracy and efficiency of image restoration.

[0219] Figure 4 is a flow chart of a fast super-depth imaging method provided by some embodiments of the present application. Figure 4 As shown, the fast super-depth-of-field imaging method includes: step 410 and step 420.

[0220] Step 410: Acquire a single-frame zoom image;

[0221] Step 420: input the single-frame zoom image into a fast super-depth-of-field imaging model to obtain a super-depth-of-field image;

[0222] Among them, the fast super depth of field imaging model is obtained by trimming and deploying a multi-input multi-output network model, and the multi-input multi-output network model is obtained by executing the model training method as mentioned above.

[0223] In some embodiments, acquiring a single-frame zoom image includes:

[0224] Constructing an optical working condition consistent with a benchmark data set used to train the multi-input multi-output network model, the optical working condition including a preset working distance range;

[0225] A single-frame zoom image within the preset working distance range is acquired.

[0226] It can be understood that constructing optical working conditions consistent with the benchmark data set used to train the multi-input multi-output network model improves the consistency between the model application environment and the training environment, and helps to improve the performance of the multi-input multi-output network model.

[0227] In the above technical scheme, by constructing an optical working condition that matches the multi-input and multi-output network model training benchmark data set, including setting a specific working distance range and obtaining a single-frame zoom image under this condition, the consistency between the model application environment and the training environment can be improved, thereby improving the accuracy and efficiency of super-depth of field image generation.

[0228] It is understandable that in order to improve processing speed and efficiency, the original multi-input multi-output network model is pruned and optimized to meet the needs of fast ultra-depth imaging. Optionally, the pruned model includes reducing the number of network layers, simplifying the network structure, etc., so that the model can achieve fast response and processing while maintaining high performance.

[0229] In the above technical scheme, a super depth of field image is obtained by acquiring a single-frame zoom image and inputting it into a fast super depth of field imaging model that has been cropped and optimized. The multi-input multi-output network model can be used for image restoration tasks to obtain a super depth of field image. The fast super depth of field imaging model is obtained based on the deployment after cropping the multi-input multi-output network model, which improves the efficiency of super depth of field image generation.

[0230] In some embodiments of the present application, the method further includes:

[0231] Acquire the multi-input multi-output network model;

[0232] Deleting the output convolution module of the second layer and the output convolution module of the third layer of the multi-output single decoder of the multi-input multi-output network model to obtain a multi-input single-output test model;

[0233] Use the multi-input single-output test model to load the parameters of the multi-input multi-output network model when training is completed to obtain an intermediate model;

[0234] The intermediate model is deployed on acceleration hardware to obtain the fast super-depth-of-field imaging model.

[0235] It is understandable that in the process of training a multi-input multi-output network model, by calculating the loss between the output results of multiple different levels and the label map, the loss results of different levels are superimposed to generate a total loss, and the model is trained based on the total loss, which can improve the training effect of the model. In order to improve the processing speed and efficiency of the model, the multi-input multi-output network model needs to be pruned. Optionally, the output convolution module of the second layer and the output convolution module of the third layer of the multi-output single decoder are deleted, thereby reducing the complexity and computational complexity of the model and retaining the main functions of the model. The pruned model becomes a multi-input single-output test model.

[0236] Use the pruned MIMO test model to load the parameters of the original MIMO network model when training is completed, so as to obtain an intermediate model, such as the onnx model that can be used across different platforms. The intermediate model not only inherits the image capability of the MIMO network model, but also combines the efficiency brought by pruning, achieving a balance between performance and speed.

[0237] Optionally, set the intermediate model dynamic input so that the intermediate model can reconstruct blurred images of different sizes.

[0238] Deploying the intermediate model on hardware with acceleration capabilities can further improve the model's efficiency. Optionally, use TensorRT, a high-performance deep learning inference engine provided by NVIDIA, to optimize the intermediate model with dynamic input, and deploy the intermediate model on the NVIDIA GPU. During the deployment process, use fp16 precision quantization technology to further improve the model's inference efficiency. The engine model file generated after deployment is used as the compressed model file for actual use.

[0239] In the above technical scheme, a multi-input multi-output network model is obtained, some output convolution modules of its multi-output single decoder are cropped to obtain a multi-input single-output test model, the original model training parameters are loaded to obtain an intermediate model, and the intermediate model is deployed on acceleration hardware to obtain a fast super depth of field imaging model. While maintaining the image restoration capability of the fast super depth of field imaging model, the working efficiency of the fast super depth of field imaging model is improved.

[0240] The model training method provided in the embodiment of the present application can be executed by a model training device. In the embodiment of the present application, the model training device executing the model training method is taken as an example to illustrate the model training device provided in the embodiment of the present application.

[0241] Figure 5 Schematic diagram of the structure of the model training device provided in some embodiments of the present application. Figure 5 As shown, the model training device 500 includes:

[0242] The first acquisition module 501 is used to acquire a single-frame zoom image sample and multiple-frame fixed-focus image samples captured within a preset working distance range;

[0243] A data set generation module 502 is used to fuse the multiple frames of fixed-focus image samples to generate fully focused image samples, and match the single-frame zoom image samples with the fully focused image samples to construct a benchmark data set for image restoration tasks;

[0244] A construction module 503 is used to construct a multi-input multi-output network model suitable for the image restoration task;

[0245] A training module 504 is used to train the multi-input multi-output network model using a single-frame zoom image sample in the benchmark data set as an input and a fully focused image sample matched with the single-frame zoom image sample as a label;

[0246] The storage module 505 is used to store the parameters of the multi-input multi-output network model when the training is completed.

[0247] Optionally, the multi-input multi-output network model includes: a multi-input single encoder and a multi-output single decoder;

[0248] The multi-input single encoder is used to input the original image into the network at different sizes, extract features of the original image from different sizes, and generate multi-layer multi-size feature maps;

[0249] The multi-output single decoder is used to perform image reconstruction at each layer to restore the details and texture information of the original image.

[0250] Optionally, the training module 504 is used to:

[0251] Inputting the single-frame zoom image sample into the multi-input single encoder in different sizes to obtain a multi-layer multi-size feature map;

[0252] Inputting the multi-layer and multi-size feature maps into the multi-output single decoder to obtain an image restoration result;

[0253] The loss between the image restoration result and the label map is calculated, and the multi-input multi-output network model is trained based on the loss.

[0254] Optionally, the multi-input single encoder includes: a first convolution module, a shallow convolution module, a first residual convolution module, a downsampling module and a feature attention module,

[0255] The step of inputting the single-frame zoom image sample into the multi-input single encoder in different sizes to obtain a multi-layer multi-size feature map comprises:

[0256] Inputting the single-frame zoom image sample into the first convolution module and the first residual convolution module in sequence for processing to generate a first-layer feature map;

[0257] Inputting the first layer feature map into the downsampling module to generate a first layer downsampling feature map;

[0258] Reducing the length and width of the single-frame zoom image sample to 1 / 2 of the original length and width to obtain a first reduced image;

[0259] Inputting the first reduced image into a shallow convolution module to generate a second layer input image;

[0260] Inputting the second layer input image and the first layer down-sampled feature map into the feature attention module and then into the first residual convolution module to generate a second layer feature map;

[0261] Inputting the second layer feature map into the downsampling module to generate a second layer downsampling feature map;

[0262] Reducing the length and width of the single-frame zoom image sample to 1 / 4 of the original length and width to obtain a second reduced image;

[0263] Inputting the second reduced image into a shallow convolution module to generate a third layer input image;

[0264] The third layer input image and the second layer down-sampled feature map are input into the feature attention module together and then input into the first residual convolution module to generate a third layer feature map.

[0265] Optionally, the multi-output single decoder includes: a second convolution module, a second residual convolution module, an upsampling module and an output convolution module, and the output convolution module includes output convolution modules of the first layer, the second layer and the third layer;

[0266] The step of inputting the multi-layer multi-size feature map into the multi-output single decoder to obtain an image restoration result comprises:

[0267] Inputting the third layer feature map into the second residual convolution module to obtain a third layer intermediate result;

[0268] Inputting the third layer intermediate result into the output convolution module of the third layer to obtain a first convolution result;

[0269] Adding the first convolution result and the second reduced image to obtain a 1 / 4 reduced image restoration result;

[0270] Inputting the third layer intermediate result into the upsampling module to obtain the second layer upsampling feature map;

[0271] The second layer up-sampled feature map and the second layer down-sampled feature map are concatenated according to channels and sequentially input into the second convolution module and the second residual convolution module to obtain a second layer intermediate result;

[0272] Inputting the second layer intermediate result into the output convolution module of the second layer to obtain a second convolution result;

[0273] Adding the second convolution result to the first reduced image to obtain a 1 / 2 reduced image restoration result;

[0274] Input the second layer intermediate result into the upsampling module to obtain the first layer upsampling feature map;

[0275] The first layer up-sampled feature map and the first layer down-sampled feature map are concatenated according to channels and sequentially input into the second convolution module and the second residual convolution module to obtain a first layer intermediate result;

[0276] Inputting the first layer intermediate result into the output convolution module of the first layer to obtain a third convolution result;

[0277] The third convolution result is added to the single-frame zoom image sample to obtain an image restoration result with the same size as the single-frame zoom image sample.

[0278] Optionally, the training module 504 is further used for:

[0279] Scaling the label image to obtain a 1 / 2 reduced label image and a 1 / 4 reduced label image;

[0280] Calculate the loss between the 1 / 4 reduced image restoration result and the 1 / 4 reduced label map, calculate the loss between the 1 / 2 reduced image restoration result and the 1 / 2 reduced label map, calculate the loss between the image restoration result and the label map, and superimpose the losses as content loss;

[0281] Transforming the 1 / 4 reduced image restoration result, the 1 / 2 reduced image restoration result and the image restoration result into the frequency domain, transforming the 1 / 4 reduced label map, the 1 / 2 reduced label map and the label map into the frequency domain, calculating the distance result between the 1 / 4 reduced image restoration result and the 1 / 4 reduced label map in the frequency domain, calculating the distance result between the 1 / 2 reduced image restoration result and the 1 / 2 reduced label map in the frequency domain, calculating the distance result between the image restoration result and the label map in the frequency domain, and superimposing the distance results as auxiliary losses;

[0282] Based on the content loss and the auxiliary loss, the parameters of the multi-input multi-output network model are optimized.

[0283] In the above technical scheme, the model training device obtains single-frame zoom image samples and multi-frame fixed-focus image samples, generates fully focused image samples by fusing the multi-frame fixed-focus image samples to construct a benchmark data set, constructs a multi-input multi-output network model, and uses the data set for training. Finally, the trained model parameters are saved, thereby realizing the training of the multi-input multi-output network model. The trained multi-input multi-output network model can be used for image restoration tasks to obtain super-depth of field images, thereby improving the accuracy and efficiency of super-depth of field image generation.

[0284] The model training device in the embodiment of the present application can be an electronic device, or a component in the electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal, or other devices other than a terminal. Exemplarily, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, a vehicle-mounted electronic device, a mobile Internet device (Mobile Internet Device, MID), an augmented reality (augmented reality, AR) / virtual reality (virtualreality, VR) device, a robot, a wearable device, an ultra-mobile personal computer (ultra-mobile personal computer, UMPC), a netbook or a personal digital assistant (personal digital assistant, PDA), etc., and can also be a server, a network attached storage (Network Attached Storage, NAS), a personal computer (personalcomputer, PC), a television (television, TV), a teller machine or a self-service machine, etc., and the embodiment of the present application is not specifically limited.

[0285] The model training device in the embodiment of the present application may be a device having an operating system. The operating system may be a Microsoft (Windows) operating system, an Android (Android) operating system, an IOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0286] The model training device provided in the embodiment of the present application can implement each process implemented in the embodiment of the model training method. To avoid repetition, it will not be repeated here.

[0287] The fast super depth of field imaging method provided in the embodiment of the present application can be performed by a fast super depth of field imaging device. In the embodiment of the present application, the fast super depth of field imaging device performing the fast super depth of field imaging method is taken as an example to illustrate the fast super depth of field imaging device provided in the embodiment of the present application.

[0288] Figure 6 Schematic diagram of the structure of a fast ultra-depth imaging device provided by some embodiments of the present application. Figure 5 As shown, the fast super depth of field imaging device 600 includes:

[0289] Optical imaging module 601, used to obtain a single-frame zoom image;

[0290] Processing module 602, used for inputting the single-frame zoom image into a fast super-depth-of-field imaging model to obtain a super-depth-of-field image;

[0291] Among them, the fast super depth of field imaging model is obtained by trimming and deploying a multi-input multi-output network model, and the multi-input multi-output network model is obtained by executing the above-mentioned model training method.

[0292] Optionally, the optical imaging module 601 is used for:

[0293] Constructing an optical working condition consistent with a benchmark data set used to train the multi-input multi-output network model, the optical working condition including a preset working distance range;

[0294] A single-frame zoom image within the preset working distance range is acquired.

[0295] Optionally, the fast super-depth-of-field imaging device 600 further includes a processing unit, and the processing unit is used to:

[0296] Acquire the multi-input multi-output network model;

[0297] Deleting the output convolution module of the second layer and the output convolution module of the third layer of the multi-output single decoder of the multi-input multi-output network model to obtain a multi-input single-output test model;

[0298] Use the multi-input single-output test model to load the parameters of the multi-input multi-output network model when training is completed to obtain an intermediate model;

[0299] The intermediate model is deployed on acceleration hardware to obtain the fast super-depth-of-field imaging model.

[0300] Optionally, the optical imaging module 601 includes: an optical imaging module 603 and an electronic control module 604,

[0301] The optical imaging module 603 includes a liquid lens module 605;

[0302] The electronic control module 604 includes a current or voltage control module 606 , and the current or voltage control module 606 is used to drive the liquid lens module 605 to change the diopter through a current or voltage signal, so as to adjust the working distance of the liquid lens module 605 .

[0303] The optical imaging module includes a liquid lens module and a current / voltage control module. The electronic control module accurately controls the refractive power of the liquid lens through current or voltage signals, realizes flexible adjustment of the working distance, and improves the adaptability and imaging quality of the fast super-depth of field imaging device.

[0304] In the above technical solution, the fast super-depth of field imaging device obtains a single-frame zoom image and inputs it into a fast super-depth of field imaging model that has been cropped and optimized to obtain a super-depth of field image. The multi-input multi-output network model can be used for image restoration tasks to obtain a super-depth of field image. The fast super-depth of field imaging model is obtained by cropping the multi-input multi-output network model and then deploying it, thereby improving the efficiency of super-depth of field image generation.

[0305] The fast super-depth imaging device in the embodiment of the present application can be an electronic device, or a component in the electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal, or other devices other than a terminal. Exemplarily, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, a vehicle-mounted electronic device, a mobile Internet device (Mobile Internet Device, MID), an augmented reality (augmented reality, AR) / virtual reality (virtual reality, VR) device, a robot, a wearable device, an ultra-mobile personal computer (ultra-mobile personal computer, UMPC), a netbook or a personal digital assistant (personal digital assistant, PDA), etc. It can also be a server, a network attached storage (Network Attached Storage, NAS), a personal computer (personal computer, PC), a television (television, TV), a teller machine or a self-service machine, etc., which is not specifically limited in the embodiment of the present application.

[0306] The fast super depth of field imaging device in the embodiment of the present application may be a device having an operating system. The operating system may be a Microsoft (Windows) operating system, an Android (Android) operating system, an IOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0307] The fast super-depth-of-field imaging device provided in the embodiment of the present application can implement each process implemented in the embodiment of the fast super-depth-of-field imaging method. To avoid repetition, they will not be described here.

[0308] In some embodiments, Figure 7 As shown, an embodiment of the present application also provides an electronic device 700, including a processor 701, a memory 702, and a computer program stored in the memory 702 and executable on the processor 701. When the program is executed by the processor 701, each process of the above-mentioned model training method or the fast super-depth of field imaging method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be described here.

[0309] It should be noted that the electronic devices in the embodiments of the present application include the mobile electronic devices and non-mobile electronic devices mentioned above.

[0310] An embodiment of the present application also provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the various processes of the above-mentioned model training method or the fast super-depth of field imaging method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0311] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0312] An embodiment of the present application also provides a computer program product, including a computer program, which implements the above-mentioned model training method or fast super-depth of field imaging method when executed by a processor.

[0313] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0314] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned model training method or the fast super-depth of field imaging method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0315] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0316] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0317] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the relevant technology, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0318] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

[0319] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "illustrative embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials, or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0320] Although the embodiments of the present application have been shown and described, those skilled in the art will appreciate that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present application, and that the scope of the present application is defined by the claims and their equivalents.

Claims

1. A model training method, characterized in that: include: Obtaining single-frame zoom image samples and multiple-frame fixed-focus image samples captured within a preset working distance range; Fusing the multiple frames of fixed-focus image samples to generate fully focused image samples, and matching the single-frame zoom image samples with the fully focused image samples to construct a benchmark data set for image restoration tasks; Constructing a multi-input multi-output network model suitable for the image restoration task; Taking a single-frame zoom image sample in the benchmark data set as input and a fully focused image sample matched with the single-frame zoom image sample as a label map, training the multi-input multi-output network model; The parameters of the multi-input multi-output network model are saved when the training is completed.

2. The method according to claim 1, characterized in that: The multi-input multi-output network model includes: a multi-input single encoder and a multi-output single decoder; The multi-input single encoder is used to input the original image into the network at different sizes, extract features of the original image from different sizes, and generate multi-layer multi-size feature maps; The multi-output single decoder is used to perform image reconstruction at each layer to restore the details and texture information of the original image.

3. The method according to claim 2, characterized in that The method of taking a single-frame zoom image sample in the benchmark data set as input and a fully focused image sample matched with the single-frame zoom image sample as a label map to train the multi-input multi-output network model includes: Inputting the single-frame zoom image sample into the multi-input single encoder in different sizes to obtain a multi-layer multi-size feature map; Inputting the multi-layer and multi-size feature maps into the multi-output single decoder to obtain an image restoration result; The loss between the image restoration result and the label map is calculated, and the multi-input multi-output network model is trained based on the loss.

4. The method according to claim 3, characterized in that: The multi-input single encoder includes: a first convolution module, a shallow convolution module, a first residual convolution module, a downsampling module and a feature attention module, The step of inputting the single-frame zoom image sample into the multi-input single encoder in different sizes to obtain a multi-layer multi-size feature map comprises: Inputting the single-frame zoom image sample into the first convolution module and the first residual convolution module in sequence for processing to generate a first-layer feature map; Inputting the first layer feature map into the downsampling module to generate a first layer downsampling feature map; Reducing the length and width of the single-frame zoom image sample to 1 / 2 of the original length and width to obtain a first reduced image; Inputting the first reduced image into a shallow convolution module to generate a second layer input image; Inputting the second layer input image and the first layer down-sampled feature map into the feature attention module and then into the first residual convolution module to generate a second layer feature map; Inputting the second layer feature map into the downsampling module to generate a second layer downsampling feature map; Reducing the length and width of the single-frame zoom image sample to 1 / 4 of the original length and width to obtain a second reduced image; Inputting the second reduced image into a shallow convolution module to generate a third layer input image; The third layer input image and the second layer down-sampled feature map are input into the feature attention module together and then input into the first residual convolution module to generate a third layer feature map.

5. The method according to claim 4, characterized in that The multi-output single decoder includes: a second convolution module, a second residual convolution module, an upsampling module and an output convolution module, and the output convolution module includes output convolution modules of the first layer, the second layer and the third layer; The step of inputting the multi-layer multi-size feature map into the multi-output single decoder to obtain an image restoration result comprises: Inputting the third layer feature map into the second residual convolution module to obtain a third layer intermediate result; Inputting the third layer intermediate result into the output convolution module of the third layer to obtain a first convolution result; Adding the first convolution result and the second reduced image to obtain a 1 / 4 reduced image restoration result; Inputting the third layer intermediate result into the upsampling module to obtain the second layer upsampling feature map; The second layer up-sampled feature map and the second layer down-sampled feature map are concatenated according to channels and sequentially input into the second convolution module and the second residual convolution module to obtain a second layer intermediate result; Inputting the second layer intermediate result into the output convolution module of the second layer to obtain a second convolution result; Adding the second convolution result to the first reduced image to obtain a 1 / 2 reduced image restoration result; Input the second layer intermediate result into the upsampling module to obtain the first layer upsampling feature map; The first layer up-sampled feature map and the first layer down-sampled feature map are concatenated according to channels and sequentially input into the second convolution module and the second residual convolution module to obtain a first layer intermediate result; Inputting the first layer intermediate result into the output convolution module of the first layer to obtain a third convolution result; The third convolution result is added to the single-frame zoom image sample to obtain an image restoration result with the same size as the single-frame zoom image sample.

6. The method according to claim 5, characterized in that The calculating the loss between the image restoration result and the label map, and training the multi-input multi-output network model based on the loss, comprises: Scaling the label image to obtain a 1 / 2 reduced label image and a 1 / 4 reduced label image; Calculate the loss between the 1 / 4 reduced image restoration result and the 1 / 4 reduced label map, calculate the loss between the 1 / 2 reduced image restoration result and the 1 / 2 reduced label map, calculate the loss between the image restoration result and the label map, and superimpose the losses as content loss; Transforming the 1 / 4 reduced image restoration result, the 1 / 2 reduced image restoration result and the image restoration result into the frequency domain, transforming the 1 / 4 reduced label map, the 1 / 2 reduced label map and the label map into the frequency domain, calculating the distance result between the 1 / 4 reduced image restoration result and the 1 / 4 reduced label map in the frequency domain, calculating the distance result between the 1 / 2 reduced image restoration result and the 1 / 2 reduced label map in the frequency domain, calculating the distance result between the image restoration result and the label map in the frequency domain, and superimposing the distance results as auxiliary losses; Based on the content loss and the auxiliary loss, the parameters of the multi-input multi-output network model are optimized.

7. A fast ultra-depth imaging method, characterized in that: include: Get a single-frame zoom image; Inputting the single-frame zoom image into a fast super-depth-of-field imaging model to obtain a super-depth-of-field image; Among them, the fast super depth of field imaging model is obtained based on the deployment of a multi-input multi-output network model after trimming, and the multi-input multi-output network model is obtained by executing the model training method as described in any one of claims 1 to 6.

8. The method according to claim 7, characterized in that The method further comprises: Acquire the multi-input multi-output network model; Deleting the output convolution module of the second layer and the output convolution module of the third layer of the multi-output single decoder of the multi-input multi-output network model to obtain a multi-input single-output test model; Use the multi-input single-output test model to load the parameters of the multi-input multi-output network model when training is completed to obtain an intermediate model; The intermediate model is deployed on acceleration hardware to obtain the fast super-depth-of-field imaging model.

9. The method according to claim 7, characterized in that: The step of acquiring a single-frame zoom image comprises: Constructing an optical working condition consistent with a benchmark data set used to train the multi-input multi-output network model, the optical working condition including a preset working distance range; A single-frame zoom image within the preset working distance range is acquired.

10. A model training device, characterized in that: include: A first acquisition module is used to acquire single-frame zoom image samples and multiple-frame fixed-focus image samples captured within a preset working distance range; A data set generation module, used to fuse the multiple frames of fixed-focus image samples to generate fully focused image samples, and match the single-frame zoom image samples with the fully focused image samples to construct a benchmark data set for image restoration tasks; A construction module, used to construct a multi-input and multi-output network model suitable for the image restoration task; A training module, used to train the multi-input multi-output network model using a single-frame zoom image sample in the benchmark data set as an input and a fully focused image sample matched by the single-frame zoom image sample as a label; A storage module is used to save the parameters of the multi-input multi-output network model when training is completed.

11. A fast ultra-depth imaging device, characterized in that: include: An optical imaging module, used to obtain a single-frame zoom image; A processing module, used for inputting the single-frame zoom image into a fast super-depth-of-field imaging model to obtain a super-depth-of-field image; Among them, the fast super depth of field imaging model is obtained based on the deployment of a multi-input multi-output network model after trimming, and the multi-input multi-output network model is obtained by executing the model training method as described in any one of claims 1 to 6.

12. The device according to claim 11, characterized in that The optical imaging module includes: an optical imaging module and an electronic control module. The optical imaging module includes a liquid lens module; The electronic control module includes a current or voltage control module, and the current or voltage control module is used to drive the liquid lens module to change the diopter through a current or voltage signal to adjust the working distance of the liquid lens module.

13. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, it implements the model training method as described in any one of claims 1-6, or implements the fast super-depth of field imaging method as described in any one of claims 7-9.

14. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, it implements the model training method as described in any one of claims 1 to 6, or implements the fast super-depth of field imaging method as described in any one of claims 7 to 9.

Images