Large depth of field synthesis optimization method and device based on different depth of field feature learning, storage medium and program product

Abstract

The application discloses a large-depth-of-field synthesis optimization method and device based on different depth-of-field feature learning, a storage medium and a program product. In view of the depth map error problem caused by the excessively large peak width of the focal point response curve, the inaccurate peak position or the multi-peak confusion in the super-depth-of-field fusion process under a large-depth-of-field objective lens, the deep learning mode is adopted to optimize the large-depth-of-field synthesized depth map, and in order to solve the problem of difficult acquisition of end-to-end training data, a feature domain calibration training mode is designed to reduce the requirement for training data, and the training data is obtained through the cross-objective lens registration processing method in the process.

Description

Technical Field

[0001] This invention belongs to the fields of image processing and deep learning, and specifically relates to a depth-of-field fusion technology based on deep learning. Background Technology

[0002] The concept of super depth-of-field fusion technology arose because a shallow depth of field in the objective lens prevents the capture of the entire object in a single shot. This is because the object's surface has undulations, causing varying distances from the lens to different parts of the surface, making it impossible to capture the entire surface at a single distance. Super depth of field involves adjusting the distance between the object and the lens, taking multiple shots to obtain multiple images. In these images, parts of the object's surface will be sharp, while other parts will be blurred. Super depth-of-field fusion extracts the sharp parts from these multiple images and combines them to create a final image where the entire object's surface is sharp. This process also generates a depth map. Assuming the total distance traveled during super depth-of-field shooting is h, each movement... If you take one more photo, then you will have taken a total of [number] photos. This picture. When viewing the images stacked together, the bottom layer represents the lowest point of the object's surface. If a portion of the object's surface is clearly visible in the bottom image, that area represents the lowest point of the object's surface. Similarly, the area clearly visible in the top image represents the highest point of the object's surface. This pattern continues, and the height of any clearly visible location in any image can be calculated. Let n be the nth image from the bottom (assuming the bottom image is the starting point of height 0). From this, the height values ​​of all areas can be calculated and drawn into a height map, which is the depth map of the hyper-depth-of-field fusion.

[0003] As can be seen from the above, when the depth of field is small, the same area can generally only be seen in certain locations. If a region appears sharp in one of the images, its depth assignment is relatively accurate. However, with a large depth of field, a certain area may appear sharp in multiple images, making it more difficult to calculate the actual height value that this area should belong to. When the sharpness values ​​calculated for a certain location on this set of images are plotted as a curve (the horizontal axis is the image number, and the vertical axis is the sharpness value), this graph is called the focus response curve for that point. The sharpness response peaks corresponding to high-resolution objectives with small depth of field are usually narrow and well-defined, making it easy to determine the peak position, i.e., the depth to which the point should belong. The sharpness response peaks corresponding to low-resolution objectives with large depth of field are usually wide, and may even have multiple peaks. This can cause problems such as edge misalignment, local jumps, depth drift in flat areas, and loss of height detail in the depth map, all of which are essentially due to incorrect peak position determination.

[0004] Currently, there's no perfect solution to the problem of inaccurate depth synthesis in large depth-of-field lenses. A common compromise is to select an image number at the middle position, or to further compare the sharpness values ​​of that location across several images and choose the position with the highest sharpness value. However, choosing the middle position can easily lead to excessively large depth differences between adjacent pixels. As for comparing sharpness values, since the calculation relies on the gradient of adjacent grayscale values, this method itself is inaccurate. It depends on the values ​​of adjacent pixels and is difficult to withstand noise, especially when comparing relatively sharp areas across multiple images. The calculated sharpness values ​​may be very close, and the numerical error is more likely due to noise and calculation errors, and does not necessarily represent the actual sharpness of that point. Summary of the Invention

[0005] To address the technical problems mentioned in the background section, this invention proposes a method, device, storage medium, and program product for large depth-of-field synthesis optimization based on learning different depth-of-field features.

[0006] To achieve the above-mentioned technical objectives, the technical solution of the present invention is as follows: A large depth-of-field synthesis optimization method based on learning different depth-of-field features is disclosed. The method inputs a sequence of actually acquired large depth-of-field images into a trained depth map generation model, which outputs a corresponding depth map. The depth map generation model includes a feature extraction module, a feature calibration module, and a depth map generation module. Training the depth map generation model comprises two training stages. In the first training stage, the output of the feature extraction module is connected to the input of the depth map generation module. This stage uses a sequence of small depth-of-field images and their corresponding depth maps as training samples. The small depth-of-field image sequence is input into the feature extraction module, which extracts feature maps from the input images and transmits them to the depth map generation module. The depth map generation module then generates the corresponding depth map based on the received feature maps. In the second training stage, two depth maps trained in the first training stage are configured... The system employs a trained feature extraction module and a feature calibration module, defined as the first feature extraction module and the second feature extraction module, respectively. The second-level training phase utilizes paired training samples, consisting of a sequence of images with identical content (large depth-of-field and a sequence of images with identical content). The large depth-of-field image sequence is input into the first feature extraction module, and the small depth-of-field image sequence is input into the second feature extraction module. The output of the first feature extraction module is connected to the input of the feature calibration module. A similarity loss is calculated between the calibrated feature map output by the feature calibration module and the feature map output by the second feature extraction module. This similarity loss is then used to optimize the feature calibration module. Furthermore, during the second-level training phase, the parameters of both feature extraction modules are fixed and they do not participate in the training process.

[0007] Furthermore, the paired training samples are obtained through cross-objective registration processing, which includes spatial registration and field-of-view matching. Spatial registration is to calculate the scale transformation between the two depth-of-field objective images based on the calibration board. Field-of-view matching is to stitch together the shallow depth-of-field image sequence at multiple positions so that the field-of-view coverage of the shallow depth-of-field image sequence is consistent with the field-of-view coverage of the large depth-of-field image sequence.

[0008] Furthermore, the similarity loss is achieved by combining cosine loss with discriminator loss.

[0009] Furthermore, when calculating the cosine loss, the feature map needs to be converted into feature vectors: Let the size of the feature map be [c,h,w], where c represents the channel, and h and w represent the image height and width, respectively. Expand h and w of the feature map into one dimension, that is, convert the feature map into c feature vectors of length h*w. Calculate the cosine loss for each of the c feature vectors and take the average as the final cosine loss result.

[0010] Secondly, the present invention also designs a computer device, including a processor and a memory, wherein the memory stores a computer program, and the processor is used to execute the computer program to implement the above-described large depth-of-field synthesis optimization method based on learning different depth-of-field features.

[0011] Furthermore, the present invention also provides a computer storage medium storing a computer program, which, when executed on a processor, implements the aforementioned large depth-of-field synthesis optimization method based on learning different depth-of-field features.

[0012] Finally, the present invention also provides a computer program product, including a computer program or instructions, which, when executed by a processor, implement the above-mentioned large depth-of-field synthesis optimization method based on learning different depth-of-field features. Attached Figure Description

[0013] Figure 1 This is a schematic diagram of the first stage of model construction and training in the embodiment; Figure 2 This is a schematic diagram of the second stage of model construction and training in the embodiment; Figure 3 This is a schematic diagram illustrating the application of the model in the embodiment. Detailed Implementation

[0014] The technical solution of this embodiment will be described in detail below with reference to the accompanying drawings.

[0015] The main framework of this embodiment is to optimize the depth of super-depth-of-field fusion under large depth-of-field, low-resolution objectives using deep learning. A deep learning model is trained, taking a sequence of stacked images of large depth-of-field, low-resolution focus points as input and a high-quality depth map as output. However, this approach requires a large amount of paired training data across objectives, targeting the same object, and with the same field of view. In actual acquisition, different objectives have different fields of view, resolution, distortion, depth of field, focal plane spacing, and illumination response. Small field-of-field, high-resolution objectives often require stitching together data from multiple locations to cover the range of a large field-of-field objective. Furthermore, the focal plane heights of different objectives do not naturally correspond one-to-one, resulting in high costs for large-scale acquisition and calibration. To better address these technical problems using deep learning, this embodiment includes the following four parts.

[0016] Part 1: The First Stage of Building and Training Deep Learning Models This stage does not include images with large depth of field (low resolution); instead, it uses image sequences with small depth of field (high resolution) to train the model. Image data from small depth-of-field lenses is easy to acquire, and the depth maps obtained from these image sequences using traditional super-depth-of-field synthesis methods are high-quality depth maps that can be used to train the model. The construction of the deep learning model is as follows: Figure 1As shown, the model is mainly divided into two parts: one part extracts features and generates feature maps, and the second part generates depth maps using a generator. The specific structure of these two parts uses convolution and attention mechanisms, such as LSNET or a bidirectional transformer generator. The main purpose of this model is to generate high-quality depth maps, and its input data is also a good image sequence (i.e., a sequence of high-resolution images with a small depth of field).

[0017] For the training data in this stage, a microscope with a small depth of field can be used for acquisition. Since it does not involve cross-acquisition with other large depth-of-field objectives, data acquisition is relatively simple. Prepare common samples, such as steel plates, rusty iron plates, screws, plastic products of various colors, wooden boards of different textures, insect specimens, and plant specimens. Place them under the microscope and acquire ultra-depth-of-field images. The ground truth required for the training data is a depth map obtained using existing ultra-depth-of-field synthesis techniques. Additionally, publicly available datasets can be used to expand the training dataset.

[0018] Part Two: Constructing Paired Training Samples Across Objectives For image sequences with large depth of field and low resolution, a feature calibration module needs to be constructed and trained. The training data requires that the same target be captured in both the small depth-of-field, high-resolution image sequence and the large depth-of-field, low-resolution image sequence. Because the two objectives have different fields of view, the objective with the smaller field of view needs to be moved and stitched together, and then cropped and matched with the image sequence from the other objective to achieve consistent image content. This embodiment concretizes the above process as cross-objective registration processing. Cross-objective registration processing includes spatial registration and field-of-view matching. Spatial registration is used to calculate the scale transformation between the two objective images based on a calibration board, i.e., the actual length relationship represented by the same pixel distance under the two objectives. Field-of-view matching is used to handle the problem of different fields of view between the two objectives. Typically, the field of view of a small depth-of-field objective is smaller than that of a large depth-of-field objective. For image sequences with small field-of-view focus stacks, multiple stitching positions are required to obtain an image sequence that can cover the large field of view. After cross-objective registration processing, paired training samples with corresponding pixel positions and corresponding focal plane relationships are obtained. These samples are used to train the subsequent feature calibration module.

[0019] Part Three: Construction and Training of the Deep Learning Model in the Second Stage (Feature Calibration Module) The weights of the feature extraction module trained in the first stage are fixed and no longer trained; this module is used for feature map extraction in this stage. Feature maps generated from high-resolution image sequences with shallow depth of field are considered ideal feature maps, while feature maps generated from low-resolution image sequences with large depth of field are considered flawed feature maps. The goal is to use a feature calibration module to generate feature maps similar to the ideal feature maps, such as... Figure 2As shown. The purpose of this part is to train a feature calibration module that can optimize the features extracted from large depth-of-field (low resolution) image sequences, transforming them into features similar to those extracted from small depth-of-field (high resolution) image sequences. Features generated from small depth-of-field high-resolution image sequences. Figure 1 Features generated from large depth-of-field, high-resolution image sequences serve as ideal feature maps (and as the target ground truth for training). Figure 2 It is a flawed feature map, which shows the features. Figure 2 After the feature calibration module, features are generated. Figure 1 Similar feature maps.

[0020] The method for training this part is to simultaneously pass both image sequences through the same feature extraction module to generate features. Figure 1 ,feature Figure 2 (At this point, it is required that the images captured by the two image sequences are exactly the same), features Figure 2 The feature calibration module generates an optimized feature map, which is then compared with the features... Figure 1 Similarity loss is calculated and used to optimize the feature calibration module, while the feature extraction module remains unaffected by the optimization (i.e., its weights are fixed). The feature calibration module's structure is designed based on convolution and attention, with input and output feature maps of equal size. This module is designed to be lightweight to control computational load, enabling real-time application in practical scenarios. Furthermore, the lightweight design allows the module to achieve good model performance even with limited data.

[0021] The similarity loss is implemented by combining cosine loss and discriminator loss. Specifically, the two loss methods can be combined by setting a balancing parameter. Cosine loss essentially calculates the similarity of feature vectors, but the feature extraction module obtains feature maps rather than feature vectors. Therefore, the feature map needs to be modified to accommodate the cosine loss calculation. The feature map size is [c, h, w]. We expand h and w of this feature map into one dimension, becoming [c, h*w]. At this point, we consider there to be c vectors, each with a length of h*w. Figure 1 Perform the same operation as optimizing the feature map, then calculate the c-th cosine loss, and finally take the average of the c cosine losses as the final cosine loss. Borrowing the idea of ​​the discriminator in GAN models, design a CNN-based discriminator to distinguish the features here. Figure 1 Is it similar to optimizing the feature map? The idea behind this is that the features... Figure 1Both the optimized feature maps and CNN feature maps can be easily integrated into CNN-based discriminators without requiring further transformations. Discriminators are frequently used in vision tasks involving domain transformations because they can effectively identify differences between two feature maps, thus enabling efficient model training. Therefore, this task, which involves calibrating features from two different domains, is well-suited to the characteristics of discriminators.

[0022] Part Four: Applications of Deep Learning Models In practical applications, such as Figure 3 As shown, a sequence of images with large depth of field and low resolution, after passing through a feature extraction module trained in the first stage, a feature calibration module trained in the second stage to optimize the feature map, and then through a trained depth map generation module, can generate a better depth map.

[0023] This embodiment also relates to a computer device, which includes a processor and a memory. The memory stores a computer program, and the processor executes the computer program to implement the above-described contents. This embodiment also relates to a computer storage medium storing a computer program, which, when executed on a processor, implements the above-described contents. This embodiment also relates to a computer program product, including a computer program or instructions, which, when executed by a processor, implements the above-described contents.

[0024] The embodiments are merely illustrative of the technical concept of the present invention and should not be construed as limiting the scope of protection of the present invention. Any modifications made to the technical solution based on the technical concept proposed in this invention shall fall within the scope of protection of this invention.

Claims

1. A method for optimizing large depth-of-field synthesis based on learning different depth-of-field features, characterized in that: A sequence of actually acquired large depth-of-field images is input into a trained depth map generation model, which outputs a corresponding depth map. The depth map generation model includes a feature extraction module, a feature calibration module, and a depth map generation module. Training the depth map generation model involves two training phases. In the first training phase, the output of the feature extraction module is connected to the input of the depth map generation module. This phase uses a sequence of small depth-of-field images and their corresponding depth maps as training samples. The small depth-of-field image sequence is input into the feature extraction module, which extracts the feature map from the input image and transmits it to the depth map generation module. The depth map generation module then generates the corresponding depth map based on the received feature map. In the second training phase, two feature extraction modules trained in the first training phase and one... The system comprises two feature extraction modules, defined as the first feature extraction module and the second feature extraction module, respectively. The second-level training phase uses paired training samples, consisting of a large depth-of-field image sequence and a small depth-of-field image sequence with identical content. The large depth-of-field image sequence is input into the first feature extraction module, and the small depth-of-field image sequence is input into the second feature extraction module. The output of the first feature extraction module is connected to the input of the feature calibration module. The calibrated feature map output by the feature calibration module and the feature map output by the second feature extraction module are compared using a similarity loss calculation. The similarity loss is then used to optimize the feature calibration module. During the second-level training phase, the parameters of the two feature extraction modules are fixed and they do not participate in the training.

2. The large depth-of-field synthesis optimization method based on learning different depth-of-field features according to claim 1, characterized in that: The paired training samples are obtained through cross-objective registration processing, which includes spatial registration and field-of-view matching. Spatial registration is to calculate the scale transformation between two depth-of-field objective images based on a calibration board. Field-of-view matching is to stitch together the shallow depth-of-field image sequence at multiple positions so that the field-of-view coverage of the shallow depth-of-field image sequence is consistent with the field-of-view coverage of the large depth-of-field image sequence.

3. The large depth-of-field synthesis optimization method based on learning different depth-of-field features according to claim 1, characterized in that: The similarity loss is achieved by combining cosine loss with discriminator loss.

4. The large depth-of-field synthesis optimization method based on learning different depth-of-field features according to claim 2, characterized in that: When calculating the cosine loss, the feature map needs to be converted into feature vectors: Let the size of the feature map be [c,h,w], where c represents the channel, and h and w represent the image height and width, respectively. Expand h and w of the feature map into one dimension, that is, convert the feature map into c feature vectors of length h*w. Calculate the cosine loss for each of the c feature vectors and take the average as the final cosine loss result.

5. A computer device comprising a processor and a memory, wherein the memory stores a computer program, characterized in that: The processor is used to execute the computer program to implement the large depth-of-field synthesis optimization method based on learning different depth-of-field features as described in any one of claims 1-4.

6. A computer storage medium storing a computer program, characterized in that: When the computer program is executed on the processor, it implements the large depth-of-field synthesis optimization method based on learning different depth-of-field features as described in any one of claims 1-4.

7. A computer program product, comprising a computer program or instructions, characterized in that: When the computer program or instructions are executed by the processor, they implement the large depth-of-field synthesis optimization method based on learning different depth-of-field features as described in any one of claims 1-4.

Images