Image processing method and device, electronic equipment, storage medium and microscope

Abstract

The embodiment of the invention provides an image processing method and device, electronic equipment, a storage medium and a microscope, and relates to the field of image processing. Comprising the steps of obtaining image features corresponding to a to-be-processed image through a pre-trained self-supervised feature extraction network; clustering the image features to obtain regional image features corresponding to the at least one region; determining a reference depth map corresponding to the to-be-processed image according to the regional image features, the initial depth map corresponding to the to-be-processed image and the confidence map; and inputting the regional image features into a depth regression network, continuously optimizing the depth regression network through the reference depth map and the initial depth map, and taking a depth map output by the optimized depth regression network as a target depth map corresponding to the to-be-processed image. The method can effectively optimize the noise in the depth map through setting the reference depth map and optimizing the fitting depth regression network, does not need to depend on a large amount of label data to carry out supervision training, and saves time and labor.

Description

Technical Field

[0001] This disclosure relates to the field of image processing, and more particularly to an image processing method, apparatus, electronic device, storage medium, and microscope. Background Technology

[0002] Depth maps are an important tool for representing three-dimensional information about a scene and have wide applications in computer vision and image processing.

[0003] There are various ways to acquire depth maps. For example, 3D ultra-depth-of-field microscopy can obtain the depth information of the observed object by using multiple images with different information, thus enabling the preliminary construction of depth information from a full-depth-of-field image. In the field of autonomous driving, technologies such as LiDAR can be used to acquire the depth information of the observed object, thereby enabling the preliminary construction of depth information from images acquired by cameras.

[0004] The initial depth map contains a lot of noise. Figure 1 Example diagrams showing panoramic depth images and corresponding preliminary depth maps in related technologies are provided, such as... Figure 1 As shown, Figure 1 (a) is a full depth-of-field image obtained by a super depth-of-field microscope. Figure 1 (b) is the initial depth map constructed using a super depth-of-field microscope. Figure 1 (b) The area indicated by the ellipse or circle contains a lot of noise with inaccurate depth information.

[0005] Suppressing noise in the initial depth map has become an urgent technical problem to be solved. Summary of the Invention

[0006] The purpose of this disclosure is to provide an image processing method, apparatus, electronic device, storage medium, and microscope.

[0007] To solve the above-mentioned technical problems, the embodiments of this disclosure are achieved through the following aspects.

[0008] According to a first aspect of the present disclosure, an image processing method is provided, the method comprising: acquiring image features corresponding to an image to be processed through a pre-trained self-supervised feature extraction network, wherein the image to be processed is a full depth image; clustering the image features to obtain region image features corresponding to at least one region; determining a reference depth map corresponding to the image to be processed based on the region image features, an initial depth map corresponding to the image to be processed, and a confidence map; inputting the region image features into a depth regression network, continuously optimizing the depth regression network through the reference depth map and the initial depth map, and using the depth map output by the optimized depth regression network as the target depth map corresponding to the image to be processed.

[0009] According to a second aspect of the present disclosure, an image processing method is provided, the method comprising: acquiring first image features corresponding to an image to be processed through a pre-trained self-supervised feature extraction network, wherein the image to be processed is a full depth image; acquiring second image features corresponding to a fused image through the self-supervised feature extraction network, wherein the fused image is obtained by fusing the image to be processed and a confidence map corresponding to the image to be processed; inputting an initial depth map corresponding to the image to be processed, the first image features, and the second image features into a pre-constructed attention module to obtain an attention depth map corresponding to the image to be processed; clustering the attention depth map to obtain at least one region; acquiring a target region depth map corresponding to each region in the at least one region, and merging the target region depth maps corresponding to each region to obtain a target depth map corresponding to the image to be processed.

[0010] According to a third aspect of the present disclosure, an image processing apparatus is provided, the apparatus comprising:

[0011] A first acquisition module is used to acquire image features corresponding to the image to be processed through a pre-trained self-supervised feature extraction network, wherein the image to be processed is a full-depth image; a feature clustering module is used to cluster the image features to obtain region image features corresponding to at least one region; a first determination module is used to determine a reference depth map corresponding to the image to be processed based on the region image features, the initial depth map and the confidence map corresponding to the image to be processed; a first optimization module is used to input the region image features into a depth regression network, continuously optimize the depth regression network through the reference depth map and the initial depth map, and use the output of the optimized depth regression network as the target depth map corresponding to the image to be processed.

[0012] According to a fourth aspect of the present disclosure, an image processing apparatus is provided, the apparatus comprising: a second acquisition module, configured to acquire a first image feature corresponding to an image to be processed through a pre-trained self-supervised feature extraction network, wherein the image to be processed is a full depth image; the second acquisition module is further configured to acquire a second image feature corresponding to a fused image through the self-supervised feature extraction network, wherein the fused image is obtained by fusing the image to be processed and a confidence map corresponding to the image to be processed; a second determination module, configured to input an initial depth map corresponding to the image to be processed, the first image feature, and the second image feature into a pre-constructed attention module to obtain an attention depth map corresponding to the image to be processed; a depth map clustering module, configured to cluster the attention depth map to obtain at least one region; and a second optimization module, configured to acquire a target region depth map corresponding to each region in the at least one region, and merge the target region depth maps corresponding to each region to obtain a target depth map corresponding to the image to be processed.

[0013] According to a fifth aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to perform the steps of the image processing method described in the first or second aspect above.

[0014] According to a sixth aspect of the present disclosure, a computer-readable storage medium is provided that stores one or more programs, which, when executed by an electronic device including a plurality of applications, cause the electronic device to perform the steps of the image processing method described in the first or second aspect.

[0015] According to a seventh aspect of the present disclosure, a microscope is provided, the microscope being a panoramic depth microscope, including an image acquisition device for acquiring panoramic depth images, and an image processing device as described in the third or fourth aspect for acquiring a target depth map of the panoramic depth images.

[0016] One of the above technical solutions has the following advantages or beneficial effects: it effectively utilizes the high-level and low-level features contained in the features extracted by the self-supervised network to establish the connection between depth information and panoramic depth image; by setting a reference depth map, it optimizes the method of fitting the depth regression network, optimizes the depth information of low-confidence areas in the initial depth map, effectively optimizes the noise in the depth map, and does not require a large amount of labeled data for supervised training, saving time and effort and being easy to implement.

[0017] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure.

[0018] Other features and advantages of this disclosure will be described in detail in the following detailed description section. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 An example diagram showing a panoramic depth image and a corresponding preliminary depth map in related technologies is provided.

[0021] Figure 2 This illustration shows a flowchart of an image processing method provided in an embodiment of the present disclosure;

[0022] Figure 3 Example diagrams showing panoramic depth images, preliminary depth maps, and confidence maps provided in embodiments of this disclosure are shown.

[0023] Figure 4 This illustration shows another schematic flowchart of the image processing method provided in an embodiment of the present disclosure;

[0024] Figure 5 This diagram illustrates a processing block diagram of an image processing method provided in an embodiment of the present disclosure;

[0025] Figure 6 This diagram illustrates yet another flow chart of the image processing method provided in an embodiment of the present disclosure;

[0026] Figure 7 Another processing block diagram of the image processing method provided in the embodiments of this disclosure is shown;

[0027] Figure 8 This diagram illustrates yet another flow chart of the image processing method provided in an embodiment of the present disclosure;

[0028] Figure 9 This diagram illustrates yet another processing block diagram of the image processing method provided in an embodiment of the present disclosure;

[0029] Figure 10 This diagram illustrates yet another flow chart of the image processing method provided in an embodiment of the present disclosure;

[0030] Figure 11 This diagram illustrates yet another processing block diagram of the image processing method provided in an embodiment of the present disclosure;

[0031] Figure 12This illustration shows another effect diagram of the image processing provided in the embodiments of this disclosure;

[0032] Figure 13 This diagram shows a block diagram of an image processing apparatus provided in an embodiment of this application;

[0033] Figure 14 This diagram illustrates a block diagram of another image processing apparatus provided in an embodiment of this application;

[0034] Figure 15 A schematic diagram of the hardware structure of an electronic device for performing the image processing method provided in the embodiments of this disclosure. Detailed Implementation

[0035] To enable those skilled in the art to better understand the technical solutions in this disclosure, the technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. Based on the embodiments in this disclosure, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this disclosure.

[0036] In related technologies, depth maps can be denoised and optimized using image processing algorithms, such as mean filtering, Gaussian filtering, or median filtering, to reduce noise and improve image quality. However, this method can only optimize noise to a limited extent, is inefficient, and produces poor results. With the advent of deep learning, supervised training networks can be used for adaptive denoising and optimization of depth maps. However, supervised training requires a large amount of training data with real depth labels. Since image information has a wide range of categories and few definable repetitive samples, it is difficult to construct a large amount of effective training data, and the process is time-consuming and labor-intensive.

[0037] Figure 2 This illustration shows a flowchart of an image processing method provided in an embodiment of the present disclosure, such as... Figure 2 As shown, the method may include the following steps:

[0038] In step S101, image features corresponding to the image to be processed are obtained through a pre-trained self-supervised feature extraction network.

[0039] The image to be processed is a full-depth image.

[0040] Self-supervised feature extraction networks can be trained using massive amounts of image data to automatically extract image features without the need for manually labeled data.

[0041] The self-supervised feature extraction network described above can be trained in various ways. One possible implementation is to use DINOv2 (Dual-Stage Implicit Object-Oriented Network), a Transformer-based visual model that effectively combines image classification and object detection tasks. Another possible implementation is to use FPN (Feature Pyramid Networks) as the self-supervised feature extraction network. It is understood that the image features obtained by the self-supervised feature extraction network can include rich high-level semantic features as well as rich low-level detail features.

[0042] In some embodiments, before obtaining the image features corresponding to the image to be processed, the image to be processed can be preprocessed to adjust its size to meet the input requirements of the self-supervised feature extraction network. For example, the image adjustment and cropping methods provided by the OpenCV library in Python can be used to adjust and crop the image size to meet the requirements of the self-supervised feature extraction network, such as M*M.

[0043] The image features of the image to be processed obtained by the pre-trained self-supervised feature extraction network can be represented as N*N*C. In some embodiments, the image features can also be transformed into the same size as the image to be processed by upsampling or downsampling. For example, the image features can be transformed into M*M*C by upsampling or downsampling, where C is the dimension of the image features corresponding to each pixel in the image to be processed. It can be understood that the C-dimensional image features include the position information of the pixels, which can be used to perform clustering based on the image features.

[0044] In step S102, the image features are clustered to obtain the region image features corresponding to at least one region.

[0045] Taking the image to be processed provided by the 3D ultra-depth microscope as an example, since the depth information in the initial depth map is less reliable in the low confidence region, if a depth regression network is directly constructed based on the initial depth map, a lot of noise will be introduced.

[0046] Clustering can be performed based on image features to divide the image to be processed into different regions, which makes it easier to combine the confidence map to process the regions with low confidence, thereby helping to optimize the noise in the depth map.

[0047] In one possible implementation, image features can be clustered using K-Means clustering to obtain at least one region's corresponding region image features.

[0048] Specifically, image features can be clustered through the following steps to obtain at least one region's corresponding region image features.

[0049] Step 10: Perform K-Means clustering based on image features to obtain at least one region.

[0050] Step 11: Segment the image features according to at least one region to obtain regional image features.

[0051] In step S103, a reference depth map corresponding to the image to be processed is determined based on the regional image features, the initial depth map corresponding to the image to be processed, and the confidence map.

[0052] The initial depth map is Figure 1 (b) shows the initial constructed depth map. The confidence map is the confidence level of each pixel in the image to be processed, obtained during the process of acquiring the full depth image.

[0053] Figure 3 Example diagrams showing the panoramic depth image, preliminary depth map, and confidence map provided in embodiments of this disclosure are shown, such as... Figure 3 As shown, Figure 3 (c) is an example image of the confidence map provided in the embodiments of this disclosure. The confidence map and the preliminary depth map have the same size as the full depth image. In the confidence map, the gray level of each pixel represents the confidence level of the depth information corresponding to that pixel in the initial depth map.

[0054] Understandably, confidence maps may also introduce noise during the generation process, for example... Figure 3 There are two flying lines in (a), which obviously correspond to a shallower depth. Figure 3 In the confidence map in (c), some pixels at the positions corresponding to the two flying lines have low confidence (i.e., the corresponding pixels in the confidence map are black). This is also a problem that needs to be solved in the depth map optimization based on the confidence map.

[0055] Figure 4 This illustration shows another schematic flowchart of the image processing method provided in an embodiment of the present disclosure, such as... Figure 4 As shown, step S103 may include the following steps.

[0056] In step S1031, a target region is determined in at least one region based on the region image features and the confidence map.

[0057] In the target area of ​​the confidence map, the proportion of pixels with a confidence level lower than the first preset confidence threshold is higher than the preset proportion threshold.

[0058] Since image features include the positional features of pixels, different regions obtained through clustering are complete and unified semantic regions with different dimensions. During the acquisition of reference depth maps, only the target region (the region with a relatively low degree of overlap with the high confidence level) can be processed, while the non-target region (the region with a relatively high degree of overlap with the high confidence level) can be left unprocessed, which can effectively reduce the impact of self-supervised feature extraction errors.

[0059] In some embodiments, a target region in at least one region can be determined based on region image features and a confidence map by the following steps.

[0060] In step 20, the proportion of low confidence in any candidate region is obtained.

[0061] The low confidence ratio represents the proportion of pixels with a confidence level lower than the first preset confidence threshold in the candidate region to the total number of pixels in the candidate region.

[0062] In step 21, one or more candidate regions with a low confidence ratio higher than a preset ratio threshold are selected as target regions.

[0063] In some possible implementations, the target region corresponding to any region in at least one region can be determined by the following formula. For example, the region that satisfies the following formula can be used as the target region.

[0064]

[0065] in, Let i be the mask for the i-th region. For the i-th region, the confidence level is less than the first preset confidence threshold C. Th The intersection of the regions, C Th The first preset reliability threshold is preferably 0.9, M. Th This is the preset ratio threshold, which can be set as needed.

[0066] In step S1032, the depth values ​​of the pixels in the target region of the initial depth map are set to a specified depth to obtain a reference depth map.

[0067] In some embodiments, the specified depth is a preset depth value, or the average depth of pixels with a confidence level higher than a second preset confidence threshold in the target region of the confidence map.

[0068] By setting the depth values ​​of pixels in the target region of the initial depth map to a specified depth, the depth of the high-confidence region is implicitly diffused to the low-confidence region through the relationship between depth and the image to be processed, without processing the high-confidence region, thus effectively reducing the impact of feature extraction errors.

[0069] Figure 5 This diagram illustrates a processing block diagram of an image processing method provided in an embodiment of the present disclosure, such as... Figure 5 As shown, the image features of the image to be processed are extracted by a self-supervised feature extraction network, and the image features are clustered to obtain at least one region of the image to be processed. Based on the region image features, the initial depth map and confidence map corresponding to the image to be processed, a specified depth setting is applied to the target region with low confidence, and the reference depth map corresponding to the image to be processed is determined.

[0070] contrast Figure 5 The initial and reference depth maps show that clustering identifies at least one region and identifies low-confidence target regions within that region. Setting a specified depth can reduce noise in the initial depth map. (See also...) Figure 1 (b) The noise region is indicated by a circle or ellipse. For regions with high confidence in the initial depth map (e.g., the region corresponding to the flying line in the image), even if there is some noise in the confidence map (the flying line region in the confidence map corresponds to some black pixels), the large proportion of high-confidence points in this region effectively prevents it from being identified as a target region. Furthermore, even if this region is identified as a target region, setting it as the average depth of pixels with a confidence level higher than the second preset confidence threshold can diffuse the depth of the high-confidence region to the low-confidence region through connections, thereby obtaining accurate depth information for that region in the reference depth map. Figure 5 As can be seen, although the reference depth map is a significant improvement over the initial depth map, there are still some point noises that need to be optimized.

[0071] In step S104, the region image features are input into the depth regression network. The depth regression network is continuously optimized using the reference depth map and the initial depth map. The depth map output by the optimized depth regression network is used as the target depth map corresponding to the image to be processed.

[0072] To avoid overfitting to incorrect reference depths, depth regression networks should avoid excessive complexity and large parameters. One possible implementation is a CNN (Convolutional Neural Networks) + Batch Normalization (BN) + ReLU (Rectified Linear Unit) structure. This deep regression network is a feedforward neural network with a deep structure. It takes image features (i.e., regional image features of each area) as input and outputs an optimized depth map.

[0073] By adopting the above technical solution, the high-level and low-level features contained in the features extracted by the self-supervised network are effectively utilized to establish the connection between depth information and panoramic depth image. By setting a reference depth map, the method of fitting the depth regression network is optimized, the depth information of the low confidence area in the initial depth map is optimized, and the noise in the depth map is effectively optimized. At the same time, it does not need to rely on a large amount of labeled data for supervised training, which saves time and effort and is easy to implement.

[0074] Figure 6 This illustration shows yet another flowchart of the image processing method provided in an embodiment of the present disclosure, such as... Figure 6 As shown, step S104 may include the following steps.

[0075] In step S1041, a loss function is constructed using the initial depth map, the reference depth map, and the depth map output by the depth regression network.

[0076] In some embodiments, the loss function can be constructed using the following formula 2.

[0077]

[0078] Where L is the loss function, D pre D is the depth map output by the deep regression network. raw For the initial depth map, D ref For reference, in the depth map, || represents the 1-norm. In the depth map, regions with high confidence can be further divided into K sub-regions with different confidence levels, w i w is the preset weight corresponding to the sub-region with the i-th confidence level. ref M represents the preset weights corresponding to the target region. ref This refers to the proportion of pixels whose confidence level is lower than the first preset confidence threshold in Formula 1.

[0079] In another embodiment, the loss function can further consider a smoothing loss function, thereby further enhancing the smoothness of the depth map output by the deep regression network. The loss function can be constructed using the following formula (Formula 3).

[0080]

[0081] Among them, w TV To smooth the weights of the loss function, L TV These are constraints that limit smoothness.

[0082] In step S1042, the depth regression network is iteratively optimized according to the loss function. If the preset stopping condition is met, the depth map output by the depth regression network is used as the target depth map corresponding to the image to be processed.

[0083] Figure 7 Another processing block diagram of the image processing method provided in this disclosure embodiment is shown, such as... Figure 7 As shown, a deep regression network can take image features (i.e., regional image features of each region) as input and output an optimized depth map. The loss function, constructed based on the initial depth map, reference depth map, and the depth map output by the deep regression network, iteratively optimizes the deep regression network. The network parameters are updated through backpropagation, and this process is repeated multiple times. When a preset stopping condition is met, the depth map output by the deep regression network is used as the target depth map corresponding to the image to be processed.

[0084] In some embodiments, the stopping condition can be at least one of the following:

[0085] The value of the loss function is less than or equal to the preset loss function threshold;

[0086] The number of iterations is greater than or equal to the preset iteration threshold.

[0087] By adopting the above technical solution, when designing the loss function of the depth regression network, factors such as the accuracy, smoothness, and fidelity of the depth map are comprehensively considered, which can ensure that the optimization of the depth map can effectively improve the accuracy and visualization effect of the depth information in the target depth map.

[0088] Figure 8 This illustration shows yet another flowchart of the image processing method provided in an embodiment of the present disclosure, such as... Figure 8 As shown, the method may include the following steps.

[0089] In step S201, the first image features corresponding to the image to be processed are obtained through a pre-trained self-supervised feature extraction network.

[0090] The image to be processed is a full-depth image.

[0091] For specific steps, please refer to the steps in S101, which will not be elaborated here.

[0092] In step S202, the second image features corresponding to the fused image are obtained through a self-supervised feature extraction network.

[0093] The fused image is obtained by fusing the image to be processed and the confidence map corresponding to the image to be processed.

[0094] For specific steps, please refer to the steps in S101, which will not be elaborated here.

[0095] In step S203, the initial depth map, the first image feature, and the second image feature corresponding to the image to be processed are input into the pre-constructed attention module to obtain the attention depth map corresponding to the image to be processed.

[0096] Attention mechanism is a widely used technique in deep learning. The core idea of ​​attention mechanism is to dynamically select and weight input data based on contextual information, so as to better capture important information.

[0097] Figure 9 This diagram illustrates yet another processing block diagram of the image processing method provided in this disclosure, such as... Figure 9 As shown, the initial depth map corresponding to the image to be processed is used as the value matrix of the attention module, the first image feature is used as the query matrix of the attention module, and the second image feature is used as the key matrix of the attention module. The attention module effectively mines the relationship between confidence and the initial depth map, organically combining the initial depth map and the confidence map, which can significantly mitigate the impact of high-confidence noise.

[0098] In step S204, the attention depth map is clustered to obtain at least one region.

[0099] In some possible implementations, the attention depth map can be clustered using K-Means clustering to obtain at least one region.

[0100] In step S205, the target region depth map corresponding to each region in at least one region is obtained, and the target region depth maps corresponding to each region are merged to obtain the target depth map corresponding to the image to be processed.

[0101] In this embodiment, the relationship between confidence and the initial depth map is effectively explored through the attention mechanism, thereby enabling a more ideal division of the depth region in the initial depth map. Then, combined with the method of iterative optimization of the region, the depth map based on the attention mechanism can be effectively combined with the fitting ability of the optimization fitting to fit details.

[0102] Figure 10This illustration shows yet another flowchart of the image processing method provided in an embodiment of the present disclosure, such as... Figure 10 As shown, step S205 may include the following steps.

[0103] In step S2051, a target region is determined from at least one region.

[0104] In step S2052, the depth map of the target area corresponding to the target area is obtained.

[0105] In some embodiments, the target region depth map corresponding to the target region can be obtained through the following steps:

[0106] The first image features corresponding to the target region are input into the deep regression network. The deep regression network is continuously optimized by using the attention depth map and the initial depth map corresponding to the target region. The depth map output by the optimized deep regression network is used as the target region depth map corresponding to the target region.

[0107] When iteratively optimizing the target region, the design of the loss function can still refer to the design of the first aspect embodiment, and will not be elaborated here.

[0108] In step S2053, the next target region is determined from the remaining regions, and the step of obtaining the target region depth map corresponding to the target region is repeated until at least one region has been traversed.

[0109] Figure 11 This diagram illustrates yet another processing block diagram of the image processing method provided in this disclosure, such as... Figure 11 As shown, the attention depth map is first clustered to obtain at least one region. For each region, the corresponding depth regression network is iteratively optimized, and the depth map output by the optimized depth regression network is used as the target region depth map. After obtaining the target region depth maps for each region, they can be merged to obtain the target depth map for the image to be processed.

[0110] By adopting the above technical solution, the high-level and low-level features contained in the features extracted by the self-supervised network are effectively utilized. The connection between depth information and panoramic depth image is established through the attention mechanism, which can effectively optimize the noise in the depth map. At the same time, combined with the method of regional optimization fitting, it can effectively combine the depth discrimination advantage of the depth map based on the attention mechanism with the fitting ability of the optimization fitting to fit details. It can suppress both small areas of high confidence noise and large areas of low confidence noise. It also does not require a large amount of labeled data for supervised training, which saves time and effort and is easy to implement.

[0111] Figure 12This illustration shows another effect diagram of the image processing provided in the embodiments of this disclosure, such as... Figure 12 As shown, where, Figure 12 (a) is the fused image of the image to be processed and the confidence map. Figure 12 (b) is the confidence plot. Figure 12 (c) is the initial depth map. Figure 12 (d) is the final target depth map. Figure 12 (e) is the difference between the initial depth map and the target depth map. It can be seen that the technical solution of this application can suppress high-confidence noise in small areas and low-confidence noise in large areas, especially the noise in large low-confidence areas.

[0112] Figure 13 This diagram illustrates a block diagram of an image processing apparatus according to an embodiment of this application, such as... Figure 13 As shown, the image processing apparatus 300 includes:

[0113] The first acquisition module 310 is used to acquire image features corresponding to the image to be processed through a pre-trained self-supervised feature extraction network;

[0114] The feature clustering module 320 is used to cluster image features to obtain at least one region corresponding to each region;

[0115] The first determining module 330 is used to determine the reference depth map corresponding to the image to be processed based on the regional image features, the initial depth map corresponding to the image to be processed, and the confidence map;

[0116] The first optimization module 340 is used to input the regional image features into the depth regression network, continuously optimize the depth regression network by using the reference depth map and the initial depth map, and use the output of the optimized depth regression network as the target depth map corresponding to the image to be processed.

[0117] Optionally, the first determining module 330 is also used for:

[0118] Based on the regional image features and the confidence map, a target region is determined in at least one region, wherein in the target region of the confidence map, the proportion of pixels with a confidence level lower than a first preset confidence threshold is higher than a preset proportion threshold.

[0119] The depth values ​​of pixels in the target region of the initial depth map are set to a specified depth to obtain a reference depth map.

[0120] Optionally, the first optimization module 340 is also used for:

[0121] The loss function is constructed using the initial depth map, the reference depth map, and the depth map output by the depth regression network.

[0122] The depth regression network is iteratively optimized according to the loss function. When the loss function meets the preset stopping condition, the depth map output by the depth regression network is used as the target depth map corresponding to the image to be processed.

[0123] Optionally, the feature clustering module 320 is also used for:

[0124] K-Means clustering is performed based on image features to obtain at least one region;

[0125] The image features are segmented according to at least one region to obtain regional image features.

[0126] The device 300 provided in this application embodiment can execute the methods in the preceding method embodiments and realize the functions and beneficial effects of the methods in the preceding method embodiments, which will not be repeated here.

[0127] Figure 14 This diagram illustrates a block diagram of another image processing apparatus provided in an embodiment of this application, such as... Figure 14 As shown, the image processing apparatus 400 includes:

[0128] The second acquisition module 410 is used to acquire the first image features corresponding to the image to be processed through a pre-trained self-supervised feature extraction network;

[0129] The second acquisition module 410 is also used to acquire the second image features corresponding to the fused image through a self-supervised feature extraction network;

[0130] The second determining module 420 is used to input the initial depth map, the first image feature and the second image feature corresponding to the image to be processed into the pre-constructed attention module to obtain the attention depth map corresponding to the image to be processed.

[0131] The depth map clustering module 430 is used to cluster the attention depth map to obtain at least one region;

[0132] The second optimization module 440 is used to obtain the target region depth map corresponding to each region in at least one region, and merge the target region depth maps corresponding to each region to obtain the target depth map corresponding to the image to be processed.

[0133] Optionally, the second optimization module 440 is also used for:

[0134] Determine the target area from at least one region;

[0135] Obtain the depth map of the target region;

[0136] Determine the next target region from the remaining regions, and repeat the steps of obtaining the target region depth map corresponding to the target region until at least one region has been traversed.

[0137] The device 400 provided in this application embodiment can execute the methods in the preceding method embodiments and realize the functions and beneficial effects of the methods in the preceding method embodiments, which will not be repeated here.

[0138] Figure 15 This diagram illustrates the hardware structure of an electronic device implementing embodiments of the present disclosure, such as... Figure 15 As shown, at the hardware level, the electronic device includes at least one processor, and optionally, an internal bus, a network interface, and memory. The memory may include main memory, such as high-speed random-access memory (RAM), or it may also include non-volatile memory, such as at least one disk drive. Of course, the electronic device may also include other hardware required for its functions.

[0139] The processor, network interface, and memory can be interconnected via an internal bus, which can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. This bus can be categorized as an address bus, data bus, control bus, etc. For ease of illustration, only a single bidirectional arrow is used in this diagram, but this does not imply that there is only one bus or one type of bus.

[0140] Memory stores programs. Specifically, the program may include program code, which includes at least one computer operation instruction. Memory may include main memory and non-volatile memory, and provides instructions and data to the processor.

[0141] At least one processor reads a corresponding computer program from non-volatile memory into memory and then runs it, forming a device for locating a target user at the logical level. At least one processor executes the program stored in memory and specifically performs the method disclosed in the embodiments of the first or second aspect, achieving the functions and beneficial effects of the methods described in the foregoing method embodiments, which will not be repeated here.

[0142] The methods disclosed in the embodiments shown in the first or second aspect of this disclosure can be applied to at least one processor, or implemented by at least one processor. The processor may be an integrated circuit with signal processing capabilities. During implementation, each step of the above method can be completed by integrated logic circuits in the hardware or by instructions in software form within at least one processor. The processor may be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it may also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The methods, steps, and logic block diagrams disclosed in the embodiments of this disclosure can be implemented or executed. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this disclosure can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software module can reside in a mature storage medium in the field, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method.

[0143] The electronic device can also execute the methods described in the preceding method embodiments and achieve the functions and beneficial effects of the methods described in the preceding method embodiments, which will not be repeated here.

[0144] Of course, in addition to software implementation, the electronic device disclosed herein does not exclude other implementation methods, such as logic devices or a combination of hardware and software, etc. In other words, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0145] This disclosure also proposes a computer-readable storage medium that stores one or more programs, which, when executed by at least one processor, implement the methods disclosed in the embodiments of the first or second aspect and achieve the functions and beneficial effects of the methods described in the foregoing method embodiments, which will not be repeated here.

[0146] The computer-readable storage medium includes read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks, etc.

[0147] Furthermore, this disclosure also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium. The computer program includes program instructions that, when executed by a computer, implement the following process: the method disclosed in the embodiments of the first or second aspect and achieve the functions and beneficial effects of the methods described in the foregoing method embodiments, which will not be repeated here.

[0148] This application provides a computer program product, which includes a computer program. When executed by a processor, the computer program implements the various processes of the method disclosed in the embodiments of the first or second aspect and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0149] This application provides a microscope, which is a panoramic depth microscope, including an image acquisition device for acquiring panoramic depth images, and an image processing device as described in the third or fourth aspect for acquiring a target depth map of the panoramic depth image.

[0150] In summary, the above description is merely a preferred embodiment of this disclosure and does not limit the scope of protection of this disclosure. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

[0151] The systems, modules, or units described in the above embodiments can be implemented by a computer or entity, or by a product with a certain function. A typical implementation device is a computer. Specifically, a computer can be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smartphone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or any combination of these devices.

[0152] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can store information accessible to a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0153] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0154] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

Claims

1. A method of image processing, characterized by, The method comprises: obtaining image features corresponding to a to-be-processed image through a pre-trained self-supervised feature extraction network, the to-be-processed image being a panoramic depth image; clustering the image features to obtain region image features corresponding to at least one region; determining a reference depth map corresponding to the to-be-processed image according to the region image features, an initial depth map corresponding to the to-be-processed image, and a confidence map; inputting the region image features into a depth regression network, continuously optimizing the depth regression network through the reference depth map and the initial depth map, and taking a depth map output by the optimized depth regression network as a target depth map corresponding to the to-be-processed image.

2. The self-supervised feature extraction network is a DINOv2 network, and the clustering of the image features to obtain region image features corresponding to at least one region comprises: performing K-Means clustering on the image features to obtain the at least one region; segmenting the image features according to the at least one region to obtain the region image features.

3. The method of claim 1, wherein, The determination of the reference depth map corresponding to the to-be-processed image according to the region image features, the initial depth map corresponding to the to-be-processed image, and the confidence map comprises: determining a target region in the at least one region based on the region image features and the confidence map, wherein, in the target region of the confidence map, the proportion of pixel points with a confidence lower than a first preset confidence threshold is higher than a preset proportion threshold; setting the depth values of pixel points in the target region of the initial depth map to a specified depth to obtain the reference depth map.

4. The method of claim 3, wherein, The determination of the target region in the at least one region based on the region image features and the confidence map comprises: obtaining a low-confidence proportion of any candidate region, the low-confidence proportion representing the proportion of pixel points with a confidence lower than the first preset confidence threshold in the candidate region relative to the total pixel points in the candidate region; taking one or more candidate regions with the low-confidence proportion higher than the preset proportion threshold as the target region.

5. The method of claim 3, wherein, The specified depth is a preset depth value, or the average value of the depths of pixel points with a confidence higher than a second preset confidence threshold in the target region of the confidence map.

6. The method according to any one of claims 1-5, characterized in that, The continuous optimization of the depth regression network through the reference depth map and the initial depth map, and the taking of a depth map output by the optimized depth regression network as the target depth map corresponding to the to-be-processed image comprise: constructing a loss function through the initial depth map, the reference depth map, and the depth map output by the depth regression network; iteratively optimizing the depth regression network according to the loss function, and taking a depth map output by the depth regression network as the target depth map corresponding to the to-be-processed image when a preset stopping condition is met, wherein the loss function comprises a smoothness loss function.

7. A method of image processing, characterized by, The method comprises: obtaining first image features corresponding to a to-be-processed image through a pre-trained self-supervised feature extraction network, the to-be-processed image being a panoramic depth image; obtaining, by the self-supervised feature extraction network, a second image feature corresponding to the fusion image, the fusion image being obtained by fusing the to-be-processed image and a confidence map corresponding to the to-be-processed image; inputting, into a pre-constructed attention module, an initial depth map corresponding to the to-be-processed image, the first image feature and the second image feature, to obtain an attention depth map corresponding to the to-be-processed image; clustering the attention depth map to obtain at least one region; obtaining a target region depth map corresponding to each region of the at least one region respectively, and merging the target region depth map corresponding to each region respectively to obtain a target depth map corresponding to the to-be-processed image.

8. The method of claim 7, wherein, The obtaining of the target region depth map corresponding to each region of the at least one region respectively comprises: determining a target region from the at least one region; obtaining a target region depth map corresponding to the target region; determining a next target region from the remaining regions, and repeating the step of obtaining the target region depth map corresponding to the target region until the at least one region is traversed.

9. The method of claim 8, wherein, The obtaining of the target region depth map corresponding to the target region comprises: inputting the first image feature corresponding to the target region into a depth regression network, continuously optimizing the depth regression network by using the attention depth map corresponding to the target region and the initial depth map corresponding to the target region, and taking a depth map output by the optimized depth regression network as the target region depth map corresponding to the target region.

10. An apparatus for image processing, characterized by The device comprises: a first obtaining module configured to obtain an image feature corresponding to a to-be-processed image by using a pre-trained self-supervised feature extraction network, the to-be-processed image being a panoramic depth image; a feature clustering module configured to cluster the image feature to obtain a region image feature corresponding to at least one region respectively; a first determining module configured to determine a reference depth map corresponding to the to-be-processed image according to the region image feature, an initial depth map corresponding to the to-be-processed image and a confidence map; a first optimization module configured to input the region image feature into a depth regression network, continuously optimize the depth regression network by using the reference depth map and the initial depth map, and take an output of the optimized depth regression network as a target depth map corresponding to the to-be-processed image.

11. An apparatus for image processing, characterized by The method comprises: a second obtaining module configured to obtain a first image feature corresponding to a to-be-processed image by using a pre-trained self-supervised feature extraction network, the to-be-processed image being a panoramic depth image; the second obtaining module is further configured to obtain a second image feature corresponding to a fusion image by using the self-supervised feature extraction network, the fusion image being obtained by fusing the to-be-processed image and a confidence map corresponding to the to-be-processed image; a second determining module configured to input an initial depth map corresponding to the to-be-processed image, the first image feature and the second image feature into a pre-constructed attention module to obtain an attention depth map corresponding to the to-be-processed image; a depth map clustering module configured to cluster the attention depth map to obtain at least one region. The second optimization module is configured to obtain a target region depth map corresponding to each of the at least one region, and combine the target region depth maps corresponding to the at least one region to obtain a target depth map corresponding to the to-be-processed image pair.

12. An electronic device, comprising: The application also provides a computer program product, comprising a computer readable storage medium, and a computer program stored in the computer readable storage medium and capable of being executed on a processor. The application also provides a computer program product, comprising a computer readable storage medium, and a computer program stored in the computer readable storage medium and capable of being executed on a processor.

13. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and the computer program is executed on the processor to implement the image processing method according to any one of claims 1 to 9.

14. A microscope, characterized by The microscope is a full-aperture depth microscope, comprising an image acquisition device configured to acquire a full-aperture depth image, and the image processing device according to claim 10 or claim 11, configured to obtain a target depth map of the full-aperture depth image.

Images