Depth estimation model training method, depth map estimation method and related device

Abstract

The invention relates to an image processing technology, and provides a depth estimation model training method, a depth map estimation method and a related device, and the method comprises the steps: obtaining an original image shot for an overhead transmission line scene; generating a depth pseudo-label graph of the original image, wherein the depth value of the overhead transmission line region in the depth pseudo-label graph is related to the reference depth value of a preset reference region in the original image and the distance between the overhead transmission line region and a reference position in the preset reference region; keeping the change trend of the depth value of the depth pseudo-label graph unchanged, and performing degradation processing on the depth pseudo-label graph to obtain a depth prompt graph; and taking the depth prompt graph as a training sample, taking the depth pseudo-label graph as a label of the training sample, and training a pre-constructed depth estimation model to obtain a trained depth estimation model. The depth estimation accuracy of the overhead transmission line can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to a depth estimation model training method, a depth map estimation method and related devices. BACKGROUND

[0002] In the power transmission line construction scene, accidents caused by the collision of large construction machinery and power transmission lines occur from time to time. In order to overcome the problem that the reliability of artificial judgment of the threat of large construction machinery to power transmission lines is insufficient and misjudgment occurs from time to time, the relative depth relationship and the relative position relationship between the construction machinery and the power transmission line are usually analyzed to quantify the risk level of the power transmission line damage risk.

[0003] Due to the inherent characteristics of the conductor in the image captured by the camera, such as small pixel ratio, slender, and far-end blur, the existing depth estimation method is difficult to correctly identify the power transmission line, difficult to clearly define the power transmission line range, and difficult to accurately predict the relative depth of the power transmission line, which ultimately affects the identification of the power transmission line damage risk. SUMMARY

[0004] The present application aims to provide a depth estimation model training method, a depth map estimation method and related devices, which can improve the accuracy of overhead power transmission line depth estimation.

[0005] Embodiments of the present application can be implemented as follows: In a first aspect, the present application provides a depth estimation model training method, comprising: obtaining an original image captured for an overhead power transmission line scene; generating a depth pseudo-label map of the original image, wherein the depth value of the overhead power transmission line region in the depth pseudo-label map is related to the reference depth value of a preset reference region in the original image and the distance of the overhead power transmission line region relative to the reference position in the preset reference region; keeping the change trend of the depth value of the depth pseudo-label map unchanged, performing degradation processing on the depth pseudo-label map to obtain a depth hint map; using the depth hint map as a training sample and using the depth pseudo-label map as a label, training a pre-constructed depth estimation model to obtain a trained depth estimation model.

[0006] In an optional implementation, the reference position is determined by two reference points located on the same horizontal line, and the two reference points are respectively the pixel points of the end points of the two power transmission lines in the preset reference region. The step of generating the depth pseudo-label map of the original image comprises: For each pixel row in a preset marking region of the original image, determine first pixel points belonging to a first power transmission line and second pixel points belonging to a second power transmission line in the each pixel row, a geometric shape of the preset marking region reflects distance changes of different positions of the overhead power transmission line region relative to a shooting device, the first power transmission line and the second power transmission line are two different power transmission lines in the original image; According to the reference depth value, a distance between the first pixel point and the second pixel point of the each pixel row, and a distance between the two reference points, determine a marking depth value of the each pixel row; Using the marking depth value of the each pixel row, update a depth value of a pixel point belonging to the overhead power transmission line region in a corresponding pixel row in a relative depth map of the original image, to obtain a marked depth map; Randomly transform the marked depth map to obtain the depth pseudo-label map.

[0007] In an optional embodiment, the original image is multiple, each of the original images has a pseudo-label, the pseudo-label includes a relative depth map of the original image, a first transformed image obtained by randomly transforming the relative depth map of the original image, a marked depth map of the original image, a second transformed image obtained by randomly transforming the marked depth map of the original image, the marked depth map is obtained by updating a depth value of an overhead power transmission line region of the relative depth map; The step of training a pre-constructed depth estimation model using the depth hint map as a training sample and the depth pseudo-label map as a label includes: Using the original image as a first training sample, using the relative depth map of the original image and the marked depth map of the original image in the pseudo-label of the original image as a first label; Using a depth hint map of each pseudo-label as a second training sample, using the first transformed image and the second transformed image in the pseudo-label corresponding to the depth hint map as a second label, the depth hint map is obtained by degrading the pseudo-label map while keeping a change trend of the depth value of the pseudo-label unchanged; Using a combination of all the first training samples and first labels and all the second training samples and second labels as training samples and labels; Inputting the training samples and labels into a pre-constructed depth estimation model to train the depth estimation model.

[0008] In an optional implementation, the depth estimation model comprises a first encoder, a second encoder, a first decoder and a second decoder; the step of inputting the training sample and the label into the pre-constructed depth estimation model for training to obtain the trained depth estimation model comprises: If the training sample is an original image, the training sample is input into the first encoder for image feature extraction to obtain first image features; If the training sample is a depth hint map, the training sample is input into the second encoder for image feature extraction to obtain second image features; The first image features and the second image features are fused to obtain fused features; The first image features are input into the first decoder to output a first predicted depth map; The fused features are input into the second decoder to output a second predicted depth map; According to the first predicted depth map, the first label, the second predicted depth map and the second label, a total loss is calculated, and parameters of the depth estimation model are updated based on the total loss until a preset termination condition is met, to obtain the trained depth estimation model.

[0009] In an optional implementation, the original image comprises an overhead power transmission line region and a background region other than the overhead power transmission line region, and the step of calculating a total loss according to the first predicted depth map, the first label, the second predicted depth map and the second label comprises: According to the mask of the overhead power transmission line region, the mask of the background region, the first predicted depth map and the first label, a basic loss is calculated; According to the mask of the overhead power transmission line region, the mask of the background region, the second predicted depth map and the second label, a hint loss is calculated; According to the basic loss and the hint loss, the total loss is calculated.

[0010] In an optional implementation, the method further comprises: An intermediate image and a camera depth map obtained by performing depth extraction on the intermediate image by using a depth camera are acquired; The intermediate image and the camera depth map are input into the trained depth estimation model to obtain a predicted depth map; The intermediate image is taken as a training sample, the predicted depth map is taken as a label, and a new depth estimation model is trained to obtain a trained new depth estimation model.

[0011] In a second aspect, the application provides a depth map estimation method, which comprises: obtaining an image to be processed; if the image to be processed comprises a target image and a camera depth map, inputting the target image and the camera depth map into a trained depth estimation model to obtain a depth map of the target image, the trained depth estimation model being trained by any of the above-mentioned methods of the preceding embodiments; if the image to be processed comprises a target image, inputting the target image into a trained new depth estimation model to obtain a depth map of the target image, the new depth estimation model being trained by the method of the above-mentioned preceding embodiments.

[0012] In a third aspect, the present application provides a depth estimation model training device, the device comprising: an obtaining module configured to obtain an original image captured in an overhead power transmission line scene; a generating module configured to generate a depth pseudo-label map of the original image, wherein a depth value of an overhead power transmission line region in the depth pseudo-label map is related to a reference depth value of a preset reference region in the original image and a distance of the overhead power transmission line region relative to a reference position in the preset reference region; The generating module is further configured to keep the change trend of the depth value of the depth pseudo-label map unchanged, perform degradation processing on the depth pseudo-label map to obtain a depth hint map. a training module configured to take the depth hint map as a training sample, take the depth pseudo-label map as a label, and train a pre-constructed depth estimation model to obtain a trained depth estimation model.

[0013] In a fourth aspect, the present application provides a depth map estimation device, the device comprising an estimation module, the estimation module being configured to: obtain an image to be processed; if the image to be processed comprises a target image and a camera depth map, inputting the target image and the camera depth map into a trained depth estimation model to obtain a depth map of the target image, the trained depth estimation model being trained by any of the above-mentioned methods of the preceding embodiments; if the image to be processed comprises a target image, inputting the target image into a trained new depth estimation model to obtain a depth map of the target image, the new depth estimation model being trained by the method of the above-mentioned preceding embodiments.

[0014] In a fifth aspect, the present application provides an electronic device comprising a processor and a memory, the memory being configured to store a program, and the processor being configured to implement the depth estimation model training method of the first aspect or the depth map estimation method of the second aspect when executing the program.

[0015] Compared with the prior art, the present application has the following beneficial effects: The present application generates a depth pseudo-label map of the original image for the overhead power line scene, degrades the depth pseudo-label map while keeping the change trend of the depth value unchanged, obtains a depth hint map, takes the depth hint map as a training sample, takes the depth pseudo-label map as a label of the training sample, trains a pre-constructed depth estimation model, and obtains a trained depth estimation model. Since the depth value of the overhead power line region in the depth pseudo-label map is related to the reference depth value of the preset reference region in the original image and the distance of the overhead power line region relative to the reference position in the preset reference region, the change of the depth value of the pixel point of the overhead power line region in the depth pseudo-label map can reflect the distance change of the overhead power line relative to the shooting device, so that the pixel value of the pixel point of the overhead power line region is more accurate, and the finally obtained trained depth estimation model is more accurate in predicting the depth value. BRIEF DESCRIPTION OF DRAWINGS

[0016] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments. It should be understood that the following drawings only show some embodiments of the present application, and therefore should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can also be obtained without creative labor on the basis of these drawings.

[0017] Figure 1 An example of the flow of the depth estimation model training method provided for the present embodiment Figure 1 .

[0018] Figure 2 An example of the preset annotation region provided for the present embodiment.

[0019] Figure 3 An example of the reference point and the depth value reference point provided for the present embodiment.

[0020] Figure 4 An example of the random transformation provided for the present embodiment.

[0021] Figure 5 An example of the structure of the depth estimation model provided for the present embodiment.

[0022] Figure 6 An example of the flow of the depth estimation model training method provided for the present embodiment Figure 2 .

[0023] Figure 7 An example of the flow of the depth estimation model training method provided for the present embodiment

[0024] Figure 8A block diagram of an example of the depth estimation model training apparatus provided in the embodiment.

[0025] Figure 9 A block diagram of an example of the depth map estimation apparatus provided in the embodiment.

[0026] Figure 10 A block diagram of an example of the electronic device provided in the embodiment.

[0027] Icon: 10-electronic device; 11-processor; 12-memory; 13-bus; 100-depth estimation model training apparatus; 110-acquisition module; 120-generation module; 130-training module; 200-depth map estimation apparatus; 210-estimation module. DETAILED DESCRIPTION

[0028] In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings can be arranged and designed in various different configurations.

[0029] Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the claimed present application, but only represents selected embodiments of the present application. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in the present application without creative labor are within the scope of protection of the present application.

[0030] It should be noted that: similar reference numerals and letters represent similar items in the following drawings, and therefore, once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings.

[0031] In the description of the present application, it should be noted that if the terms "upper", "lower", "inner", "outer" and the like indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, or the orientation or positional relationship in which the product of the present application is usually placed, and are only for the convenience of describing the present application and simplifying the description, and therefore cannot be understood as indicating or implying that the indicated device or element must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as limiting the present application.

[0032] In addition, if the terms "first", "second" and the like appear, they are only used to distinguish the description, and cannot be understood as indicating or implying relative importance.

[0033] It should be noted that the features in the embodiments of the present application can be combined with each other without conflict.

[0034] To obtain the depth of images taken for overhead power line scenes, two techniques are generally used: (1) directly using existing large-scale pre-trained deep learning networks to estimate the depth of images; (2) using depth cameras to estimate the depth of images.

[0035] For the former technique, commonly used deep learning network-based methods generally include the following: a) Depth Anything network-based method This method trains a depth estimation network using a large-scale image dataset, combining labeled images and larger-scale unlabeled images to train the model, where the prediction results of the teacher model are used as pseudo-labels for training. The disadvantage is that the precision is insufficient and cannot identify small power lines; the network is a relative depth estimation model, and the predicted relative depth does not guarantee a constant ratio to the true depth.

[0036] b) Depth Anything V2 network-based method This method is an improvement on the Depth Anything method, which only uses simulated image datasets to train the model to further improve the precision and resolution of the depth estimation results. The disadvantage is that it still cannot identify small power lines and often misidentifies power lines as background areas; the network is a relative depth estimation model.

[0037] c) Depth Crafter network-based method This method uses a variational autoencoder and a diffusion model to train a video depth prediction network for video datasets. The disadvantage is that it is mainly for video datasets, has a large number of parameters, requires high computing power, and has low real-time performance; the network is a relative depth estimation model, and the sky area in the output depth map has large-area artifacts.

[0038] For the latter technique, the disadvantage is that the depth map obtained using a depth camera has insufficient precision and cannot identify small power lines.

[0039] In view of this, the present embodiment provides a depth estimation model training method, a depth map estimation method and related devices, the core improvement of which is to introduce a depth pseudo-label map and a depth hint map, the depth value of the overhead transmission line area in the depth pseudo-label map is related to the reference depth value of the preset reference area in the original image and the distance of the overhead transmission line area relative to the reference position in the preset reference area, the depth hint map is used as a training sample, and the depth pseudo-label map is used as a label to train a pre-constructed depth estimation model, and the trained depth estimation model can accurately estimate the depth of the overhead transmission line, which will be described in detail below.

[0040] This example first introduces the training method of the depth estimation model. Figure 1 , Figure 1 This is a flow chart of an example of a depth estimation model training method provided in this embodiment, which includes the following steps: Step S101: Acquire an original image captured for an overhead transmission line scene.

[0041] In this embodiment, the original image may be an image of a real scene of overhead power lines captured by a binocular depth camera, and has clear texture and structure information.

[0042] Step S102, generate a depth pseudo-label map of the original image, wherein the depth value of the overhead transmission line area in the depth pseudo-label map is related to the reference depth value of the preset reference area in the original image and the distance of the overhead transmission line area relative to the reference position in the preset reference area.

[0043] In this embodiment, the preset reference region, serving as a reference for the overhead power line region, is a region in the original image that has stable structural features and maintains a relatively stable position relative to the overhead power line region. The reference depth value represents the depth of the preset reference region and can be the relative depth value of any pixel within the preset reference region or the average of the relative depth values ​​of all pixels within the preset reference region. The preset reference region can be the tower region or a portion of the tower region in the original image.

[0044] In this embodiment, the reference position may be the position of a line segment defined by two points in a predetermined reference area. Based on the reference position, different locations in the overhead transmission line area may have different depth values ​​if their distances from the reference position are different.

[0045] In this embodiment, the depth pseudo-label map can be a depth map obtained by updating the depth values of the overhead power transmission line region in the relative depth map of the original image. The relative depth map can be obtained by using a pre-trained relative depth estimation network, including but not limited to Depth Anything or Depth Anything V2, etc. The depth values of the overhead power transmission line region in the depth pseudo-label map are related to the reference depth values of the pre-set reference region in the original image and the distance of the overhead power transmission line region relative to the reference position, which means that in the overhead power transmission line region in the depth pseudo-label map, the depth values of the pixel points will change as the distance between the pixel points and the reference position changes.

[0046] In step S103, the depth pseudo-label map is degraded to obtain a depth hint map, while keeping the change trend of the depth values of the depth pseudo-label map unchanged.

[0047] In this embodiment, the purpose of degrading the depth pseudo-label map is to simulate the relatively rough depth map obtained by the depth camera, and the degradation process includes but is not limited to the following operations: adding Gaussian noise or salt and pepper noise; down-sampling and up-sampling the resolution of the depth map; introducing random occlusion or holes in part of the region; applying blur or smoothing processing to the edge region, etc. The depth hint map no longer retains the fine structure of the original image, but the overall depth trend remains consistent with the depth pseudo-label map.

[0048] In step S104, the depth pseudo-label map is used as a label to train the pre-constructed depth estimation model, and the depth hint map is used as a training sample to obtain a trained depth estimation model.

[0049] In this embodiment, the depth hint map is used to guide the depth estimation model to focus on the approximate depth structure in the overhead power transmission line scene during the model training stage. By simulating the depth map characteristics output by the depth camera through the depth hint map, the trained model can effectively utilize the rough depth information provided by the actual depth camera in the inference stage, thereby improving the depth recovery capability of the model for the overhead power transmission line.

[0050] The above method provided in this embodiment introduces the depth pseudo-label map and the depth hint map, uses the depth hint map as a training sample, uses the depth pseudo-label map as a label, and trains the pre-constructed depth estimation model. Since the depth values of the overhead power transmission line region in the depth pseudo-label map are related to the reference depth values of the pre-set reference region in the original image and the distance of the overhead power transmission line region relative to the reference position in the pre-set reference region, the change of the depth values of the pixel points in the overhead power transmission line region in the depth pseudo-label map can reflect the distance change of the overhead power transmission line relative to the shooting device, so that the pixel values of the pixel points in the overhead power transmission line region are more accurate, and the finally obtained trained depth estimation model is more accurate in predicting the depth values.

[0051] In an optional implementation, when the extending direction of the overhead power transmission line in the overhead power transmission line scene is consistent with the direction of the light source light of the photographing device, for two points respectively located on two overhead power transmission lines and in the same horizontal plane, the closer the two points are to the photographing device, the greater the distance between the two points in the original image, otherwise, the smaller the distance, and the embodiment provides an implementation of generating a depth pseudo-label map of the original image for the scene: First, for each pixel row in a preset annotation region of the original image, determine the first pixel point belonging to the first power transmission line and the second pixel point belonging to the second power transmission line in each pixel row, the geometric shape of the preset annotation region reflects the distance change of different positions of the overhead power transmission line region relative to the photographing device, and the first power transmission line and the second power transmission line are two different power transmission lines in the original image. In the embodiment, based on the two points in the same horizontal plane, the closer the two points are to the photographing device, the greater the distance between the two points in the original image, otherwise, the smaller the distance, the embodiment pre-annotates a preset annotation region on the original image, please refer to Figure 2 , Figure 2 The example diagram of the preset annotation region provided for the embodiment is shown in Figure 2 , the geometric shape of the preset annotation region is a trapezoid, which covers the overhead power transmission line region, the upper base of the trapezoid (i.e. the shorter side of the trapezoid) represents the position of the far-end power transmission line, which is far away from the photographing device, and the lower base of the trapezoid (i.e. the longer side of the trapezoid) represents the position of the near-end power transmission line, which is close to the photographing device.

[0052] It should be noted that Figure 2 the red trapezoid in Figure 2 is only an example of the preset annotation region, in fact, the preset annotation region can cover the region of all overhead power transmission lines, or cover two or more of the multiple overhead power transmission lines, and the preset annotation region can also be the green trapezoid in

[0053] Secondly, determine the annotation depth value of each pixel row according to the reference depth value, the distance between the first pixel point and the second pixel point of each pixel row, and the distance between the two reference points. In the embodiment, in order to ensure the accuracy of the reference, each pixel row is in the same horizontal line, the distance between the two reference points is the horizontal distance between the two reference points, and if the preset reference region is the tower region, the reference points are the anchor points where the overhead power transmission line is anchored to the iron tower, and the horizontal distance between the reference points is the horizontal distance in the camera imaging plane when the two overhead power transmission lines are finally connected to the far-end iron tower. Please refer to Figure 3 , Figure 3An example diagram of reference points and depth value reference points provided for the embodiment, Figure 3 The DA model estimation result in the DA (Depth Anything) model output is a relative depth map of the original image, reference point 1 and reference point 2 are anchor points at which the two power transmission lines are finally connected to the remote tower, and depth value reference point a is a point on the remote tower, the depth value of which is taken as a reference depth value and represents the depth of the remote tower.

[0054] In the embodiment, the first power transmission line and the second power transmission line are two different power transmission lines in the original image, the first pixel points in all the pixel rows belong to the first power transmission line, and the second pixel points belong to the second power transmission line, so that the distance between the first pixel points and the second pixel points in different pixel rows changes with the distance from the shooting device, and finally the change trend of the depth values of the pixel points in the overhead power transmission line region can be reflected.

[0055] As an implementation manner, the preset annotation region is taken as an example. Figure 2 Taking the red trapezoid in the embodiment as an example, the first power transmission line and the second power transmission line can be located on the left and right boundary power transmission lines of the overhead power transmission line region. Specifically, in the trapezoidal region, the actual horizontal length of each row is obtained by calculating the left and right boundaries (i.e., the first pixel point and the second pixel point are located on the left and right boundaries of each row) of each row in a row-by-row scanning manner. Based on the proportion between the horizontal length and the reference length (i.e., the horizontal distance between the reference points), the depth value of each pixel point is calculated according to the rule that "the depth is inversely proportional to the horizontal length". That is, the closer to the bottom edge in the trapezoidal region, the greater the horizontal length, and the smaller the corresponding depth value; the closer to the top edge, the greater the depth value.

[0056] In the embodiment, the annotation depth values of the overhead power transmission line region in the same pixel row are the same, and the calculation formula of the annotation depth value for any pixel row can be: annotation depth value = reference depth value x (reference length / horizontal length), wherein the horizontal length is the distance between the first pixel point and the second pixel point of the pixel row, and the reference length is the distance between the two reference points.

[0057] Thirdly, the depth values of the pixel points in the corresponding pixel row in the relative depth map of the original image and belonging to the overhead power transmission line region are updated by using the annotation depth value of each pixel row, to obtain an annotation depth map. In the embodiment, since the overhead power transmission line in the original image is regarded as a background when a relative depth map is generated by a common relative depth estimation method, such as the Depth Anything V2 model, in order to facilitate updating the depth values of the pixel points in each pixel row in the relative depth map of the original image and belonging to the overhead power transmission line region, the embodiment provides an implementation manner of a power transmission line mask map: (1) Generate a transmission line mask map; The process of generating a transmission line mask map can be: a) preprocessing the original image, converting it into a grayscale image, and normalizing it to a floating point range of 0 to 1; b) using the Sobel operator to perform edge detection on the grayscale image to obtain an edge intensity map (i.e., a Sobel intensity map); c) setting a binarization threshold (such as 0.01) to binarize the Sobel intensity map to obtain an initial Sobel edge mask map, where the area with edge intensity greater than the binarization threshold in the initial Sobel edge mask map is set as the foreground (e.g., the pixel value is set to 255), and the rest is set as the background (e.g., the pixel value is set to 0); d) extracting high confidence areas with depth values ​​greater than 0.8 from the relative depth map output by the Depth Anything V2 model to obtain areas that the Depth Anything V2 model considers to be the background; e) based on the characteristic that the Depth Anything V2 model will identify the transmission line as the background, performing a bitwise AND operation on the initial Sobel edge mask map and the high confidence area in the relative depth map, retaining pixels that are foregrounds in both, and obtaining a transmission line area mask map.

[0058] (2) Update the depth value of each pixel row in the area where the relative depth map of the original image overlaps with the power line mask map to the labeled depth value of the corresponding pixel row.

[0059] Finally, the annotated depth map is randomly transformed to obtain a depth pseudo-label map.

[0060] In this embodiment, random transformation can be to perturb the depth value in the annotated depth map of a single channel. Random transformation can include operations such as Gamma transformation, exponential mapping, linear scaling or offset, etc. Its purpose is to enhance the diversity of depth value distribution and improve the adaptability of the model to different depth distributions. As a specific implementation method, a monotonic function can be used to randomly set the parameters of the monotonic function to achieve the purpose of random transformation. Monotonic functions include, but are not limited to, power functions, exponential functions, logarithmic functions, linear functions, etc. Please refer to Figure 4 , Figure 4 This is an example diagram of random transformation provided in this embodiment. Figure 4 In the figure, the purple line is the power function Medium parameters When An example of a random transformation of , where the blue line is a logarithmic function Medium parameters When An example of a random transformation of , where the green line is an exponential function Medium parameters When An example of random transformation of .

[0061] In an optional embodiment, in order to reduce the workload of generating the labeled depth map, when training the depth estimation model, based on a small amount of labeled depth maps, the effective utilization of the depth estimation model for the change trend of the depth value can be realized, and at the same time, the normal relative depth map estimation function can be compatible. In this embodiment, only a small part of the original images in the multiple original images need to be generated with a labeled depth map, and then the labeled depth map is randomly transformed to obtain a depth pseudo-label map. For the remaining most original images, the relative depth map is directly randomly transformed to obtain a depth pseudo-label map. Then, under the premise of keeping the change trend of the depth value unchanged, the degradation processing is performed on all the depth pseudo-label maps to obtain a depth hint map. Therefore, for an original image, the pseudo-labels thereof include the relative depth map of the original image, the first transformed image obtained by randomly transforming the relative depth map of the original image, the labeled depth map of the original image, and the second transformed image obtained by randomly transforming the labeled depth map of the original image. Finally, the original image and the depth hint map are used as sample images to train the depth estimation model. A possible implementation manner is as follows: Firstly, the original image is taken as a first training sample, and the relative depth map of the original image and the labeled depth map of the original image in the pseudo-labels of the original image are taken as first labels. In this embodiment, there are multiple original images, each of which has a pseudo-label. The pseudo-label is obtained by randomly transforming the relative depth map of the original image or by randomly transforming the labeled depth map of the original image. The labeled depth map is obtained by updating the depth value of the overhead transmission line region of the relative depth map. That is, when the pseudo-label is obtained by randomly transforming the labeled depth map of the original image, the pseudo-label is the depth pseudo-label map of the original image.

[0062] Secondly, the depth hint map of each pseudo-label is taken as a second training sample, and the first transformed image and the second transformed image in the pseudo-label corresponding to the depth hint map are taken as second labels. The depth hint map is obtained by degrading the pseudo-label map while keeping the change trend of the depth value unchanged. Thirdly, the union of all the first training samples and the first labels and all the second training samples and the second labels is taken as training samples and labels. In this embodiment, the first labels and the second labels both include the depth pseudo-label map of the original image and the pseudo-label obtained by randomly transforming the relative depth map of the original image. Part of the labels in the label set of the first labels and the second labels can be used as a supervision signal in the overhead transmission line region, and part of the labels do not have a supervision effect in the overhead transmission line region, thereby forming a training data set suitable for a mixed supervision scene.

[0063] Finally, the training sample and the label are input into the pre-constructed depth estimation model for training, to obtain the trained depth estimation model.

[0064] In an optional embodiment, in order to make the constructed depth estimation model have both the basic ability of relative depth estimation and the fine depth prediction ability, the embodiment takes a VisionTransformer (ViT) based encoder as a backbone network for extracting multi-scale visual features of the original image, and a decoder based on a DPT (Dense Prediction Transformer) structure has a multi-scale fusion ability for restoring a high-resolution depth map. On this basis, a depth hint guiding branch network is added for fusing feature information of the original image and the depth hint map. The depth hint guiding branch network introduces a parallel branch in the encoding stage to extract depth structure features of the depth hint map and fuse the depth structure features with the original image at the middle layer. The depth estimation model includes two outputs: the first output is a relative depth map based on the original image, which maintains the basic ability of relative depth estimation, and the second output is a fine depth prediction depth map, which fuses the depth structure features of the depth hint map and focuses on restoring detailed structures such as power transmission lines on the basis of the depth distribution of the depth hint map. Please refer to Figure 5 , Figure 5 FIG. 1 is a structural diagram of a depth estimation model provided by the embodiment, Figure 5 In the embodiment, the branch where the first encoder and the first decoder are located is a backbone network, and the first predicted depth map output by the backbone network is a relative depth map of the original image. The branch where the second encoder and the second decoder are located is a depth hint guiding branch network, and the second predicted depth map output by the depth hint guiding branch network is a fine depth prediction depth map, which is an absolute depth map. Figure 5 The residual convolution unit in the embodiment is used to fuse the first image features and the second image features output by the first encoder and the second encoder. Figure 5 In the embodiment, the first label is used as a label for supervised training of the backbone network, and the second label is used as a label for supervised training of the depth hint guiding branch network.

[0065] Based on Figure 5 One way of training the depth estimation model can be as follows: If the training sample is an original image, the training sample is input into the first encoder for image feature extraction to obtain first image features. If the training sample is a depth hint map, the training sample is input into the second encoder for image feature extraction to obtain second image features. The first image features and the second image features are fused to obtain fused features. input the first image feature into a first decoder to output a first predicted depth map; input the fusion feature into a second decoder to output a second predicted depth map; calculate a total loss according to the first predicted depth map, the first label, the second predicted depth map and the second label, and update parameters of the depth estimation model based on the total loss until a preset termination condition is met to obtain a trained depth estimation model.

[0066] In the embodiment, the preset termination condition can be a preset training number, a preset convergence condition that the total loss is less than a preset loss value, or a difference between the current total loss and a previous total loss is less than a preset difference value.

[0067] In an optional implementation, in order to make the overhead power line region in the label annotated depth map participate in loss calculation as an effective supervision region, the relative depth map in the label is ignored so as to not generate loss and only be used for auxiliary training of the remaining background structure, the embodiment provides an implementation of calculating the total loss: First, calculate a basic loss according to the mask of the overhead power line region, the mask of the background region, the first predicted depth map and the first label; Second, calculate a prompt loss according to the mask of the overhead power line region, the mask of the background region, the second predicted depth map and the second label; Finally, calculate the total loss according to the basic loss and the prompt loss.

[0068] In the embodiment, the total loss can be calculated by the following formula: wherein, , , is the prompt loss, is the basic loss, and are respectively a first weight parameter and a second weight parameter, is the mask of the overhead power line region, is the mask of the background region, , is a coordinate of a pixel in the image, is a number of pixels participating in loss calculation, is the second predicted depth map, is the first predicted depth map, is the second label, is the first label, if the first label or the second label is obtained according to random transformation of the annotated depth map, then is 1, if the first label or the second label is obtained by random transformation according to the relative depth map, then is 0.

[0069] It should be noted that the calculation order of the basic loss and the prompt loss is not limited, and the two can be performed in sequence or in parallel.

[0070] The prediction result of the trained depth estimation model obtained by the training method is compared with the original Depth Anything V2 relative depth estimation model and the Depth Crafter model with an input resolution of 1024. The power line area recognition rate on the simulation image as the test set is increased from 2.7% and 69.6% to 81.9%, respectively. The accuracy of the full image and the power line area is greatly improved compared with the existing depth estimation model, and the output result is an absolute depth estimation result, which improves the practicability of the model in real working scenarios.

[0071] In the inference stage, the trained depth estimation model obtained by the training method needs to input the target image and the corresponding camera depth map. However, in the application scenario without a depth camera, in order to be able to input only the target image in the inference stage to obtain the predicted depth map, the embodiment further provides another training method based on the trained depth estimation model. Please refer to Figure 6 , Figure 6 The flowchart of the depth estimation model training method provided by the embodiment Figure 2 includes the following steps: Step S201, obtaining an intermediate image and a camera depth map obtained by depth extraction of the intermediate image by a depth camera; In the embodiment, the intermediate image can be the original image described above, or other images similar to the scene of the original image. The camera depth map can be a depth map calculated by a binocular depth camera for the left eye image and the right eye image of the overhead power line scene. The depth value in the depth map is an absolute depth value.

[0072] Step S202, inputting the intermediate image and the camera depth map into the trained depth estimation model to obtain a predicted depth map; Step S203, taking the intermediate image as a training sample, taking the predicted depth map as a label, and training a new depth model to obtain a trained new depth estimation model.

[0073] In the embodiment, the trained new depth estimation model only needs to input the target image in the inference stage, without inputting the corresponding camera depth map, so that the trained new depth estimation model obtained by the training manner can be used for a mobile platform or a historical image library without a depth camera to complete the depth estimation task in the power transmission line scene, thereby improving the adaptability and generalization ability of the new depth estimation model in the actual deployment scene.

[0074] It should be further noted that the execution subject of the first training manner, i.e., the training manner of steps S101-S104 and its sub-steps, and the second training manner, i.e., the training manner of steps S201-S203, can be the same electronic device or different electronic devices, which is not limited in the embodiment.

[0075] The above introduces two training manners, the trained models obtained by the two training manners can be independently used according to the respective suitable scenes, or the two models can be integrated together to automatically select a suitable model for depth map estimation according to actual needs. For this purpose, the embodiment provides a specific implementation manner, please refer to Figure 7 , Figure 7 The embodiment provides a flowchart of a depth map estimation method, which comprises the following steps: Step S301, obtaining an image to be processed; Step S302, if the image to be processed comprises a target image and a camera depth map, inputting the target image and the camera depth map into a trained depth estimation model to obtain a depth map of the target image, the trained depth estimation model being trained by the first training manner in the embodiment; Step S303, if the image to be processed comprises a target image, inputting the target image into a trained new depth estimation model to obtain a depth map of the target image, the new depth estimation model being trained by the second training manner in the embodiment.

[0076] In order to execute the corresponding steps in the above embodiment and each possible implementation manner, an implementation manner of a depth estimation model training device 100 is given as follows. Please refer to Figure 8 , Figure 8 A block schematic diagram of the depth estimation model training device provided in the embodiment is shown, and it should be noted that the depth estimation model training device 100 provided in the embodiment has the same basic principle and technical effects as the corresponding above-mentioned embodiment, and part of the embodiment is not mentioned for brief description.

[0077] The depth estimation model training device 100 comprises an acquisition module 110, a generation module 120 and a training module 130.

[0078] The acquisition module 110 is configured to acquire an original image captured for an overhead power transmission line scene. The generation module 120 is configured to generate a depth pseudo-label map of the original image, wherein a depth value of an overhead power transmission line region in the depth pseudo-label map is related to a reference depth value of a preset reference region in the original image and a distance of the overhead power transmission line region relative to a reference position in the preset reference region. The generation module 120 is further configured to keep a variation trend of the depth value of the depth pseudo-label map unchanged, perform degradation processing on the depth pseudo-label map, and obtain a depth hint map. The training module 130 is configured to take the depth hint map as a training sample and take the depth pseudo-label map as a label, train a pre-constructed depth estimation model, and obtain a trained depth estimation model.

[0079] In an optional implementation, the reference position is determined by two reference points located on the same horizontal line, and the two reference points are pixel points of end points of two power transmission lines in the preset reference region. For each pixel row in the preset annotation region of the original image, the first pixel point belonging to the first power transmission line and the second pixel point belonging to the second power transmission line in each pixel row are determined, the geometric shape of the preset annotation region reflects distance variation of different positions of the overhead power transmission line region relative to the shooting device, and the first power transmission line and the second power transmission line are two different power transmission lines in the original image. According to the reference depth value, the distance between the first pixel point and the second pixel point of each pixel row, and the distance between the two reference points, the annotation depth value of each pixel row is determined. The depth value of the pixel point belonging to the overhead power transmission line region in the corresponding pixel row in the relative depth map of the original image is updated by using the annotation depth value of each pixel row, and an annotation depth map is obtained. The annotation depth map is randomly transformed to obtain a depth pseudo-label map.

[0080] In an optional implementation, the original image is multiple, and each original image has a pseudo-label. The pseudo-label includes a relative depth map of the original image, a first transformed image obtained by randomly transforming the relative depth map of the original image, an annotation depth map of the original image, and a second transformed image obtained by randomly transforming the annotation depth map of the original image. The annotation depth map is obtained by updating the depth value of the overhead power transmission line region in the relative depth map. The training module 130 is configured to: take the original image as a first training sample, and take the relative depth map of the original image and the annotation depth map of the original image in the pseudo-label of the original image as a first label. the depth hint map of each pseudo label is taken as a second training sample, and the first transformed image and the second transformed image in the pseudo label corresponding to the depth hint map are taken as a second label, the depth hint map being obtained by performing degradation processing on the pseudo label image while keeping the change trend of the depth value of the pseudo label unchanged; a union of all the first training samples and the first labels and all the second training samples and the second labels is taken as training samples and labels; the training samples and the labels are input into a pre-constructed depth estimation model for training, to obtain a trained depth estimation model.

[0081] In an optional implementation, the depth estimation model includes a first encoder, a second encoder, a first decoder, and a second decoder; when training the training samples and the labels into the pre-constructed depth estimation model to obtain the trained depth estimation model, the training module 130 is specifically configured to: if the training sample is an original image, the training sample is input into the first encoder for image feature extraction, to obtain first image features; if the training sample is a depth hint map, the training sample is input into the second encoder for image feature extraction, to obtain second image features; the first image features and the second image features are fused, to obtain fused features; the first image features are input into the first decoder, to output first predicted depth maps; the fused features are input into the second decoder, to output second predicted depth maps; a total loss is calculated according to the first predicted depth maps, the first labels, the second predicted depth maps, and the second labels, and parameters of the depth estimation model are updated based on the total loss until a preset termination condition is met, to obtain the trained depth estimation model.

[0082] In an optional implementation, the original image includes an overhead power transmission line region and a background region other than the overhead power transmission line region, and when calculating the total loss according to the first predicted depth maps, the first labels, the second predicted depth maps, and the second labels, the training module 130 is specifically configured to: a basic loss is calculated according to a mask of the overhead power transmission line region, a mask of the background region, the first predicted depth maps, and the first labels; a hint loss is calculated according to the mask of the overhead power transmission line region, the mask of the background region, the second predicted depth maps, and the second labels; the total loss is calculated according to the basic loss and the hint loss.

[0083] In an optional implementation, the training module 130 is specifically configured to: an intermediate image and a camera depth map obtained by performing depth extraction on the intermediate image using a depth camera are acquired; input the intermediate image and the camera depth map into the trained depth estimation model to obtain a predicted depth map; The intermediate image is taken as a training sample, and the predicted depth map is taken as a label to train the pre-constructed new depth model to obtain the trained new depth estimation model.

[0084] To perform the corresponding steps in the above-mentioned embodiments and various possible implementations, an implementation of a depth map estimation device 200 is given below. Please refer to Figure 9 , Figure 9 A block schematic diagram of the depth map estimation device 200 provided in the present embodiment is given, and it should be noted that the depth map estimation device 200 provided in the present application has the same basic principle and technical effects as the corresponding above-mentioned embodiments, and for brief description, some parts of the present embodiment are not mentioned.

[0085] The depth map estimation device 200 comprises an estimation module 210.

[0086] The estimation module 210 is configured to: obtain a to-be-processed image; if the to-be-processed image comprises a target image and a camera depth map, input the target image and the camera depth map into a trained depth estimation model to obtain a depth map of the target image, the trained depth estimation model being trained by using the first training manner in the present embodiment; if the to-be-processed image comprises a target image, input the target image into a trained new depth estimation model to obtain a depth map of the target image, the new depth estimation model being trained by using the second training manner in the present embodiment.

[0087] The present embodiment further provides a block schematic diagram of an electronic device 10, and the electronic device 10 implements the depth estimation model training method or the depth map estimation method of the above-mentioned embodiments. Please refer to Figure 10 , Figure 10 The block schematic diagram of the electronic device 10 provided in the present embodiment comprises a processor 11, a memory 12 and a bus 13, and the processor 11 and the memory 12 are connected through the bus 13.

[0088] The processor 11 can be an integrated circuit chip with signal processing capability. In the implementation process, each step of the depth estimation model training method or the depth map estimation method of the above embodiments can be completed by the integrated logic circuit of hardware or the instruction in the form of software in the processor 11. The processor 11 described above can be a general-purpose processor, including a CPU (Central Processing Unit), an NP (Network Processor), and the like; and can also be a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Logic Gate Array) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component.

[0089] The memory 12 is used to store a program for implementing the depth estimation model training method or the depth map estimation method, which can be a software function module stored in the memory 12 in the form of software or firmware or solidified in the OS (Operating System) of the electronic device 10.

[0090] After receiving the execution instruction, the processor 11 executes the program to implement the depth estimation model training method or the depth map estimation method of the above embodiments.

[0091] The embodiment provides a computer storage medium, which stores a computer program, and the computer program is executed by a processor to implement the depth estimation model training method or the depth map estimation method as described in the above embodiments.

[0092] In summary, the embodiment of the present application provides a depth estimation model training method, a depth map estimation method and related devices, the method comprising: obtaining an original image captured for an overhead power transmission line scene; generating a depth pseudo-label map of the original image, the depth value of the overhead power transmission line region in the depth pseudo-label map being related to the reference depth value of a preset reference region in the original image and the distance of the overhead power transmission line region relative to the reference position in the preset reference region; keeping the change trend of the depth value of the depth pseudo-label map unchanged, degrading the depth pseudo-label map to obtain a depth hint map; taking the depth hint map as a training sample and taking the depth pseudo-label map as the label of the training sample, training a pre-constructed depth estimation model to obtain a trained depth estimation model. Compared with the prior art, the embodiment has at least the following advantages: (1) the depth pseudo-label map and the depth hint map are introduced, the depth hint map is taken as a training sample, the depth pseudo-label map is taken as a label, and a pre-constructed depth estimation model is trained. Since the depth value of the overhead power transmission line region in the depth pseudo-label map is related to the reference depth value of the preset reference region in the original image and the distance of the overhead power transmission line region relative to the reference position in the preset reference region, the change of the depth value of the pixel point of the overhead power transmission line region in the depth pseudo-label map can reflect the distance change of the overhead power transmission line relative to the shooting device, the overhead power transmission line region is avoided to be ignored as background, the pixel value of the pixel point of the overhead power transmission line region is more accurate, and finally the trained depth estimation model is more accurate in predicting the depth value; (2) a double branch including a trunk network and a branch, which is a depth hint guide branch network, is constructed. During training, the trunk network maintains the prediction ability of the relative depth map and has the ability to prevent forgetting, and also has the prediction ability of fine depth; (3) the camera imaging principle and structure region recognition are used to reasonably label the depth value of the overhead power transmission line, solving the problem of difficult depth value estimation of the power transmission line; (4) for application scenarios without a depth camera, the depth estimation task in the power transmission line scene can also be completed, thereby improving the adaptability and generalization ability in actual deployment scenarios.

[0093] The above merely describes various embodiments of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of changes or replacements within the technical range disclosed by the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A depth estimation model training method, characterized in that: The method comprises: Obtaining original images shot for overhead power line scenes; generating a depth pseudo-label map of the original image, wherein a depth value of an overhead power line region in the depth pseudo-label map is correlated with a reference depth value of a preset reference region in the original image and a distance of the overhead power line region relative to a reference position in the preset reference region; Maintaining a change trend of the depth value of the depth pseudo-label image unchanged, degrading the depth pseudo-label image to obtain a depth hint image; The depth hint map is used as a training sample, and the depth pseudo-label map is used as a label to train a pre-built depth estimation model to obtain a trained depth estimation model.

2. The method according to claim 1, characterized in that The reference position is determined by two reference points located on the same horizontal line, and the two reference points are respectively pixel points of the endpoints of the two power transmission lines in the preset reference area; the step of generating the depth pseudo label map of the original image includes: For each pixel row in a preset annotated area of ​​the original image, determining a first pixel point belonging to a first power line and a second pixel point belonging to a second power line in each pixel row, wherein the geometric shape of the preset annotated area reflects the distance variation of different positions of the overhead power line area relative to the camera, and the first power line and the second power line are two different power lines in the original image; Determining a marked depth value of each pixel row according to the reference depth value, a distance between a first pixel point and a second pixel point of each pixel row, and a distance between the two reference points; Using the annotated depth value of each pixel row, the depth values ​​of the pixels in the corresponding pixel row in the relative depth map of the original image and belonging to the overhead transmission line area are updated to obtain an annotated depth map; The annotated depth map is randomly transformed to obtain the depth pseudo-label map.

3. The method according to claim 1, characterized in that There are multiple original images, each of which has a pseudo label, and the pseudo label includes a relative depth map of the original image, a first transformed image obtained by randomly transforming the relative depth map of the original image, an annotated depth map of the original image, and a second transformed image obtained by randomly transforming the annotated depth map of the original image, wherein the annotated depth map is obtained by updating the depth value of the overhead transmission line area in the relative depth map; The step of using the depth hint map as a training sample and the depth pseudo label map as a label to train a pre-built depth estimation model to obtain a trained depth estimation model includes: Using the original image as a first training sample, and using the relative depth map of the original image and the annotated depth map of the original image in the pseudo-label of the original image as a first label; Using the depth cue map of each pseudo-label as a second training sample, and using the first transformed image and the second transformed image in the pseudo-label corresponding to the depth cue map as a second label, wherein the depth cue map is obtained by maintaining a change trend of the depth value of the pseudo-label unchanged and degrading the pseudo-label map; The collection of all the first training samples and first labels and all the second training samples and second labels is used as a training sample and label; The training samples and labels are input into a pre-built depth estimation model for training to obtain a trained depth estimation model.

4. The method according to claim 3, characterized in that The depth estimation model includes a first encoder, a second encoder, a first decoder, and a second decoder; and the step of inputting the training samples and labels into the pre-built depth estimation model for training to obtain the trained depth estimation model includes: If the training sample is an original image, inputting the training sample into the first encoder to extract image features to obtain first image features; If the training sample is a depth cue image, inputting the training sample into the second encoder to extract image features to obtain second image features; fusing the first image feature and the second image feature to obtain a fused feature; Inputting the first image feature into the first decoder and outputting a first predicted depth map; Inputting the fused features into the second decoder and outputting a second predicted depth map; According to the first predicted depth map, the first label, the second predicted depth map and the second label, a total loss is calculated, and the parameters of the depth estimation model are updated based on the total loss until a preset termination condition is met, thereby obtaining a trained depth estimation model.

5. The method according to claim 4, characterized in that The original image includes an overhead transmission line area and a background area excluding the overhead transmission line area, and the step of calculating the total loss based on the first predicted depth map, the first label, the second predicted depth map, and the second label includes: Calculating a base loss based on the mask of the overhead transmission line area, the mask of the background area, the first predicted depth map, and the first label; Calculating a prompt loss based on the mask of the overhead transmission line area, the mask of the background area, the second predicted depth map, and the second label; The total loss is calculated based on the basic loss and the prompt loss.

6. The method according to claim 1, characterized in that The method further comprises: Acquire an intermediate image and a camera depth map obtained by performing depth extraction on the intermediate image using a depth camera; Inputting the intermediate image and the camera depth map into the trained depth estimation model to obtain a predicted depth map; The intermediate image is used as a training sample, and the predicted depth map is used as a label to train a pre-built new depth model to obtain a trained new depth estimation model.

7. A depth map estimation method, characterized in that: The method comprises: Get the image to be processed; If the image to be processed includes a target image and a camera depth map, inputting the target image and the camera depth map into a trained depth estimation model to obtain a depth map of the target image, wherein the trained depth estimation model is trained using any one of the methods of claims 1 to 5 above; If the image to be processed includes a target image, the target image is input into the trained new depth estimation model to obtain a depth map of the target image, and the new depth estimation model is trained using the method of claim 6 above.

8. A depth estimation model training device, characterized in that: The device comprises: An acquisition module, used for acquiring original images shot for overhead transmission line scenes; a generating module, configured to generate a depth pseudo-label map of the original image, wherein a depth value of an overhead power line region in the depth pseudo-label map is correlated with a reference depth value of a preset reference region in the original image and a distance of the overhead power line region relative to a reference position in the preset reference region; The generating module is further configured to maintain a change trend of the depth value of the depth pseudo-label image unchanged, and to degrade the depth pseudo-label image to obtain a depth hint image; A training module is used to train a pre-built depth estimation model using the depth hint map as a training sample and the depth pseudo-label map as a label to obtain a trained depth estimation model.

9. A depth map estimation device, characterized in that The apparatus comprises an estimation module, wherein the estimation module is configured to: Get the image to be processed; If the image to be processed includes a target image and a camera depth map, inputting the target image and the camera depth map into a trained depth estimation model to obtain a depth map of the target image, wherein the trained depth estimation model is trained using any one of the methods of claims 1 to 5 above; If the image to be processed includes a target image, the target image is input into the trained new depth estimation model to obtain a depth map of the target image, and the new depth estimation model is trained using the method of claim 6 above.

10. An electronic device, characterized in that: It includes a processor and a memory, the memory is used to store a program, and the processor is used to implement the depth estimation model training method according to any one of claims 1 to 6, or the depth map estimation method according to claim 7 when executing the program.

Images