Self-supervised depth estimation network training methods, systems, and storage media

Abstract

This invention discloses a self-supervised depth estimation network training method, system, and storage medium, including acquiring video frame images as training samples and dividing the training samples into target views and source views; training the depth estimation network by using the target views as input to the depth estimation network and the semantic segmentation network, and using the source views as input to the camera network, to obtain a trained depth estimation network for depth map estimation. The depth estimation network includes a first encoder and a first decoder, and the semantic segmentation network includes a second encoder and a second decoder. The total loss function used during training includes feature vector alignment loss, vertical loss, photometric loss, and semantic segmentation loss. The feature vector alignment loss is used to align the feature maps output by the two encoders, and the vertical loss is used to ensure that the depth map output by the first decoder satisfies vertical consistency based on the semantic segmentation map output by the second decoder.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, specifically to a self-supervised deep estimation network training method, system, and storage medium. Background Technology

[0002] Monocular depth estimation aims to determine the depth information between a camera and scene objects from a single RGB image. This depth information has wide applications in fields such as autonomous driving, virtual reality, and 3D reconstruction. In recent years, the development of deep learning has greatly promoted the advancement of depth estimation technology. Traditional monocular depth estimation methods rely on hardware devices and prior assumptions, while the introduction of deep learning methods allows models to automatically learn effective feature representations from large amounts of data. This data-driven approach can better capture depth information in complex scenes, significantly improving the accuracy and robustness of monocular depth estimation.

[0003] Supervised monocular depth estimation requires real depth information as a supervision signal, but this information needs to be acquired using specialized equipment such as LiDAR or depth cameras, which is not only costly but also difficult to fully label in large-scale outdoor scenes. Against this backdrop, self-supervised monocular depth estimation has become a research hotspot due to its characteristic of not requiring real depth information as supervision. The core idea of ​​self-supervised monocular depth estimation methods is to construct a learning framework based on visual geometric constraints. The model generates a reprojected view that matches the target image by predicting disparity, and iteratively optimizes the depth estimation network by minimizing the loss between the reprojected view and the target view. This method relies on the geometric relationships in image pairs or sequences, and has good practicality and generalization. In recent years, significant breakthroughs have been achieved in the performance of self-supervised monocular depth estimation models by introducing innovative methods such as the Transforms network structure, motion consistency constraint mechanisms, temporal feature aggregation modules, and multimodal auxiliary information fusion.

[0004] However, existing self-supervised monocular depth estimation methods still have two key problems in the depth maps they generate: First, the depth maps lack object-level semantic understanding because most current self-supervised depth estimation models output depth values ​​through pixel-wise regression or classification (discrete depth intervals), which ignores the semantic consistency that objects should possess as a whole; Second, there is a geometric distortion problem caused by the lack of strong supervision signals, that is, when performing 3D visualization on the depth maps output by existing models, it is found that objects exhibit severe tilting.

[0005] In related technologies, patent application document CN118229750A proposes to build a shared parameter model that combines semantic segmentation information and depth estimation information. The shared parameter network model uses semantic segmentation information to guide depth estimation to learn additional information. However, this scheme is based on supervised monocular depth estimation and relies on real depth labels. In addition, although the scheme uses a shared encoder, it does not systematically design the parameter sharing mechanism, lacks structural and optimization strategies, and fails to achieve consistent feature learning across different tasks. The self-supervised depth estimation network proposed in patent application CN116258756A includes a Transformer branch and a convolutional branch. The Transformer branch is used to capture global contextual information of the image, and the convolutional branch is used to extract local contextual information of the image. This scheme fuses the features of the CNN branch and the Transformer branch before the last decoding layer, which is a post-fusion strategy in the decoder stage. At this time, the feature map has reached a large size, which is computationally expensive. Moreover, due to the late fusion time, the semantic difference is large. The CNN and Transformer branches are independent for a long time, resulting in poor semantic alignment, high learning difficulty, and the feature space may have large differences during fusion, resulting in unstable fusion operation. Furthermore, this scheme lacks deep collaborative modeling, and local details and global context are not complementary in the intermediate layers. The patent application document with publication number CN112634341A proposes to embed visual semantics into the depth estimation model, and improve the depth estimation accuracy of obstacles by deeply approximating the essence of human perception through a multi-task collaborative and shared learning mode. However, the stereo vision constraints and disparity geometry mentioned in this scheme are concepts of binocular depth estimation, which rely on the geometric relationship between the viewpoints of the left and right cameras. The stereo vision constraints it sets refer to the use of the difference in viewpoints to infer the spatial position of objects by capturing the same scene image through two cameras (left / right) in a binocular vision system. The disparity geometry it sets describes the difference in the horizontal position of the same scene point in the pixel coordinates in the images captured by the two cameras, which is called disparity. Moreover, binocular depth estimation requires left and right images, so the dataset used is more difficult to collect, the training time is longer, and the cost is higher.

[0006] The paper "A Review of Self-Supervised Monocular Dynamic Scene Depth Estimation Based on Deep Learning, Cheng Binbin et al., Journal of Remote Sensing" proposes combining semantic information for dynamic object tracking and depth estimation. It introduces object size constraints on the basic model of self-supervised monocular depth estimation. These object size constraints are specifically designed to constrain the depth estimation of dynamic object regions. They are based on the fundamental assumption that dynamic objects of the same category have similar sizes in the real world, and therefore their depth values ​​should have a certain consistency in the image. Thus, this approach alleviates the uncertainty of dynamic objects in self-supervised depth estimation by introducing object size priors. Summary of the Invention

[0007] One of the technical problems that this invention aims to solve is the geometric distortion problem caused by the lack of strong supervision signals in depth maps.

[0008] The present invention solves the above-mentioned technical problems through the following technical means:

[0009] In a first aspect, this invention proposes a self-supervised depth estimation network training method, the method comprising the following steps:

[0010] Video frame images are collected as training samples, and the training samples are divided into target view and source view;

[0011] The target view is used as the input to the depth estimation network and the semantic segmentation network, and the source view is used as the input to the camera network for training. The trained depth estimation network is used for depth map estimation. The depth estimation network includes a first encoder and a first decoder, and the semantic segmentation network includes a second encoder and a second decoder.

[0012] The total loss function used during training includes feature vector alignment loss, vertical loss, photometric loss, and semantic segmentation loss. Feature vector alignment loss is used to align the feature maps output by the first encoder and the second encoder. Vertical loss is used to make the depth map output by the first decoder satisfy vertical consistency based on the semantic segmentation map output by the second decoder. Photometric loss is used to quantify the difference between the reconstructed view and the target view. Semantic segmentation loss is used to determine the similarity between the semantic segmentation map and the ground truth.

[0013] Furthermore, the first encoder includes a convolutional branch and a Transformers branch. The convolutional branch includes several convolutional neural networks connected in sequence, and the Transformers branch includes several Transformers network blocks connected in sequence. The number of convolutional neural networks and Transformers network blocks is the same. The convolutional branch and the Transformers branch are connected through a multi-stage adaptive fusion network.

[0014] Furthermore, the multi-stage adaptive fusion network includes several layers of fusion modules for connecting convolutional neural networks and Transformers network blocks at corresponding layers, and the output of the last layer of fusion module is connected to the first decoder.

[0015] Furthermore, the first decoder employs an HR-Net network.

[0016] Furthermore, the camera network employs a ResNet-18 network.

[0017] Furthermore, the structure of the second encoder is the same as that of the first encoder, and the structure of the second decoder is the same as that of the first decoder.

[0018] Furthermore, the formula for the feature vector alignment loss is expressed as:

[0019]

[0020] In the formula, This represents the feature map output by the first encoder. This represents the feature map output by the second encoder. Represents pixels, , These represent the height and width of the feature map, respectively. This represents the loss for eigenvector alignment.

[0021] Furthermore, the method also includes:

[0022] The feature vector alignment loss is compared with the alignment tolerance to correct the feature vector alignment loss, as expressed by the formula:

[0023]

[0024] In the formula, This represents the corrected feature vector alignment loss. This represents the feature vector alignment loss. This indicates the alignment tolerance.

[0025] Furthermore, the formula for the vertical loss is expressed as:

[0026]

[0027] In the formula, Indicates vertical loss. Represents the pixels of a vertical object. Represents the normal vector. This represents the unit vector of the direction of gravity in the camera coordinate system. This represents the vertical object mask obtained from the semantic segmentation map.

[0028] Furthermore, the formula for the photometric loss is expressed as:

[0029] = +

[0030] In the formula, , , ,in For hyperparameters, Represents the target view With Reconstructed View Structural similarity between them, || || indicates Norm, which is used to calculate the absolute error at the pixel level.

[0031] Furthermore, the semantic segmentation loss employs the cross-entropy loss function.

[0032] Furthermore, the formula for the total loss function is expressed as follows:

[0033]

[0034] In the formula, Represents the total loss function. Indicates luminosity loss, Represents semantic segmentation loss. Represents the loss of the smoothing term. This represents the corrected feature vector alignment loss. Indicates vertical loss. , , This represents hyperparameters.

[0035] Secondly, the present invention also proposes a self-supervised depth estimation network training system, the system comprising:

[0036] The acquisition module is used to acquire video frame images as training samples and divide the training samples into target view and source view;

[0037] The training module is used to train the target view as the input of the depth estimation network and the semantic segmentation network, and the source view as the input of the camera network, to obtain a trained depth estimation network for depth map estimation. The depth estimation network includes a first encoder and a first decoder, and the semantic segmentation network includes a second encoder and a second decoder.

[0038] The total loss function used during training includes feature vector alignment loss, vertical loss, photometric loss, and semantic segmentation loss. Feature vector alignment loss is used to align the feature maps output by the first encoder and the second encoder. Vertical loss is used to make the depth map output by the first decoder satisfy vertical consistency based on the semantic segmentation map output by the second decoder. Photometric loss is used to quantify the difference between the reconstructed view and the target view. Semantic segmentation loss is used to determine the similarity between the semantic segmentation map and the ground truth.

[0039] Thirdly, the present invention also proposes a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, it implements the self-supervised depth estimation network training method as described above.

[0040] The advantages of this invention are:

[0041] (1) In the encoder training stage, the present invention introduces a pre-trained semantic segmentation network and constrains the spatial consistency between the features of the first encoder and the semantic features output by the second encoder through feature vector alignment loss, thereby implicitly embedding high-level semantic information (such as object category and boundary) into the deep features. The semantic segmentation task implicitly guides the depth estimation network to learn a geometrically reasonable scene representation. In the decoder training stage, the vertical loss is used to make the semantic segmentation map output by the second decoder satisfy the vertical consistency, thereby improving the overall geometric accuracy.

[0042] (2) The depth estimation network is constructed using a hybrid architecture of CNN and Transformers, which enables the depth estimation network to capture local detail features and global context information at the same time.

[0043] Other beneficial effects and advantages of the present invention will be described in the following description, and some of the contents will be reflected by the description of specific embodiments or verified by the actual application of the present invention. Attached Figure Description

[0044] Figure 1 This is a flowchart illustrating a self-supervised depth estimation network training method proposed in one embodiment of the present invention.

[0045] Figure 2 This is a schematic diagram illustrating the training principle of a self-supervised depth estimation network in one embodiment of the present invention;

[0046] Figure 3 This is a schematic diagram of the original frame image, mask image and network output results in one embodiment of the present invention, wherein (a) is the original frame image, (b) is the mask image and (c) is the output depth map;

[0047] Figure 4 This is a schematic diagram of the structure of a self-supervised depth estimation network training system proposed in one embodiment of the present invention. Detailed Implementation

[0048] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below in conjunction with the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0049] like Figure 1As shown, the first embodiment of the present invention proposes a self-supervised depth estimation network training method, which includes the following steps:

[0050] S10. Acquire video frame images as training samples, and divide the training samples into target view and source view;

[0051] S20. The target view is used as the input to the depth estimation network and the semantic segmentation network, and the source view is used as the input to the camera network for training. The trained depth estimation network is used for depth map estimation. The depth estimation network includes a first encoder and a first decoder, and the semantic segmentation network includes a second encoder and a second decoder.

[0052] The total loss function used during training includes feature vector alignment loss, vertical loss, photometric loss, and semantic segmentation loss. Feature vector alignment loss is used to align the feature maps output by the first encoder and the second encoder. Vertical loss is used to make the depth map output by the first decoder satisfy vertical consistency based on the semantic segmentation map output by the second decoder. Photometric loss is used to quantify the difference between the reconstructed view and the target view. Semantic segmentation loss is used to determine the similarity between the semantic segmentation map and the ground truth.

[0053] This embodiment is built on a self-supervised framework, which does not rely on real depth labels and has stronger adaptability and generalization. It also proposes a clear parameter sharing scheme and combines feature alignment loss to explicitly guide the learning of shared features for different tasks in order to achieve consistent feature learning across different tasks.

[0054] Specifically, this embodiment uses video frame images as training samples, and divides the training samples into a training set, a validation set, and a test set, with a ratio of 8:1:1. After training the network using the training set, the network is then tested and validated using the test set and the validation set.

[0055] As a further preferred technical solution, such as Figure 2 As shown, the first encoder includes a convolutional branch and a Transformers branch. The convolutional branch includes several convolutional neural networks connected in sequence, and the Transformers branch includes several Transformers network blocks connected in sequence. The number of convolutional neural networks and Transformers network blocks is the same. The convolutional branch and the Transformers branch are connected through a multi-stage adaptive fusion network.

[0056] It should be noted that in this embodiment, the first encoder adopts a hybrid architecture of parallel convolutional neural networks (CNN) and Transformers. By using Transformer branches to model global contextual dependencies, the limitations of CNN in long-distance feature association are made up for, so that the depth estimation network can capture local detailed features and global contextual information at the same time.

[0057] As a further preferred technical solution, the multi-stage adaptive fusion network includes several layers of fusion modules for connecting convolutional neural networks and Transformers network blocks at corresponding layers, and the output of the last layer of fusion module is connected to the first decoder.

[0058] Specifically, in this embodiment, four convolutional neural networks are set in the convolutional branch, and four Transformers network blocks are set in the Transformers branch. To effectively integrate the feature representations of the CNN branch and the Transformers branch at different levels, this embodiment designs a multi-stage adaptive fusion module. This module aligns and fuses the output features of the two branches in the last two stages, namely the third and fourth stages. When a target view is input, the CNN branch extracts feature maps in four stages, and the number of channels and size of the output of each stage are {64, 128, 256, 512} and {1 / 4, 1 / 8, 1 / 16, 1 / 32}, respectively. Similarly, the number of channels and size of the output of the four stages of the Transformers branch are {96, 192, 384, 768} and {1 / 4, 1 / 8, 1 / 16, 1 / 32}, respectively. First, in the CNN branch, the first... The output feature size of a stage convolutional neural network: , , , They represent the first The height, width, and number of channels of the output features of the first convolutional neural network; the first in the Transformers branch The output feature size of each stage of the Transformers network block: , Specifically refers to the number of channels output by the Transformers branch.

[0059] Specifically, in the third stage, the output features of the third-stage convolutional neural network and the output features of the third-stage Transformers network block are concatenated by the first fusion module as follows:

[0060]

[0061] When using the second fusion module to concatenate the features of the fourth stage, the input of the second fusion module includes the output features of the fourth stage convolutional neural network, the output features of the fourth stage Transformers network block, and the output of the first fusion module:

[0062]

[0063] in, and This indicates the splicing features of stages 3 and 4; Concat indicates feature splicing. Indicates the feature Perform a downsampling operation, wherein the downsampling operation is set between the first fusion module and the second fusion module.

[0064] Furthermore, this embodiment also includes a convolutional layer after the second fusion module for feature compression of the spliced ​​features in the fourth stage:

[0065]

[0066] in, This represents the output feature after the two branches are merged; Indicates to Perform a 1×1 convolution operation; This indicates the number of output channels; in this experiment, it is set to 512.

[0067] It should be noted that this embodiment can also include a fusion module between each corresponding layer of the convolutional neural network and Transformer network blocks, in which case the output of the previous stage fusion module becomes the output of the next stage fusion module. Compared with the scheme described in the patent application document with publication number CN116258756A, this embodiment fuses the CNN branch and the Transformer branch in the encoder stage. Specifically, a fusion module is introduced in the last two stages of the encoder stage. Since the features in the later stages of the encoder already have strong semantic expressive power, fusing the features of the CNN branch and the Transformer branch at this time can preserve key local geometric structures while modeling the global context. Deep fusion is completed before the decoder, providing composite features with both local and global semantics for the upsampling path. Moreover, the difficulty of cross-modal feature alignment is reduced. Compared with fusion in the high-resolution stage, the feature map resolution at the end of the encoder is lower, resulting in lower computational cost in the spatial dimension, which is beneficial for realizing feature alignment and fusion operations.

[0068] As a further preferred technical solution, the first decoder uses the high-resolution network HR-Net.

[0069] It should be noted that those skilled in the art may also select other networks besides HR-Net in practical applications, and this embodiment does not impose specific limitations.

[0070] As a further preferred technical solution, the camera network adopts a lightweight ResNet-18 network, the input is set to a source view composed of two adjacent views, the number of channels is six, and the camera network outputs a pose change matrix through decoding.

[0071] As a further preferred technical solution, the structure of the second encoder is the same as that of the first encoder, and the structure of the second decoder is the same as that of the first decoder.

[0072] Specifically, the encoder structure of the semantic segmentation network is the same as that of the depth estimation network, and they are independent and do not share the same structure. The goal is to integrate semantic information into a high-dimensional, continuous feature space in the encoder. During training, both the semantic segmentation network and the depth estimation network are input to the target view, but the two networks are trained independently. In the encoder part, the semantic information is aligned in the feature vector direction in a high-dimensional, continuous feature space.

[0073] As a further preferred technical solution, in the network training process: the deep estimation network input is set as the target view. The output is a depth map. Camera network input is set to source view. or The output is the pose change matrix. .

[0074] By combining the depth map and the pose change matrix, a reconstruction formula is constructed:

[0075]

[0076] in, This indicates that the source view is projected onto the target view; K This is the camera intrinsic parameter matrix; Represents the reprojection function; This indicates bilinear interpolation sampling, used to obtain pixel values ​​at non-integer pixel coordinates.

[0077] Finally, monitoring was performed using luminosity loss:

[0078] = +

[0079] In the formula, , , ,in For hyperparameters, Represents the target view With Reconstructed View Structural similarity between them, || || indicates Norm, which is used to calculate the absolute error at the pixel level.

[0080] The semantic segmentation network employs an independent encoding / decoding structure, with the same input settings as the depth estimation network, and outputs a semantic segmentation map. The semantic segmentation loss function uses cross-entropy loss.

[0081]

[0082] in, , representing pixels Does it belong to a category? , A value of 1 indicates a pixel. Category , A value of 0 indicates a pixel. Not belonging to category ; Represents pixels Category The predicted probability.

[0083] Furthermore, the semantic prior module in the encoder stage of the semantic segmentation network incorporates the high-level semantic information extracted by the semantic segmentation network into the deep estimation network through an interactive guidance mechanism constructed during the training phase. Feature vector alignment loss is then built in the encoder part. Align the feature vector directions of semantic information in a high-dimensional, continuous feature space:

[0084]

[0085] in, This represents the feature map output by the first encoder; This represents the feature map output by the second encoder; Represents a pixel.

[0086] As a further preferred technical solution, the method further includes:

[0087] The feature vector alignment loss is compared with the alignment tolerance to correct the feature vector alignment loss, as expressed by the formula:

[0088]

[0089] In the formula, This represents the corrected feature vector alignment loss. This represents the feature vector alignment loss. This indicates the alignment tolerance.

[0090] It should be noted that, to compensate for the differences between the two tasks, this embodiment sets an alignment tolerance. Its value can be set to 0.15. When and The cosine similarity exceeds 1- The alignment loss is 0.85. The loss of this pixel is not considered.

[0091] As a further preferred technical solution, this embodiment sets up a geometric correction module to construct a vertical loss during the decoder training phase of the semantic segmentation network. The vertical loss utilizes the semantic segmentation map output by the second decoder to ensure that the depth map output by the first decoder satisfies vertical consistency, thereby improving overall geometric accuracy. The process of constructing the vertical constraint includes:

[0092] (1) Vertical object mask: When the depth output by the traditional depth estimation model is... Figure 3 After visualization, problems that violate natural phenomena often arise. Structures such as buildings, utility poles, and trees, which should be perpendicular to the ground, appear tilted, and the tilt angle worsens with increasing distance. This is due to the weak constraint on vertical geometry imposed by the photometric consistency loss during self-supervised training. Figure 3 As shown, this embodiment explicitly corrects depth by applying strong constraints to the vertical object, wherein... Figure 3 (a) in the image represents the original image. Figure 3 In the image (b), the mask of a vertical object (black pixels, telephone poles, etc.) is approximately perpendicular to the ground (gray pixels). Figure 3 (c) in the diagram represents the network's output depth map. First, the vertical object mask is obtained from the semantic segmentation network. :

[0093]

[0094] in, For image pixels; For vertical object pixels.

[0095] (2) Vertical constraint: Calculate the three-dimensional spatial coordinates of the perpendicular object. :

[0096]

[0097] in, These are the pixels of the vertical object in the depth map; Represents pixels Homogeneous coordinates.

[0098] Calculation of the tangent vector: For Perform operations on the pixels in the array to obtain the difference between adjacent pixels:

[0099]

[0100]

[0101] in, Indicates in Depth variation along the horizontal direction; Indicates in The depth variation along the vertical direction. Represents pixels The three-dimensional spatial coordinates of the projection Represents pixels The three-dimensional spatial coordinates of the projection. Represents pixels The three-dimensional spatial coordinates of the projection;

[0102] Calculation of normal vector:

[0103]

[0104] in, express .

[0105] Perpendicular geometric condition: Let the unit vector of the gravity direction in the camera coordinate system be... The vertical condition is: ;

[0106] Construction of the vertical loss function:

[0107]

[0108] In the formula, Indicates vertical loss. Represents the pixels of a vertical object. Represents the normal vector. This represents the unit vector of the direction of gravity in the camera coordinate system. This represents the vertical object mask obtained from the semantic segmentation map.

[0109] As a further preferred technical solution, the formula for the total loss function is expressed as follows:

[0110]

[0111] In the formula, Represents the total loss function. Indicates luminosity loss, Represents semantic segmentation loss. Represents the loss of the smoothing term. This represents the corrected feature vector alignment loss. Indicates vertical loss. , , This represents hyperparameters.

[0112] Specifically, smoothing term loss The calculation formula is:

[0113]

[0114] In the formula, and In the depth map x , y Gradient along the axial direction; and Indicates the image in x , y Gradient along the axial direction.

[0115] In addition, such as Figure 4 As shown, the second embodiment of the present invention also proposes a self-supervised depth estimation network training system, the system comprising:

[0116] The acquisition module 10 is used to acquire video frame images as training samples and divide the training samples into target view and source view;

[0117] The training module 20 is used to train the target view as the input of the depth estimation network and the semantic segmentation network, and the source view as the input of the camera network, to obtain a trained depth estimation network for depth map estimation. The depth estimation network includes a first encoder and a first decoder, and the semantic segmentation network includes a second encoder and a second decoder.

[0118] The total loss function used during training includes feature vector alignment loss, vertical loss, photometric loss, and semantic segmentation loss. Feature vector alignment loss is used to align the feature maps output by the first encoder and the second encoder. Vertical loss is used to make the depth map output by the first decoder satisfy vertical consistency based on the semantic segmentation map output by the second decoder. Photometric loss is used to quantify the difference between the reconstructed view and the target view. Semantic segmentation loss is used to determine the similarity between the semantic segmentation map and the ground truth.

[0119] As a further preferred technical solution, the first encoder includes a convolutional branch and a Transformers branch. The convolutional branch includes several convolutional neural networks connected in sequence, and the Transformers branch includes several Transformers network blocks connected in sequence. The number of convolutional neural networks and Transformers network blocks is the same. The convolutional branch and the Transformers branch are connected through a multi-stage adaptive fusion network.

[0120] As a further preferred technical solution, the multi-stage adaptive fusion network includes several layers of fusion modules for connecting convolutional neural networks and Transformers network blocks at corresponding layers, and the output of the last layer of fusion module is connected to the first decoder.

[0121] As a further preferred technical solution, the first decoder adopts the HR-Net network.

[0122] As a further preferred technical solution, the camera network adopts a ResNet-18 network.

[0123] As a further preferred technical solution, the structure of the second encoder is the same as that of the first encoder, and the structure of the second decoder is the same as that of the first decoder.

[0124] As a further preferred technical solution, the formula for the feature vector alignment loss is expressed as:

[0125]

[0126] In the formula, This represents the feature map output by the first encoder. This represents the feature map output by the second encoder. Represents pixels, , These represent the height and width of the feature map, respectively. This represents the loss for eigenvector alignment.

[0127] As a further preferred technical solution, the feature vector alignment loss is compared with the alignment tolerance to correct the feature vector alignment loss, as expressed by the formula:

[0128]

[0129] In the formula, This represents the corrected feature vector alignment loss. This represents the feature vector alignment loss. This indicates the alignment tolerance.

[0130] As a further preferred technical solution, the formula for the vertical loss is expressed as:

[0131]

[0132] In the formula, Indicates vertical loss. Represents the vertical pixel of an object. Represents the normal vector. This represents the unit vector of the direction of gravity in the camera coordinate system. This represents the vertical object mask obtained from the semantic segmentation map.

[0133] As a further preferred technical solution, the formula for the total loss function is expressed as follows:

[0134]

[0135] In the formula, Represents the total loss function. Indicates luminosity loss, Represents semantic segmentation loss. Represents the loss of the smoothing term. This represents the corrected feature vector alignment loss. Indicates vertical loss. , , This represents hyperparameters.

[0136] It should be noted that other embodiments or specific implementation methods of the self-supervised depth estimation network training system described in this invention can refer to the above-described method embodiments, and will not be repeated here.

[0137] Furthermore, the third embodiment of the present invention also proposes a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the self-supervised depth estimation network training method as described in the first embodiment above.

[0138] It should be noted that the computer-readable medium disclosed in this embodiment may be a computer-readable signal medium, a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0139] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device. The aforementioned computer-readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to perform a zero-sample image anomaly detection method according to the above embodiments.

[0140] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server.

[0141] In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0142] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0143] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0144] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this invention, "a plurality of" or "several" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0145] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A self-supervised depth estimation network training method, characterized in that, include: Video frame images are collected as training samples, and the training samples are divided into target view and source view; The target view is used as the input to the depth estimation network and the semantic segmentation network, and the source view is used as the input to the camera network for training. The trained depth estimation network is used for depth map estimation. The depth estimation network includes a first encoder and a first decoder, and the semantic segmentation network includes a second encoder and a second decoder. The total loss function used during training includes feature vector alignment loss, vertical loss, photometric loss, and semantic segmentation loss. Feature vector alignment loss is used to align the feature maps output by the first encoder and the second encoder. Vertical loss is used to make the depth map output by the first decoder satisfy vertical consistency based on the semantic segmentation map output by the second decoder. Photometric loss is used to quantify the difference between the reconstructed view and the target view. Semantic segmentation loss is used to determine the similarity between the semantic segmentation map and the ground truth. The formula for the vertical loss is expressed as follows: In the formula, Indicates vertical loss. Represents the pixels of a vertical object. Represents the normal vector. This represents the unit vector of the direction of gravity in the camera coordinate system. This represents the vertical object mask obtained from the semantic segmentation map.

2. The self-supervised depth estimation network training method as described in claim 1, characterized in that, The first encoder includes a convolutional branch and a Transformers branch. The convolutional branch includes several convolutional neural networks connected in sequence, and the Transformers branch includes several Transformers network blocks connected in sequence. The number of convolutional neural networks and Transformers network blocks are the same. The convolutional branch and the Transformers branch are connected by a multi-stage adaptive fusion network.

3. The self-supervised depth estimation network training method as described in claim 2, characterized in that, The multi-stage adaptive fusion network includes several layers of fusion modules for connecting convolutional neural networks and Transformers network blocks at corresponding levels, and the output of the last layer of fusion module is connected to the first decoder.

4. The self-supervised depth estimation network training method as described in claim 1, characterized in that, The first decoder uses the HR-Net network.

5. The self-supervised depth estimation network training method as described in claim 1, characterized in that, The camera network uses a ResNet-18 network.

6. The self-supervised depth estimation network training method as described in claim 1, characterized in that, The structure of the second encoder is the same as that of the first encoder, and the structure of the second decoder is the same as that of the first decoder.

7. The self-supervised depth estimation network training method as described in claim 1, characterized in that, The formula for the feature vector alignment loss is expressed as follows: In the formula, This represents the feature map output by the first encoder. This represents the feature map output by the second encoder. Represents pixels, , These represent the height and width of the feature map, respectively. This represents the loss for eigenvector alignment.

8. The self-supervised depth estimation network training method as described in claim 7, characterized in that, The method further includes: The feature vector alignment loss is compared with the alignment tolerance to correct the feature vector alignment loss, as expressed by the formula: In the formula, This represents the corrected feature vector alignment loss. This represents the feature vector alignment loss. This indicates the alignment tolerance.

9. The self-supervised depth estimation network training method as described in claim 1, characterized in that, The formula for the photometric loss is expressed as: = + In the formula, , , ,in For hyperparameters, Represents the target view With Reconstructed View Structural similarity between them, || || indicates the calculation of pixel-level absolute error.

10. The self-supervised depth estimation network training method as described in claim 1, characterized in that, The semantic segmentation loss uses the cross-entropy loss function.

11. The self-supervised depth estimation network training method as described in claim 1, characterized in that, The formula for the total loss function is expressed as follows: In the formula, Represents the total loss function. Indicates luminosity loss, Represents semantic segmentation loss. Represents the loss of the smoothing term. This represents the corrected feature vector alignment loss. Indicates vertical loss. , , This represents hyperparameters.

12. A self-supervised depth estimation network training system, characterized in that, include: The acquisition module is used to acquire video frame images as training samples and divide the training samples into target view and source view; The training module is used to train the target view as the input of the depth estimation network and the semantic segmentation network, and the source view as the input of the camera network, to obtain a trained depth estimation network for depth map estimation. The depth estimation network includes a first encoder and a first decoder, and the semantic segmentation network includes a second encoder and a second decoder. The total loss function used during training includes feature vector alignment loss, vertical loss, photometric loss, and semantic segmentation loss. Feature vector alignment loss is used to align the feature maps output by the first encoder and the second encoder. Vertical loss is used to make the depth map output by the first decoder satisfy vertical consistency based on the semantic segmentation map output by the second decoder. Photometric loss is used to quantify the difference between the reconstructed view and the target view. Semantic segmentation loss is used to determine the similarity between the semantic segmentation map and the ground truth. The formula for the vertical loss is expressed as follows: In the formula, Indicates vertical loss. Represents the pixels of a vertical object. Represents the normal vector. This represents the unit vector of the direction of gravity in the camera coordinate system. This represents the vertical object mask obtained from the semantic segmentation map.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the self-supervised depth estimation network training method as described in any one of claims 1-11.

Images