Self-supervised depth estimation network training method and system, and storage medium

Abstract

The invention discloses a self-supervised depth estimation network training method and system and a storage medium, and the method comprises the steps: collecting a video frame image as a training sample, and dividing the training sample into a target view and a source view; the target view is used as the input of a depth estimation network and a semantic segmentation network, the source view is used as the input of a camera network for training, the obtained trained depth estimation network is used for depth map estimation, and the depth estimation network comprises a first encoder and a first decoder; the semantic segmentation network comprises a second encoder and a second decoder; a total loss function adopted during training comprises feature vector alignment loss, vertical loss, luminosity loss and semantic segmentation loss, and the feature vector alignment loss is used for performing direction alignment on feature maps output by the two encoders; the vertical loss is used to cause the depth map output by the first decoder to satisfy vertical consistency based on the semantic segmentation map output by the second decoder.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a self-supervised depth estimation network training method, system and storage medium. Background Art

[0002] Monocular depth estimation aims to determine the depth between the camera and scene objects from a single RGB image. This depth information has wide-ranging applications in autonomous driving, virtual reality, 3D reconstruction, and other fields. In recent years, the development of deep learning has greatly promoted the advancement of depth estimation technology. Traditional monocular depth estimation methods rely on hardware devices and prior assumptions, while the introduction of deep learning methods allows models to automatically learn effective feature representations from large amounts of data. This data-driven approach can better capture depth information in complex scenes, significantly improving the accuracy and robustness of monocular depth estimation.

[0003] Supervised monocular depth estimation requires real depth information as a supervisory signal, but real depth information needs to be collected through specialized equipment such as LiDAR or depth cameras. This is not only costly but also difficult to fully annotate in large-scale outdoor scenes. In this context, self-supervised monocular depth estimation has become a research hotspot due to its feature of not requiring real depth information as supervision. The core idea of ​​self-supervised monocular depth estimation methods is to construct a learning framework based on visual geometric constraints. The model generates a reprojected view that matches the target image by predicting disparity, and iteratively optimizes the depth estimation network by minimizing the loss between the reprojected view and the target view. This method relies on the geometric relationship between image pairs or sequences and has good practicality and scalability. In recent years, by introducing innovative methods such as the Transforms network structure, motion consistency constraint mechanism, temporal feature aggregation module, and multimodal auxiliary information fusion, the performance of self-supervised monocular depth estimation models has achieved significant breakthroughs.

[0004] However, the depth maps generated by existing self-supervised monocular depth estimation methods still have two key problems: first, the depth maps lack object-level semantic understanding, because most current self-supervised depth estimation models output depth values ​​through pixel-by-pixel regression or classification (discrete depth intervals), which ignores the semantic consistency that the object as a whole should have; second, the lack of strong supervision signals causes geometric distortion, that is, when the depth maps output by existing models are visualized in three dimensions, it is found that the objects are severely tilted.

[0005] In the related art, the patent application document with publication number CN118229750A proposes to build a shared parameter model that combines semantic segmentation information and depth estimation information. The shared parameter network model uses semantic segmentation information to guide depth estimation to learn additional information; however, this scheme is based on a supervised monocular depth estimation method and relies on real depth labels. In addition, although the scheme adopts a shared encoder, it does not systematically design the parameter sharing mechanism, lacks structural and optimization strategy support, and does not achieve consistent learning of features between different tasks. The self-supervised depth estimation network proposed in the patent application document with publication number CN116258756A includes a Transformer branch and a convolution branch. The Transformer branch is used to capture the global context information of the image, and the convolution branch is used to extract the local context information of the image. This scheme is to fuse the features of the CNN branch and the Transformer branch before the last decoding layer, which is a post-fusion strategy in the decoder stage. At this time, the feature map has reached a large size and the computational cost is very high. Moreover, due to the late fusion time and the large semantic gap, the CNN and Transformer branches are independent for a long time, resulting in poor semantic alignment and high learning difficulty. The feature space may have large distribution differences during fusion, and the fusion operation effect is unstable. Moreover, this scheme lacks deep collaborative modeling, and local details and global context are not complemented in the middle layer. The patent application document with publication number CN112634341A proposes embedding visual semantics into the depth estimation model, and through a multi-task collaborative sharing learning model, deeply approximates the essence of human perception and improves the depth estimation accuracy of obstacles. However, the stereo vision constraints and disparity geometry mentioned in this scheme are concepts of binocular depth estimation, which rely on the geometric relationship between the left and right camera perspectives. The stereo vision constraints set here refer to capturing the same scene image through two cameras (left / right) in a binocular vision system, and using the perspective difference to infer the spatial position of the object. The disparity geometry set here describes the horizontal position difference in pixel coordinates of the same scene point in the images captured by the two cameras, which is called disparity. Moreover, binocular depth estimation requires left and right images, so the dataset used is more difficult to collect, the training time is longer, and the cost is higher.

[0006] In the paper "A Review of Self-supervised Monocular Depth Estimation of Dynamic Scenes Based on Deep Learning, Cheng Binbin et al., Journal of Remote Sensing", it is proposed to combine semantic information for dynamic object tracking and depth estimation, and introduce object size constraints on the basis of the basic model of self-supervised monocular depth estimation. This object size constraint is specifically used to constrain the depth estimation of dynamic object areas. It is based on a basic assumption: dynamic objects of the same category have similar sizes in the real world, so their depth values ​​​​should have a certain consistency in the image. Therefore, this scheme is to alleviate the uncertainty of dynamic objects in self-supervised depth estimation by introducing object size priors. Summary of the Invention

[0007] One of the technical problems to be solved by the present invention is the geometric distortion problem caused by the lack of a strong supervisory signal in the depth map.

[0008] The present invention solves the above technical problems through the following technical means: In a first aspect, the present invention proposes a self-supervised depth estimation network training method, the method comprising the following steps: Collect video frame images as training samples and divide the training samples into target views and source views; Training the target view as input to a depth estimation network and a semantic segmentation network and the source view as input to a camera network to obtain a trained depth estimation network for depth map estimation, wherein the depth estimation network includes a first encoder and a first decoder, and the semantic segmentation network includes a second encoder and a second decoder; Among them, the total loss function used during training includes feature vector alignment loss, vertical loss, photometric loss and semantic segmentation loss. The feature vector alignment loss is used to align the feature maps output by the first encoder and the second encoder in a directional manner. The vertical loss is used to force the depth map output by the first decoder to meet vertical consistency based on the semantic segmentation map output by the second decoder. The photometric loss is used to quantify the difference between the reconstructed view and the target view. The semantic segmentation loss is used to determine the similarity between the semantic segmentation map and the true value.

[0009] Furthermore, the first encoder includes a convolution branch and a Transformers branch, the convolution branch includes several sequentially connected convolutional neural networks, the Transformers branch includes several sequentially connected Transformers network blocks, and the number of convolutional neural networks and Transformers network blocks is the same, and the convolution branch and the Transformers branch are connected through a multi-stage adaptive fusion network.

[0010] Furthermore, the multi-stage adaptive fusion network includes several layers of fusion modules for connecting the convolutional neural networks and Transformers network blocks of the corresponding layers, and the output of the fusion module of the last layer is connected to the first decoder.

[0011] Furthermore, the first decoder adopts an HR-Net network.

[0012] Furthermore, the camera network adopts a ResNet-18 network.

[0013] Furthermore, the structure of the second encoder is the same as that of the first encoder, and the structure of the second decoder is the same as that of the first decoder.

[0014] Furthermore, the formula of the feature vector alignment loss is expressed as:

[0015] Where, represents the feature map output by the first encoder, represents the feature map output by the second encoder, Represents a pixel, 、 Represent the height and width of the feature map respectively, represents the feature vector alignment loss.

[0016] Furthermore, the method further comprises: The feature vector alignment loss is compared with the alignment tolerance to correct the feature vector alignment loss, which is expressed as:

[0017] Where, represents the corrected feature vector alignment loss, represents the feature vector alignment loss before correction, Indicates the alignment tolerance.

[0018] Furthermore, the formula for the vertical loss is expressed as:

[0019] Where, represents the vertical loss, Indicates vertical object pixels, represents the normal vector, Represents the unit vector of the gravity direction in the camera coordinate system, Represents the vertical object mask obtained from the semantic segmentation map.

[0020] Furthermore, the formula for the luminosity loss is expressed as: = +

[0021] Where, , , ,in is a hyperparameter, Represents the target view and refactoring views The structural similarity between || || means Norm, that is, calculating the absolute error at the pixel level.

[0022] Furthermore, the semantic segmentation loss adopts a cross entropy loss function.

[0023] Furthermore, the formula of the total loss function is expressed as:

[0024] Where, represents the total loss function, represents the loss of luminosity, represents the semantic segmentation loss, represents the smoothing loss, represents the feature vector alignment loss, represents the vertical loss, 、 、 represents a hyperparameter.

[0025] In a second aspect, the present invention further proposes a self-supervised depth estimation network training system, the system comprising: An acquisition module is used to acquire video frame images as training samples and divide the training samples into target views and source views; a training module, configured to train the target view as input to a depth estimation network and a semantic segmentation network and the source view as input to a camera network, thereby obtaining a trained depth estimation network for depth map estimation, wherein the depth estimation network includes a first encoder and a first decoder, and the semantic segmentation network includes a second encoder and a second decoder; Among them, the total loss function used during training includes feature vector alignment loss, vertical loss, photometric loss and semantic segmentation loss. The feature vector alignment loss is used to align the feature maps output by the first encoder and the second encoder in a directional manner. The vertical loss is used to force the depth map output by the first decoder to meet vertical consistency based on the semantic segmentation map output by the second decoder. The photometric loss is used to quantify the difference between the reconstructed view and the target view. The semantic segmentation loss is used to determine the similarity between the semantic segmentation map and the true value.

[0026] In a third aspect, the present invention further proposes a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the self-supervised depth estimation network training method as described above is implemented.

[0027] The advantages of the present invention are: (1) In the encoder training stage, the present invention introduces a pre-trained semantic segmentation network, and constrains the spatial consistency of the first encoder feature and the semantic feature output by the second encoder through the feature vector alignment loss, thereby implicitly embedding high-level semantic information (such as object category and boundary) into the depth feature, and implicitly guides the depth estimation network to learn geometrically reasonable scene representation through the semantic segmentation task; in the decoder training stage, the semantic segmentation map output by the second decoder is used through the vertical loss to force the depth map output by the first decoder to meet the vertical consistency, thereby improving the overall geometric accuracy.

[0028] (2) The depth estimation network is constructed using a hybrid architecture of CNN and Transformers, which enables the depth estimation network to capture local detail features and global context information at the same time.

[0029] Other beneficial effects and advantages of the present invention will be elaborated in the following description, and some of the contents will be reflected through the description of specific implementation methods or verified through the actual application of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 1 is a flow chart of a self-supervised depth estimation network training method proposed in one embodiment of the present invention; Figure 2 2 is a schematic diagram of the training principle of a self-supervised depth estimation network according to an embodiment of the present invention; Figure 3 Schematic diagram of the original frame image, mask image and network output result in one embodiment of the present invention, where (a) is the original frame image, (b) is the mask image, and (c) is the output depth map; Figure 4 2 is a structural diagram of a self-supervised depth estimation network training system proposed in one embodiment of the present invention. DETAILED DESCRIPTION

[0031] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0032] like Figure 1 As shown, the first embodiment of the present invention proposes a self-supervised depth estimation network training method, which includes the following steps: S10, collecting video frame images as training samples, and dividing the training samples into target views and source views; S20, training the target view as input to a depth estimation network and a semantic segmentation network, and the source view as input to a camera network, to obtain a trained depth estimation network for depth map estimation, wherein the depth estimation network includes a first encoder and a first decoder, and the semantic segmentation network includes a second encoder and a second decoder; Among them, the total loss function used during training includes feature vector alignment loss, vertical loss, photometric loss and semantic segmentation loss. The feature vector alignment loss is used to align the feature maps output by the first encoder and the second encoder in a directional manner. The vertical loss is used to force the depth map output by the first decoder to meet vertical consistency based on the semantic segmentation map output by the second decoder. The photometric loss is used to quantify the difference between the reconstructed view and the target view. The semantic segmentation loss is used to determine the similarity between the semantic segmentation map and the true value.

[0033] This embodiment is built on a self-supervised framework, does not rely on real depth labels, and has stronger adaptability and generalizability. It also proposes a clear parameter sharing scheme and combines feature alignment loss to clearly guide the shared feature learning of different tasks to achieve consistent feature learning between different tasks.

[0034] Specifically, this embodiment uses video frame images as training samples and divides the training samples into a training set, a validation set, and a test set, with the ratio of each part being 8:1:1. After the network is trained using the training set, the network is tested and verified using the test set and validation set.

[0035] As a further preferred technical solution, Figure 2 As shown, the first encoder includes a convolution branch and a Transformers branch. The convolution branch includes several sequentially connected convolutional neural networks, and the Transformers branch includes several sequentially connected Transformers network blocks. The number of convolutional neural networks and Transformers network blocks is the same, and the convolution branch and the Transformers branch are connected through a multi-stage adaptive fusion network.

[0036] It should be noted that the first encoder in this embodiment adopts a hybrid architecture of convolutional neural networks (CNN) and Transformers in parallel. By using Transformers branches to model global context dependencies, it compensates for the limitations of CNN in long-distance feature association, enabling the depth estimation network to capture local detail features and global context information at the same time.

[0037] As a further preferred technical solution, the multi-stage adaptive fusion network includes several levels of fusion modules for connecting the convolutional neural networks and Transformers network blocks of the corresponding levels, and the output of the fusion module of the last level is connected to the first decoder.

[0038] Specifically, in this embodiment, four convolutional neural networks are set in the convolution branch, and four Transformers network blocks are set in the Transformers branch. In order to effectively integrate the feature expressions of the CNN branch and the Transformers branch at different levels, this embodiment designs a multi-stage adaptive fusion module. This module aligns and fuses the output features of the two branches in the last two stages, namely the third stage and the fourth stage. After inputting a target view, the CNN branch has four stages to extract feature maps, and the number of channels and sizes of the outputs of each stage are {64, 128, 256, 512}, {1 / 4, 1 / 8, 1 / 16, 1 / 32} respectively. Similarly, the number of channels and sizes of the outputs of the four stages of the Transformers branch are {96, 192, 384, 768}, {1 / 4, 1 / 8, 1 / 16, 1 / 32} respectively. First, the first stage in the CNN branch The output feature size of the convolutional neural network at each stage is: , 、 、 Respectively represent The height, width, and number of channels of the convolutional neural network output features; the first The output feature size of the Transformers network block at each stage is: , Specifically refers to the number of channels output by the Transformers branch.

[0039] In particular, in the third stage, the output features of the third-stage convolutional neural network and the output features of the third-stage Transformers network block are spliced ​​through the first fusion module as follows:

[0040] When the second fusion module is used to concatenate the features of the fourth stage, the input of the second fusion module includes the output features of the fourth stage convolutional neural network, the output features of the fourth stage Transformers network block, and the output of the first fusion module:

[0041] in, and Indicates the concatenation of features from the third and fourth stages; Concat indicates feature concatenation; Represents the feature A downsampling operation is performed, wherein the downsampling operation is set between the first fusion module and the second fusion module.

[0042] Furthermore, this embodiment also provides a convolution layer after the second fusion module to perform feature compression on the splicing features of the fourth stage:

[0043] in, Represents the features output after the fusion of the two branches; Express Perform 1×1 convolution operation; Indicates the number of output channels, which is set to 512 in the experiment.

[0044] It should be noted that, in this embodiment, a fusion module can also be provided between the convolutional neural network and the Transformers network block at each corresponding level, and the output of the fusion module of the previous stage is used as the output of the fusion module of the next stage. Compared with the solution described in the patent application document with publication number CN116258756A, this embodiment fuses the CNN branch and the Transformer branch at the encoder stage. Specifically, the fusion module is introduced in the last two stages of the encoder stage for fusion. Since the features of the encoder in the later stage already have strong semantic expression capabilities, the fusion of the features of the CNN branch and the Transformer branch at this time can retain the key local geometric structure while modeling the global context, complete deep fusion before the decoder, and provide the upsampling path with composite features that have both local and global semantics. Moreover, the difficulty of cross-modal feature alignment is reduced. Compared with fusion at the high-resolution stage, the feature map resolution at the end of the encoder is lower, and the computational cost in the spatial dimension is lower, which is conducive to achieving feature alignment and fusion operations.

[0045] As a further preferred technical solution, the first decoder adopts a high-resolution network HR-Net.

[0046] It should be noted that those skilled in the art may also select other networks besides the HR-Net network in actual applications, and this embodiment does not specifically limit this.

[0047] As a further preferred technical solution, the camera network adopts a lightweight ResNet-18 network, the input is set to a source view consisting of two adjacent views, the number of channels is six, and the camera network outputs a pose change matrix through decoding.

[0048] As a further preferred technical solution, the structure of the second encoder is the same as that of the first encoder, and the structure of the second decoder is the same as that of the first decoder.

[0049] Specifically, the encoder structure of the semantic segmentation network is identical to that of the depth estimation network, and they are independent and unshared. The goal is to integrate semantic information into a high-dimensional, continuous feature space at the encoder. During training, the inputs to both the semantic segmentation network and the depth estimation network are the target view, but the two networks are trained independently. The encoder aligns the semantic information in the high-dimensional, continuous feature space by aligning the feature vectors.

[0050] As a further preferred technical solution, during the network training process: the deep estimation network input is set as the target view , the output is a depth map The camera network input is set to the source view or , the output is the pose change matrix .

[0051] By combining the depth map and the pose change matrix, the reconstruction formula is constructed:

[0052] in, Indicates projecting the source view to the target view; K is the camera intrinsic parameter matrix; represents the reprojection function; Represents bilinear interpolation sampling, which is used to obtain pixel values ​​at non-integer pixel coordinates.

[0053] Finally supervision is performed through photometric loss: = +

[0054] Where, , , ,in is a hyperparameter, Represents the target view and refactoring views The structural similarity between || || means Norm, that is, calculating the absolute error at the pixel level.

[0055] The semantic segmentation network uses an independent encoding and decoding structure. The input setting is the same as the depth estimation network. The output is a semantic segmentation map. The semantic segmentation loss function uses cross entropy loss:

[0056] in, , representing pixels Belongs to category , A value of 1 indicates a pixel Belong to category , A value of 0 indicates a pixel Not in category ; Represents pixels Belong to category The predicted probability of .

[0057] Furthermore, the semantic prior module is set in the encoder stage of the semantic segmentation network to integrate the high-level semantic information extracted by the semantic segmentation network into the depth estimation network by constructing an interactive guidance mechanism in the training stage, and constructing a feature vector alignment loss in the encoder part. Align the semantic information in the high-dimensional, continuous feature space by aligning the feature vector direction:

[0058] in, A feature map representing the output of the first encoder; A feature map representing the output of the second encoder; Represents a pixel.

[0059] As a further preferred technical solution, the method further includes: The feature vector alignment loss is compared with the alignment tolerance to correct the feature vector alignment loss, which is expressed as:

[0060] Where, represents the corrected feature vector alignment loss, represents the feature vector alignment loss before correction, Indicates the alignment tolerance.

[0061] It should be noted that in order to compensate for the differences between the two tasks, this embodiment sets the alignment tolerance , its value can be set to 0.15. and The cosine similarity of , which is 0.85, then the alignment loss The loss of this pixel is not considered.

[0062] As a further preferred technical solution, this embodiment sets a geometric correction module to construct a vertical loss during the decoder training phase of the semantic segmentation network. The vertical loss uses the semantic segmentation map output by the second decoder to force the depth map output by the first decoder to meet vertical consistency, thereby improving the overall geometric accuracy. The vertical constraint construction process includes: (1) Vertical object mask: When the depth output of the traditional depth estimation model is Figure 3 After dimensional visualization, problems often occur that violate natural phenomena. Structures that should be perpendicular to the ground, such as buildings, telephone poles, and trees, appear tilted, and the tilt angle increases with distance. This is due to the weak constraint of vertical geometry imposed by the photometric consistency loss in self-supervised training. Figure 3 As shown, this embodiment explicitly corrects depth by imposing strong constraints on vertical objects, where Figure 3 (a) in the figure represents the original image. Figure 3 (b) in the figure shows that the mask of vertical objects (black pixels, objects such as telephone poles) and the ground (gray pixels) are approximately perpendicular to each other. Figure 3 (c) in the figure is the output depth map of the network. First, we get the vertical object mask from the semantic segmentation network. :

[0063] in, is the image pixel; is the vertical object pixel.

[0064] (2) Vertical constraint: Calculate the three-dimensional space coordinates of vertical objects :

[0065] in, is the pixel point of the vertical object in the depth map; Represents pixels The homogeneous coordinates of .

[0066] Calculation of tangent vector: The pixels in are operated to obtain the adjacent pixel differences:

[0067]

[0068] in, Indicates The depth variation along the horizontal direction; Indicates The depth variation along the vertical direction, Represents pixel points The projected three-dimensional space coordinates, Represents pixel points The projected three-dimensional space coordinates, Represents pixel points The three-dimensional space coordinates of the projection; Calculation of normal vector:

[0069] in, express .

[0070] Vertical geometric conditions: Let the unit vector of gravity direction in the camera coordinate system be , the vertical condition is: ; Construction of vertical loss function:

[0071] Where, represents the vertical loss, Indicates vertical object pixels, represents the normal vector, Represents the unit vector of the gravity direction in the camera coordinate system, Represents the vertical object mask obtained from the semantic segmentation map.

[0072] As a further preferred technical solution, the formula of the total loss function is expressed as:

[0073] Where, represents the total loss function, represents the loss of luminosity, represents the semantic segmentation loss, represents the smoothing loss, represents the feature vector alignment loss, represents the vertical loss, 、 、 represents a hyperparameter.

[0074] Specifically, the smoothness loss The calculation formula is:

[0075] Where, and Indicates the depth map x , y Gradient in the axial direction; and Indicates that the image is x , y The gradient along the axis.

[0076] In addition, if Figure 4 As shown, the second embodiment of the present invention further proposes a self-supervised depth estimation network training system, the system comprising: An acquisition module 10 is configured to acquire video frame images as training samples and divide the training samples into target views and source views; a training module 20 for training the target view as an input to a depth estimation network and a semantic segmentation network and the source view as an input to a camera network, to obtain a trained depth estimation network for depth map estimation, wherein the depth estimation network includes a first encoder and a first decoder, and the semantic segmentation network includes a second encoder and a second decoder; Among them, the total loss function used during training includes feature vector alignment loss, vertical loss, photometric loss and semantic segmentation loss. The feature vector alignment loss is used to align the feature maps output by the first encoder and the second encoder in a directional manner. The vertical loss is used to force the depth map output by the first decoder to meet vertical consistency based on the semantic segmentation map output by the second decoder. The photometric loss is used to quantify the difference between the reconstructed view and the target view. The semantic segmentation loss is used to determine the similarity between the semantic segmentation map and the true value.

[0077] As a further preferred technical solution, the first encoder includes a convolution branch and a Transformers branch, the convolution branch includes several convolutional neural networks connected in sequence, the Transformers branch includes several Transformers network blocks connected in sequence, and the number of convolutional neural networks and Transformers network blocks is the same, and the convolution branch and the Transformers branch are connected through a multi-stage adaptive fusion network.

[0078] As a further preferred technical solution, the multi-stage adaptive fusion network includes several levels of fusion modules for connecting the convolutional neural networks and Transformers network blocks of the corresponding levels, and the output of the fusion module of the last level is connected to the first decoder.

[0079] As a further preferred technical solution, the first decoder adopts the HR-Net network.

[0080] As a further preferred technical solution, the camera network adopts a ResNet-18 network.

[0081] As a further preferred technical solution, the structure of the second encoder is the same as that of the first encoder, and the structure of the second decoder is the same as that of the first decoder.

[0082] As a further preferred technical solution, the formula for the feature vector alignment loss is expressed as:

[0083] Where, represents the feature map output by the first encoder, represents the feature map output by the second encoder, Represents a pixel, 、 Represent the height and width of the feature map respectively, represents the feature vector alignment loss.

[0084] As a further preferred technical solution, the feature vector alignment loss is compared with the alignment tolerance to correct the feature vector alignment loss, which is expressed as:

[0085] Where, represents the corrected feature vector alignment loss, represents the feature vector alignment loss before correction, Indicates the alignment tolerance.

[0086] As a further preferred technical solution, the formula for the vertical loss is expressed as:

[0087] Where, represents the vertical loss, Indicates vertical object pixels, represents the normal vector, Represents the unit vector of the gravity direction in the camera coordinate system, Represents the vertical object mask obtained from the semantic segmentation map.

[0088] As a further preferred technical solution, the formula of the total loss function is expressed as:

[0089] Where, represents the total loss function, represents the loss of luminosity, represents the semantic segmentation loss, represents the smoothing loss, represents the feature vector alignment loss, represents the vertical loss, 、 、 represents a hyperparameter.

[0090] It should be noted that other embodiments or specific implementation methods of the self-supervised depth estimation network training system of the present invention can refer to the above-mentioned method embodiments and will not be repeated here.

[0091] In addition, the third embodiment of the present invention further proposes a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the self-supervised depth estimation network training method described in the first embodiment above is implemented.

[0092] It should be noted that the computer-readable medium disclosed in this embodiment may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection having one or more conductors, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. This propagated data signal may take a variety of forms, including, but not limited to, electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wire, optical cable, RF (radio frequency), or any suitable combination thereof.

[0093] The computer-readable medium may be included in the electronic device or may exist independently without being incorporated into the electronic device. The computer-readable medium carries one or more programs. When executed by the electronic device, the one or more programs cause the electronic device to perform the zero-sample image anomaly detection method of the above embodiment.

[0094] Computer program code for carrying out operations of the present disclosure may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer, or entirely on the remote computer or server.

[0095] Where a remote computer is involved, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0096] It should be understood that various components of the present invention may be implemented using hardware, software, firmware, or a combination thereof. In the above-described embodiments, multiple steps or methods may be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof may be used: a discrete logic circuit having logic gate circuits for implementing logic functions on data signals, an application-specific integrated circuit having suitable combinational logic gate circuits, a programmable gate array (PGA), a field-programmable gate array (FPGA), etc.

[0097] Throughout this specification, reference to terms such as "one embodiment," "some embodiments," "examples," "specific examples," or "some examples" means that a specific feature, structure, material, or characteristic described in conjunction with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

[0098] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the number of technical features indicated. Thus, a feature specified as "first" or "second" may explicitly or implicitly include at least one such feature. In the description of the present invention, "plurality" or "several" means at least two, for example, two, three, etc., unless otherwise specifically defined.

[0099] Although the embodiments of the present invention have been shown and described above, it will be understood that the above embodiments are illustrative and are not to be construed as limitations on the present invention. A person skilled in the art may change, modify, replace and modify the above embodiments within the scope of the present invention.

Claims

1. A self-supervised depth estimation network training method, characterized in that: include: Collect video frame images as training samples and divide the training samples into target views and source views; Training the target view as input to a depth estimation network and a semantic segmentation network and the source view as input to a camera network to obtain a trained depth estimation network for depth map estimation, wherein the depth estimation network includes a first encoder and a first decoder, and the semantic segmentation network includes a second encoder and a second decoder; Among them, the total loss function used during training includes feature vector alignment loss, vertical loss, photometric loss and semantic segmentation loss. The feature vector alignment loss is used to align the feature maps output by the first encoder and the second encoder in a directional manner. The vertical loss is used to force the depth map output by the first decoder to meet vertical consistency based on the semantic segmentation map output by the second decoder. The photometric loss is used to quantify the difference between the reconstructed view and the target view. The semantic segmentation loss is used to determine the similarity between the semantic segmentation map and the true value.

2. The self-supervised depth estimation network training method according to claim 1, characterized in that The first encoder includes a convolution branch and a Transformers branch. The convolution branch includes several convolutional neural networks connected in sequence. The Transformers branch includes several Transformers network blocks connected in sequence. The number of convolutional neural networks and Transformers network blocks is the same. The convolution branch and the Transformers branch are connected through a multi-stage adaptive fusion network.

3. The self-supervised depth estimation network training method according to claim 2, wherein: The multi-stage adaptive fusion network includes several layers of fusion modules for connecting the convolutional neural networks and Transformers network blocks of the corresponding layers, and the output of the fusion module of the last layer is connected to the first decoder.

4. The self-supervised depth estimation network training method according to claim 1, wherein The first decoder adopts the HR-Net network.

5. The self-supervised depth estimation network training method according to claim 1, wherein: The camera network adopts the ResNet-18 network.

6. The self-supervised depth estimation network training method according to claim 1, characterized in that The structure of the second encoder is the same as that of the first encoder, and the structure of the second decoder is the same as that of the first decoder.

7. The self-supervised depth estimation network training method according to claim 1, characterized in that The formula of the feature vector alignment loss is expressed as: Where, represents the feature map output by the first encoder, represents the feature map output by the second encoder, Represents a pixel, 、 Represent the height and width of the feature map respectively, represents the feature vector alignment loss.

8. The self-supervised depth estimation network training method according to claim 7, wherein: The method further comprises: The feature vector alignment loss is compared with the alignment tolerance to correct the feature vector alignment loss, which is expressed as: Where, represents the corrected feature vector alignment loss, represents the feature vector alignment loss before correction, Indicates the alignment tolerance.

9. The self-supervised depth estimation network training method according to claim 1, wherein: The formula for the vertical loss is: Where, represents the vertical loss, Indicates vertical object pixels, represents the normal vector, Represents the unit vector of the gravity direction in the camera coordinate system, Represents the vertical object mask obtained from the semantic segmentation map.

10. The self-supervised depth estimation network training method according to claim 1, wherein: The formula for the luminosity loss is: = + Where, , , ,in is a hyperparameter, Represents the target view and refactoring views The structural similarity between || || means calculating pixel-level absolute error.

11. The self-supervised depth estimation network training method according to claim 1, wherein: The semantic segmentation loss adopts the cross entropy loss function.

12. The self-supervised depth estimation network training method according to claim 1, wherein: The formula of the total loss function is expressed as: Where, represents the total loss function, represents the loss of luminosity, represents the semantic segmentation loss, represents the smoothing loss, represents the feature vector alignment loss, represents the vertical loss, 、 、 represents a hyperparameter.

13. A self-supervised depth estimation network training system, characterized in that include: An acquisition module is used to acquire video frame images as training samples and divide the training samples into target views and source views; a training module, configured to train the target view as input to a depth estimation network and a semantic segmentation network and the source view as input to a camera network, thereby obtaining a trained depth estimation network for depth map estimation, wherein the depth estimation network includes a first encoder and a first decoder, and the semantic segmentation network includes a second encoder and a second decoder; Among them, the total loss function used during training includes feature vector alignment loss, vertical loss, photometric loss and semantic segmentation loss. The feature vector alignment loss is used to align the feature maps output by the first encoder and the second encoder in a directional manner. The vertical loss is used to force the depth map output by the first decoder to meet vertical consistency based on the semantic segmentation map output by the second decoder. The photometric loss is used to quantify the difference between the reconstructed view and the target view. The semantic segmentation loss is used to determine the similarity between the semantic segmentation map and the true value.

14. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the self-supervised depth estimation network training method according to any one of claims 1 to 12 is implemented.

Images