A monocular image depth estimation method, device and equipment based on diffusion model and target prompt

Abstract

The present invention provides a method, device, and apparatus for monocular image depth estimation based on a diffusion model and target prompts, and relates to the field of computer vision technology. The method comprises: inputting a sample detection image into a depth estimation model to be trained, processing the sample detection image through a pre-trained image encoder to obtain a shallow spatial representation; processing the sample detection image through a pre-trained target detection network to obtain a target detection result; processing the target detection result through a target prompt module to be trained to obtain target prompt information; inputting the shallow spatial representation and target prompt information into a denoising network to be trained to obtain multi-scale features; inputting the multi-scale features into an interactive decoder to be trained to obtain a sample depth map; and training the depth estimation model to be trained based on the label depth map and the sample depth map to obtain a trained depth estimation model for depth estimation, thereby improving the accuracy of monocular depth estimation based on the diffusion model.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a method, device and equipment for estimating depth of a monocular image based on a diffusion model and target prompts. Background Art

[0002] Depth estimation involves analyzing a single or multiple images to estimate the distance, or depth, of each pixel in a scene to the camera's imaging plane. In the field of computer vision, depth estimation is considered a crucial research direction, widely used in cutting-edge technologies such as autonomous driving, 3D scene reconstruction, and augmented reality. Accurate depth information not only helps systems better understand and interpret the surrounding environment but also supports more complex visual tasks, playing a crucial role in scenarios involving spatial perception and interaction.

[0003] Depth estimation methods can be categorized into monocular and multi-view depth estimation based on the number of images and viewpoints. In multi-view depth estimation, a common technique is to infer depth information by calculating the disparity between different viewpoints (i.e., the difference in the position of the same object in different images). For example, humans judge the distance of an object by comparing the disparity between left and right eye images. Similarly, binocular cameras use this principle for depth estimation. However, multi-view cameras are typically expensive and have drawbacks such as high cost, high sensitivity to the detection environment, and poor universality. In contrast, monocular cameras are not only inexpensive, compact, and easy to use, but also contain rich scene information in monocular images. Therefore, extracting depth information from monocular images has become a key research topic with significant academic and application value.

[0004] Early researchers mainly relied on geometric theories or manually designed features (such as focus and defocus, image vanishing points, and shadows) to extract depth information from monocular images. However, these methods have been difficult to achieve breakthroughs at the theoretical level. As research has deepened, scholars have gradually turned to using deep learning methods to solve the problem of monocular depth estimation. The introduction of deep learning enables researchers to automatically extract complex features from large amounts of data through end-to-end learning, significantly improving the performance of monocular depth estimation while also enhancing the model's adaptability to different environments and conditions. In recent years, some of the latest methods have attempted to use diffusion models to achieve monocular depth estimation tasks. Thanks to the use of large-scale datasets for training, these models have demonstrated strong generalization capabilities and can significantly improve the accuracy of predicted depth maps.

[0005] However, in the current monocular depth estimation method based on the diffusion model, the diffusion model is usually prompted only by scene category information. Due to the diversity of visual scenes, the scene category information alone cannot fully capture the detailed features of the scene, resulting in the generated prompt information lacking pertinence and accuracy. For example, in an autonomous driving scenario, relying solely on the category information of "urban roads" cannot accurately describe specific traffic conditions, weather conditions, or road structures and other details, thereby limiting the model's understanding of complex scenes and the accuracy of depth estimation. In addition, since scene category information is used as a text description of the image, the scene label or description information corresponding to each image must be obtained in advance during the training process. This requirement limits the application of the model in the absence of scene annotation data or incomplete scene information, resulting in higher requirements for the training data set. Therefore, how to improve the accuracy of monocular depth estimation based on the diffusion model is a technical problem that needs to be solved urgently in the present invention. Summary of the Invention

[0006] Based on the above technical problems, the present invention provides a method, device and equipment for monocular image depth estimation based on a diffusion model and target prompts, aiming to improve the accuracy of monocular depth estimation based on the diffusion model.

[0007] A first aspect of the present invention provides a method for monocular image depth estimation based on a diffusion model and target cues, the method comprising:

[0008] Inputting the sample detection image into a depth estimation model to be trained, wherein the depth estimation model to be trained includes at least: a pre-trained image encoder, a pre-trained object detection network, a target prompt module to be trained, a denoising network to be trained, and an interactive decoder to be trained;

[0009] Processing the sample detection image by the pre-trained image encoder to obtain a shallow spatial representation corresponding to the sample detection image;

[0010] Processing the sample detection image through the pre-trained target detection network to extract the target detection result of the sample detection image;

[0011] Processing the target detection result by the target prompt module to be trained to obtain target prompt information corresponding to the sample detection image;

[0012] Inputting the shallow spatial representation and the target prompt information into the denoising network to be trained to extract multi-scale features;

[0013] Inputting the multi-scale features into the interactive decoder to be trained to obtain a sample depth map;

[0014] Based on the label depth map corresponding to the sample detection image and the sample depth map, the target prompt module to be trained, the denoising network to be trained, and the interactive decoder to be trained are trained until a trained depth estimation model is obtained;

[0015] The image to be detected is input into the trained depth estimation model to obtain a depth map corresponding to the image to be detected.

[0016] A second aspect of the present invention provides a monocular image depth estimation device based on a diffusion model and target cues, the device comprising:

[0017] An image input module, configured to input a sample detection image into a depth estimation model to be trained, wherein the depth estimation model to be trained comprises at least: a pre-trained image encoder, a pre-trained target detection network, a target prompt module to be trained, a denoising network to be trained, and an interactive decoder to be trained;

[0018] An image encoding module, configured to process the sample detection image using the pre-trained image encoder to obtain a shallow spatial representation corresponding to the sample detection image;

[0019] An object detection module is used to process the sample detection image through the pre-trained object detection network to extract the object detection result of the sample detection image;

[0020] a prompt acquisition module, configured to process the target detection result through the target prompt module to be trained to obtain target prompt information corresponding to the sample detection image;

[0021] A multi-scale feature extraction module, configured to input the shallow spatial representation and the target prompt information into the denoising network to be trained to extract multi-scale features;

[0022] A depth map output module, configured to input the multi-scale features into the interactive decoder to be trained to obtain a sample depth map;

[0023] A model training module is used to train the target prompt module to be trained, the denoising network to be trained, and the interactive decoder to be trained based on the label depth map corresponding to the sample detection image and the sample depth map, until a trained depth estimation model is obtained;

[0024] The depth estimation module is used to input the image to be detected into the trained depth estimation model to obtain a depth map corresponding to the image to be detected.

[0025] A third aspect of the present invention provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the computer program is executed by the processor, the method for monocular image depth estimation based on a diffusion model and target cues according to the first aspect of the embodiment of the present invention is implemented.

[0026] In the monocular image depth estimation method based on a diffusion model and target cues provided by the present invention, it is proposed to use objects in the scene as cues to guide the diffusion model in the monocular depth estimation task: a pre-trained target detection network is used to perform target detection on a sample detection image, and the objects (i.e., targets) present in the sample detection image are extracted to obtain target detection results. The target detection results are then processed based on the target cue module proposed in the present invention to generate corresponding target cue information. The target cue information generated by this method is closely integrated with the shallow spatial representation (image content) corresponding to the sample detection image extracted by the pre-trained image encoder, providing richer and more diverse details, enhancing the depth estimation model's ability to understand complex scenes and the accuracy of depth estimation, while also avoiding the monotony and redundancy of template-based cues. In addition, because the target cue information generated by the present invention relies only on the input sample detection image itself, no additional scene annotation data is required. This overcomes the limitation of the model's inability to be applied in the absence of scene annotation data or incomplete scene information, and avoids the defect of high training dataset requirements. Based on this, the monocular image depth estimation method based on a diffusion model and target cues proposed by the present invention has stronger versatility and adaptability and can be widely applied to various common monocular depth estimation datasets. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments of the present invention. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0028] Figure 1 is an average attention map between prompt words and image features during the diffusion process shown in one embodiment of the present invention;

[0029] Figure 2 This is a flowchart of a method for monocular image depth estimation based on a diffusion model and target cues according to an embodiment of the present invention;

[0030] Figure 3 1 is an overall schematic diagram of performing model training on a depth estimation model to be trained, shown in one embodiment of the present invention;

[0031] Figure 4 is a structural diagram of a target prompt module according to an embodiment of the present invention;

[0032] Figure 5 is a schematic structural diagram of an interactive decoder according to an embodiment of the present invention;

[0033] Figure 6 This is a flow chart of a method for monocular image depth estimation based on a diffusion model and target cues, according to one embodiment of the present invention;

[0034] Figure 7 This is a training and reasoning flow chart of a monocular depth estimation system according to an embodiment of the present invention;

[0035] Figure 8 This is a training and reasoning flowchart of an autonomous driving obstacle detection system according to an embodiment of the present invention;

[0036] Figure 9 is a visualization diagram of prediction results of different methods shown in one embodiment of the present invention;

[0037] Figure 10 is a visualization diagram showing the impact of detected targets on the depth prediction results according to an embodiment of the present invention;

[0038] Figure 11 This is a structural block diagram of a monocular image depth estimation device based on a diffusion model and target prompts provided by one embodiment of the present invention. DETAILED DESCRIPTION

[0039] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0040] In order to address the various shortcomings of using scene information as text description of images in current monocular depth estimation methods based on diffusion models, the present invention observes the average attention map of the diffusion model denoising process and finds that the diffusion model has obvious perception of the target words in the text prompt and can generate corresponding attention maps. Figure 1 As shown, Figure 1 is an average attention map between prompt words and image features during the diffusion process shown in one embodiment of the present invention. Figure 1It can be seen that the average attention maps corresponding to the two prompt words "cat" and "burger" show their respective corresponding appearance forms. Based on this, the present invention takes the approach of extracting prompt information directly from the input image to provide fine-grained prompt information, and proposes to use objects in the scene as prompt information to guide the diffusion model to perform monocular depth estimation tasks. The prompts generated by this method are closely integrated with the image content, providing richer and more diverse details, thereby improving the depth estimation model's ability to understand complex scenes and the accuracy of depth estimation, while avoiding the monotony and redundancy of template-based prompts. And because the prompt information generated by the present invention only depends on the input image itself and does not require additional scene annotation data, it has stronger versatility and adaptability, and can be widely used in various common monocular depth estimation datasets.

[0041] Please refer to Figure 2 , Figure 2 FIG. 1 is a flowchart showing a method for monocular image depth estimation based on a diffusion model and target hints according to an embodiment of the present invention. Figure 2 As shown, the monocular image depth estimation method based on diffusion model and target hint provided in this embodiment includes at least the following steps:

[0042] Step S11: inputting the sample detection image into the depth estimation model to be trained, wherein the depth estimation model to be trained includes at least: a pre-trained image encoder, a pre-trained target detection network, a target prompt module to be trained, a denoising network to be trained and an interactive decoder to be trained.

[0043] In this embodiment, the sample detection image is first input into the depth estimation model to be trained. The depth estimation model to be trained includes at least: a pre-trained image encoder, a pre-trained target detection network, a target prompt module to be trained, a denoising network to be trained, and an interactive decoder to be trained. In this embodiment, a sample data set is prepared for training the depth estimation model. The sample data set includes: a sample detection image and a labeled depth map corresponding to the sample detection image. The labeled depth map is the label corresponding to the sample detection image, which is the real depth map corresponding to the sample detection image. The sample detection image is an image sample to be used for depth estimation training of the depth estimation model.

[0044] Step S12: Processing the sample detection image through the pre-trained image encoder to obtain a shallow spatial representation corresponding to the sample detection image.

[0045] In this embodiment, after the sample detection image is input into the depth estimation model to be trained, the sample detection image can be input into a pre-trained image encoder and a pre-trained target detection network respectively, and the sample detection image is processed accordingly by the pre-trained image encoder and the pre-trained target detection network respectively. Among them, the sample detection image can be processed in parallel by the image encoder and the target detection network. After the sample detection image is input into the pre-trained image encoder, the sample detection image can be encoded by the pre-trained image encoder to obtain a shallow spatial representation corresponding to the sample detection image.

[0046] Among them, the latent space is represented as the feature representation corresponding to the shallow space, and the shallow space (Latent Space) refers to the low-dimensional representation space of the data after being encoded by the neural network. In this space, the high-dimensional features of the data are compressed into a lower-dimensional abstract representation, which is convenient for processing and generation tasks. The pre-trained image encoder of this embodiment is a pre-trained image encoder. During the training process of the depth estimation model, the model parameters of the pre-trained image encoder are fixed. In an optional embodiment, the image encoder can be a variational autoencoder (VAE), such as a vectorized variational autoencoder (VQ-VAE) and the like. Among them, the variational autoencoder is a generative model that introduces the idea of ​​variational inference to enable the model to learn the potential distribution of data and generate new samples. VAE maps data to the latent space through the encoder, and then reconstructs the data from the latent space through the decoder.

[0047] Step S13: Processing the sample detection image through the pre-trained object detection network to extract the object detection result of the sample detection image.

[0048] In this embodiment, after the sample detection image is input into the pre-trained target detection network, the pre-trained target detection network can be used to perform target detection on the sample detection image, that is, to detect each object in the image and extract the target detection result of the sample detection image. In this embodiment, the target can be understood as various objects in the image, such as a table, a bench, a computer, etc., without limitation. The pre-trained target detection network in this embodiment is a pre-trained target detection network, and during the training process of the depth estimation model, the model parameters of the pre-trained target detection network are fixed.

[0049] Because the target hint information in this embodiment is used to assist the depth estimation model in depth prediction, it does not rely on a specific detector and can be used with various types of detectors to extract target detection results (target information in the scene) from the sample detection image. In an optional embodiment, the target detection network can be a DETR or YOLO series detector, which is not limited to this.

[0050] Step S14: Processing the target detection result by the target prompt module to be trained to obtain target prompt information corresponding to the sample detection image.

[0051] In this embodiment, after the target detection network is used to extract the objects present in the sample detection image, that is, after the target detection network extracts the target detection results of the sample detection image, the target detection results are processed by the target prompt module to be trained proposed in this embodiment, and the target detection results (that is, the object information in the scene corresponding to the sample detection image) are encoded to obtain the target prompt information corresponding to the sample detection image as a prompt for the diffusion model.

[0052] In this embodiment, the target prompt information output by the target prompt module includes the relative spatial relationship between objects in the sample detection image. According to the principle of perspective, the size of similar objects in an image is negatively correlated with their corresponding depth. For example, in the real world, two objects of similar size appear larger in the image if they are closer to the camera, while objects farther away appear smaller. Based on this, this embodiment helps generate a more accurate depth map by learning the relative spatial relationship between objects in the scene.

[0053] Step S15: inputting the shallow spatial representation and the target prompt information into the denoising network to be trained to extract multi-scale features.

[0054] The diffusion model used in this embodiment only includes the denoising network within the diffusion model and does not involve the noise addition portion of the diffusion model. After obtaining the shallow spatial representation and target cue information corresponding to the sample detection image, this embodiment can input these shallow spatial representations and target cue information into the denoising network to be trained. The trained denoising network then extracts the multi-scale features corresponding to the sample detection image.

[0055] Step S16: inputting the multi-scale features into the interactive decoder to be trained to obtain a sample depth map.

[0056] In this embodiment, after obtaining multi-scale features, the multi-scale features are input into the interactive decoder to be trained. The interactive decoder guides the fusion of features at different scales within the multi-scale features, resulting in a sample depth map output by the interactive decoder to be trained. The sample depth map is the depth map obtained by the depth estimation model to be trained performing depth estimation on the sample detection image during the model training process.

[0057] Step S17: Based on the label depth map corresponding to the sample detection image and the sample depth map, the target prompt module to be trained, the denoising network to be trained, and the interactive decoder to be trained are trained until a trained depth estimation model is obtained.

[0058] In this embodiment, the target prompt module to be trained, the denoising network to be trained and the interaction decoder to be trained can be trained based on the label depth map corresponding to the sample detection image and the sample depth map corresponding to the sample detection image until a trained target prompt module, a trained denoising network and a trained interaction decoder are obtained, thereby obtaining a trained depth estimation model consisting of a pre-trained image encoder, a pre-trained target detection network, a trained target prompt module, a trained denoising network and a trained interaction decoder.

[0059] In an optional embodiment, the loss between the label depth map and the sample depth map can be calculated (for example, the scale-invariant loss is calculated, etc.), and the model parameters of the target prompt module to be trained, the denoising network to be trained, and the interaction decoder to be trained are updated based on the calculated loss until a trained depth estimation model is obtained.

[0060] Step S18: inputting the image to be detected into the trained depth estimation model to obtain a depth map corresponding to the image to be detected.

[0061] In this embodiment, a trained depth estimation model is used to perform depth estimation on a monocular image. During the application of the trained depth estimation model, an image to be detected can be input into the trained depth estimation model to obtain a depth map corresponding to the image to be detected output by the trained depth estimation model. The image to be detected is the image for which depth estimation is required, and the depth map is the result of depth estimation for the image to be detected, including the depth value corresponding to each pixel in the image to be detected.

[0062] In this embodiment, it is proposed to use objects in the scene as cues to guide the diffusion model in the monocular depth estimation task: a pre-trained object detection network performs object detection on a sample detection image, extracts objects (i.e., targets) present in the sample detection image, and obtains a target detection result. The target detection result is then processed based on the target cue module proposed in the present invention to generate corresponding target cue information. The target cue information generated by this method is closely integrated with the shallow spatial representation (image content) corresponding to the sample detection image extracted by the pre-trained image encoder, providing richer and more diverse details, enhancing the depth estimation model's ability to understand complex scenes and the accuracy of depth estimation, while also avoiding the monotony and redundancy of template-based cues. In addition, because the target cue information generated by the present invention relies solely on the input sample detection image itself, no additional scene annotation data is required. This overcomes the limitation of the model's inability to be applied in situations where scene annotation data is lacking or scene information is incomplete, and avoids the drawback of high training dataset requirements. Based on this, the monocular image depth estimation method based on the diffusion model and target cue proposed in the present invention has greater versatility and adaptability and can be widely applied to various common monocular depth estimation datasets.

[0063] In one embodiment, if Figure 3 As shown, Figure 3 FIG. 1 is an overall schematic diagram of a depth estimation model to be trained according to an embodiment of the present invention. Figure 3 In

[15] , the depth estimation model to be trained includes at least: a pre-trained image encoder (VQVAE), a pre-trained object detection network (Detector), an object prompt module (OPM) to be trained, a denoising network (Denoising UNet) to be trained, and a interactive decoder (GLID) to be trained. The network parameters of VQVAE and Detector are fixed, while the network parameters of OPM, Denoising UNet, and GLID are updateable.

[0064] exist Figure 3In this paper, the input image x (Input) is first compressed to 1 / 8 of its original size by the Vectorized Variational Autoencoder (VQ-VAE), obtaining the corresponding latent space features Z. Simultaneously, an object detection network (Detector) is used to extract objects appearing in the input image, and an object prompt module (OPM) is used to convert the object detection results into object prompts C. A single-step denoising process is then performed in the denoising network. A cross-attention mechanism is used to interact the latent space representation Z with the object prompts C, and multi-scale features F are extracted from them. Finally, the global-local interaction decoder (GLID) proposed in this embodiment is used to guide the fusion of features at different scales and generate a final depth map (Pred). A scale-invariant loss is then calculated based on the final depth map (Pred) and the labeled depth map (GT), thereby training the trained depth estimation model.

[0065] In combination with the above embodiments, in one embodiment, the present invention further provides a monocular image depth estimation method based on a diffusion model and target cues. In this method, the target detection result includes: object category label information and bounding box information; the above step S14 may specifically include steps S21 to S23:

[0066] Step S21: In the target prompt module to be trained, the category label information is processed to obtain label prompt information corresponding to the sample detection image.

[0067] In this embodiment, in the target prompt module to be trained, the target detection results are divided into two parts: labels and bounding boxes, and the detection results corresponding to these two parts are encoded as prompt information. The category label information in the target detection results can be processed to obtain the label prompt information corresponding to the sample detection image.

[0068] Step S22: Process the bounding box information to obtain position prompt information and size prompt information corresponding to the sample detection image.

[0069] In this embodiment, in the target prompt module to be trained, the bounding box information in the target detection result can also be processed to obtain the position prompt information and size prompt information corresponding to the sample detection image.

[0070] Step S23: connecting the label prompt information, the position prompt information and the size prompt information, and obtaining the target prompt information through convolution fusion.

[0071] In this embodiment, in the target prompt module to be trained, after obtaining the label prompt information, the position prompt information and the size prompt information, the label prompt information, the position prompt information and the size prompt information can be connected, and the target prompt information can be obtained and output through convolution fusion.

[0072] In combination with the above embodiments, in one embodiment, the present invention further provides a monocular image depth estimation method based on a diffusion model and target prompts. In this method, the "processing the category label information to obtain the label prompt information corresponding to the sample detection image" in the above step S21 can specifically include steps S31 and S32:

[0073] Step S31: Based on the category label information, a target label feature vector corresponding to the category label information is retrieved from a label prompt set.

[0074] In this embodiment, after obtaining the category label information of the target detection result, the target label feature vector corresponding to the category label information can be retrieved from the label hint set based on the category label information. The label hint set includes: multiple label feature vectors, each corresponding to a different object label information; the label feature vector is a text feature obtained by encoding the object label information in the training dataset corresponding to the target detection network.

[0075] Since labels play a vital role in guiding the model to identify and understand objects in the scene, this embodiment can use a pre-trained text encoder (CLIP, Contrastive Language-Image Pretraining) to separately encode each label (i.e., each object label information) in the training data set corresponding to the target detection network. Will be converted into its corresponding text features , thus obtaining the label feature vector corresponding to each object label information. After all object label information is converted into label feature vectors, these label feature vectors can be merged together to obtain the label prompt set . It can be expressed by the following formula:

[0076]

[0077] in, represents the i-th label in the training dataset, and n represents the total number of labels.

[0078] CLIP is a model trained using contrastive learning that maps images and text into the same latent space. By jointly training image and text data, it enables the model to understand the relationship between text and images and is widely used in tasks such as image generation, search, and classification.

[0079] During model training (similar to model inference), the category label information provided by the target detection network can be used to , from the tag prompt collection Retrieve the category label information from The corresponding target label feature vector.

[0080] Step S32: obtaining the label prompt information based on the target label feature vector.

[0081] In this embodiment, label prompt information can be obtained based on the retrieved target label feature vector. Specifically, since there may be multiple valid targets (multiple objects) in the image to be detected, the corresponding target label feature vectors can be retrieved from the label prompt set based on multiple category label information, and then the retrieved multiple target label feature vectors are spliced ​​to obtain the label prompt information corresponding to the image to be detected. .

[0082] In this embodiment, the label prompt information It is learnable and its weights are continuously updated throughout the training process, so that the label hints become more refined and contextual as the model trains.

[0083] In combination with the above embodiments, in one implementation, the present invention further provides a monocular image depth estimation method based on a diffusion model and target cues. In this embodiment, the bounding box information includes: position information and size information corresponding to the object; and the above step S22 may specifically include steps S41 to S44:

[0084] Step S41: determining a target cell index based on the position information, and retrieving a target position feature vector corresponding to the position information from a position prompt set based on the target cell index.

[0085] In this embodiment, the bounding box information is used to provide the depth estimation model with the position and size information of each object, thereby achieving a rough construction of the scene. In this embodiment, the target cell index can be first determined based on the position information in the bounding box information, and then the target position feature vector corresponding to the position information can be retrieved from the position hint set based on the target cell index. The position hint set includes: multiple position feature vectors, each of which corresponds to a different cell index; the position feature vector is the position feature corresponding to a cell after the sample detection image is divided into multiple cells.

[0086] In this embodiment, the image to be detected can be divided into multiple cells in advance, such as into an n×n network, and a position hint set of the same size can be constructed. , where each cell corresponds to a position feature vector, and each cell corresponds to a cell index.

[0087] In an optional example, the target center point corresponding to the position information can be calculated based on the position information, and then the target cell index where the target center point is located can be determined based on the target center point. Finally, based on the target cell index, the target cell index can be obtained from the position hint set. The target position feature vector corresponding to the position information is retrieved.

[0088] Step S42: obtaining the position prompt information based on the target position feature vector.

[0089] In this embodiment, the position prompt information can be obtained based on the retrieved target position feature vector, and the position prompt information is used to obtain the relative position information of the target in the image to be detected. Specifically, since there may be multiple valid targets (multiple objects) in the image to be detected, the target cell index can be determined based on multiple position information, and the corresponding target position feature vectors can be retrieved from the position prompt set. Then, the retrieved multiple target position feature vectors are spliced ​​to obtain the position prompt information corresponding to the image to be detected. .

[0090] In this embodiment, in order to Keep consistent, location prompt information Also uses learnable features, i.e. position hint information It is learnable and its weights are continuously updated throughout the training process, so that the location cues become more refined and contextual as the model trains.

[0091] In an optional example, the bounding box information obtained from the object detection network The position information is expressed in the form of xyxy, which represents the coordinates of the upper left corner and the lower right corner of the object. It is worth noting that the coordinates of the position information used in this embodiment are in normalized form, that is, the coordinate values ​​are within the range of (0,1).

[0092] Step S43: determining a target size index based on the size information, and retrieving a target size feature vector corresponding to the size information from a size prompt set based on the target size index.

[0093] In this embodiment, the object size index can be determined based on the size information in the bounding box information. Then, based on the target size index, the target size feature vector corresponding to the size information can be retrieved from a size hint set. The size hint set includes: multiple size feature vectors, each corresponding to a different size index; a size feature vector is a size feature corresponding to a level after the height and width of the object are divided into multiple levels.

[0094] In this embodiment, similar to the position hint, the height and width of the object (i.e., the object) in the image to be detected can be divided into multiple levels in advance, such as n levels, and a size hint set of shape n×n is constructed. , each level corresponds to a size feature vector, where each size feature vector corresponds to a different size index.

[0095] In an optional example, the target size index corresponding to the size information can be calculated based on the height information and width information in the size information, and finally, based on the target size index, the target size index can be obtained from the size hint set. The target size feature vector corresponding to the size information is retrieved from .

[0096] Step S44: obtaining the size prompt information based on the target size feature vector.

[0097] In this embodiment, size hint information can be obtained based on the retrieved target size feature vector. Specifically, since there may be multiple valid targets (multiple objects) in the image to be detected, the target size index can be determined based on multiple size information, and the corresponding target size feature vectors can be retrieved from the size hint set. The retrieved multiple target size feature vectors are then spliced ​​to obtain the size hint information corresponding to the image to be detected. .

[0098] In this embodiment, in order to Keep consistent, size prompt information Also uses learnable features, i.e. size hint information is learnable, and its weights are continuously updated throughout the training process, so that the size cues become more refined and contextual as the model trains.

[0099] In an optional embodiment, the target height information can be used according to the following formula: and width information Calculate the corresponding target size index . You can then select the size hint from the collection Retrieve the corresponding target size feature vector from:

[0100] ;

[0101] Here, n represents the level corresponding to the height and / or width.

[0102] In an optional embodiment, after obtaining the label prompt information , location prompt information and size information Afterwards, these three types of prompt information can be connected through the following formula and the target prompt information C can be obtained through convolution fusion:

[0103] .

[0104] In one embodiment, if Figure 4 As shown, Figure 4 FIG. 1 is a schematic diagram showing the structure of a target prompt module according to an embodiment of the present invention. Figure 4 In the process, the target detection network (Detector) is used to infer the input image (image to be detected) to obtain the label information of the objects appearing in the image to be detected. and bounding box information Afterwards, the object prompt module (OPM) is divided into two parts: label and bounding box, to encode the corresponding detection results into prompt information.

[0105] This method first uses a text encoder (pre-trained CLIP) to encode each label (AllLabels) in the sample dataset separately. Will be converted into its corresponding text features After all labels are converted into text features, this method merges these features together to form the initial label prompt set This process corresponds to Figure 4 The initialize step in .

[0106] During the training and reasoning process, the target prompt module uses the label information provided by the target detection network (Detector) , from the tag prompt collection Retrieve the corresponding feature vector from Figure 4 Since there may be multiple valid targets in the input image, the retrieved feature vectors are concatenated together to form the label prompt information of the current input image. .

[0107] exist Figure 4 For location hints, the image to be detected is first divided into an n×n grid, and a set of object location hints of the same size is constructed. , where each cell corresponds to a location feature. Next, the location information (x, y) in the bounding box information provided by the target detection network (Detector) is used to calculate the cell index where the target center point is located. The corresponding position hint is retrieved from the image to provide the relative position information of the target in the image. Since there may be multiple valid targets in the input image, the retrieved position hints are spliced ​​together to form the position hint information of the current input image. .

[0108] exist Figure 4 Similar to position hints, for size hints, we first divide the height and width of the object into n levels and create a target size hint set with a shape of n×n Then, we can use the size information (height and width information of the target) in the bounding box information provided by the target detection network (Detector) to calculate the index of the corresponding size hint. Since there may be multiple valid objects in the input image, the retrieved size hints are concatenated together to form the size hint information of the current input image. .

[0109] Finally, these three prompts are connected and fused through convolution (Concat-Conv) to form the final target prompt information (Prompts).

[0110] In combination with the above embodiments, in one embodiment, the present invention further provides a monocular image depth estimation method based on a diffusion model and target cues. In this embodiment, the above step S15 may specifically include steps S51 and S52:

[0111] Step S51: In the denoising network to be trained, single-step denoising is performed on the shallow spatial representation and the target prompt information.

[0112] Step S52: The shallow spatial representation and the target prompt information are interacted through a cross-attention mechanism to extract the multi-scale features.

[0113] In this embodiment, after the shallow spatial representation and target prompt information are input into the denoising network to be trained, a single-step denoising process is performed on the shallow spatial representation and target prompt information in the denoising network to be trained, and the shallow spatial representation and target prompt information are interacted through the cross-attention mechanism to extract multi-scale features, thereby avoiding the randomness generated by the monocular depth estimation method based on the diffusion model in the related art, which usually generates a depth image by gradually denoising from random noise.

[0114] In this embodiment, compared with the traditional diffusion model that gradually restores the depth map from noise, on the one hand, the prediction efficiency of depth estimation is improved by performing single-step denoising. On the other hand, the target detection network of this embodiment is used to extract the bounding box and label of the object, and the target prompt module is used to generate the target prompt. Compared with the related art that generates images based on subtitles to describe the image, its model parameters are smaller and the prompt information generation efficiency is higher, thereby further improving the prediction efficiency of depth estimation, overcoming the problem of high time cost of the reasoning process due to the iterative denoising characteristics inherent in the diffusion model itself in the related art, and improving the stability and reasoning efficiency of the monocular depth estimation method based on the diffusion model.

[0115] Currently, related methods lack the ability to fuse multi-scale features and restore depth information, resulting in low feature utilization. Specifically, when processing multi-scale features, related methods typically directly concatenate features of different scales. This simple feature fusion approach fails to effectively capture the complementarity between multi-scale features, limiting prediction accuracy. These related technologies fail to consider the differences in spatial resolution and semantic information between features at different scales.

[0116] To solve this problem, in combination with the above embodiments, in one embodiment, the present invention further provides a method for monocular image depth estimation based on a diffusion model and target cues. In this method, the multi-scale features include at least: a first resolution feature, a second resolution feature, a third resolution feature, and a fourth resolution feature with increasing resolution; and the above step S16 may specifically include steps S61 to S63:

[0117] Step S61: taking the first resolution feature as a global feature and the second resolution feature as a local feature, performing interaction between the global feature and the local feature to obtain a first interactive fusion feature.

[0118] In this embodiment, a global-local interaction decoder (i.e., interaction decoder) is constructed to fully utilize the information between features at different scales. This decoder aims to promote interaction between multi-scale features and effectively fuse them. In this embodiment, global features are defined as those with lower resolution and contain more semantic information, while local features are defined as those with higher resolution and contain richer details.

[0119] First, the first resolution feature in the multi-scale feature is used as the global feature, and the second resolution feature in the multi-scale feature is used as the local feature. The global feature and the local feature are interacted to obtain a first interactive fusion feature.

[0120] Step S62: using the first interactive fusion feature as the global feature and the third resolution feature as the local feature, performing interaction between the global feature and the local feature to obtain a second interactive fusion feature.

[0121] In this embodiment, after obtaining the first interactive fusion feature, the first interactive fusion feature is used as the global feature, and the third resolution feature in the multi-scale feature is used as the local feature. The global feature and the local feature are interacted again to obtain the second interactive fusion feature.

[0122] Step S63: using the second interactive fusion feature as the global feature, using the fourth resolution feature as the local feature, and performing interaction between the global feature and the local feature to obtain the sample depth map.

[0123] In this embodiment, after obtaining the second interactive fusion feature, the second interactive fusion feature is used as the global feature, and the fourth resolution feature in the multi-scale feature is used as the local feature. The global feature and the local feature are interacted again to obtain the sample depth map output by the interactive decoder.

[0124] In some other optional embodiments, the multi-scale features may include more or fewer size features, such as size features with two different resolutions, or size features with 6, 8, or more different resolutions. In this case, the steps are similar to those for size features with four different resolutions: first, the two with the smallest resolutions are used to interact with each other through global features and local features to obtain fused features, and then the fused features are used as global features, and the features with the third smallest resolution are used as local features to continue to interact with each other through global features and local features to obtain fused features, and so on, until the features with the largest resolution are used as local features, and the final fused features obtained from all previous resolution features are used as global features to interact with each other through global features and local features to obtain an output sample depth map. The operation of interacting between global features and local features is the same each time.

[0125] In this embodiment, the proposed interactive decoder utilizes the complementary properties of global features and local features to promote the fusion of features of different types while retaining detailed information, and ultimately enhances the model's overall understanding of the scene.

[0126] In conjunction with the above embodiments, in one embodiment, the present invention further provides a method for monocular image depth estimation based on a diffusion model and target cues. In this method, the "interaction between the global features and the local features" in the above steps S61, S62, or S63 may specifically include steps S71 to S75:

[0127] Step S71: extract a channel attention map from the global feature, and multiply the channel attention map by the local feature element by element to obtain a first fusion feature.

[0128] In this embodiment, after determining the local features and the global features, the channel attention map is first extracted from the global features, and the obtained channel attention map is multiplied element-by-element with the local features to obtain the first fusion feature, thereby realizing the extraction of the channel attention map from the global features to provide semantic guidance for the local features.

[0129] Step S72: extract a spatial attention map from the local features, and multiply the spatial attention map by the global features element by element to obtain a second fusion feature.

[0130] In this embodiment, after determining the local features and the global features, a spatial attention map can be extracted from the local features, and the obtained spatial attention map is multiplied element-by-element with the global features to obtain a second fusion feature, thereby realizing the extraction of the spatial attention map from the local features to provide detailed guidance for the global features.

[0131] It should be noted that this embodiment does not limit the execution order between step S71 and step S72. For example, step S71 can be executed first and then step S72, or step S72 can be executed first and then step S71, or step S71 and step S72 can be executed simultaneously.

[0132] Step S73: Add the first fused feature and the local feature element by element to obtain a third fused feature.

[0133] In this embodiment, after obtaining the first fusion feature, the first fusion feature and the local feature may be added element by element to obtain a third fusion feature.

[0134] Step S74: Add the second fused feature and the global feature element by element to obtain a fourth fused feature.

[0135] In this embodiment, after obtaining the second fused feature, the second fused feature and the global feature may be added element by element to obtain a fourth fused feature.

[0136] Step S75: performing splicing and fusing on the fourth fusion feature and the third fusion feature.

[0137] In this embodiment, after obtaining the third fused feature and the fourth fused feature, the fourth fused feature and the third fused feature may be concatenated and fused. For example, the third fused feature and the fourth fused feature may be concatenated (concat) and then convolutionally fused (Conv-BN-Norm) to obtain interactive fused features, such as the first interactive fused feature, the second interactive fused feature, or the sample depth map.

[0138] In this embodiment, a dedicated decoder module (interactive decoder) is designed based on the characteristics of the monocular depth estimation task. By using different types of attention mechanisms, this module fully considers the characteristics of different feature types and generates a customized attention map for each feature, thereby achieving efficient fusion of multi-scale features and accurate reconstruction of depth information.

[0139] In a specific example, interactive fusion features It can be obtained by the following formula:

[0140] ;

[0141] in, is a global feature, is the channel attention map, is the spatial attention map, is a local feature, Concat is splicing, Conv is convolution fusion, is the fourth fusion feature, It is the third fusion feature.

[0142] In an optional embodiment, before calculating the channel attention map, the global feature is first upsampled by a factor of 2 to align the sizes of the two features.

[0143] In an optional embodiment, the channel attention map is calculated as follows:

[0144] ;

[0145] in, is a global feature, is average pooling, Conv is convolution, is the activation function, where Before performing average pooling, Perform 2x upsampling.

[0146] In another alternative embodiment, the spatial attention map is calculated as follows:

[0147] ;

[0148] To expand the receptive field, this embodiment uses the large kernel attention mechanism (LKA) to extract the spatial attention map. LKA is a lightweight attention mechanism that decomposes a convolution kernel with a large number of parameters into a depthwise convolution (DW-Conv), a depthwise dilation convolution (DW-D-Conv), and a 1×1 convolution (1×1 Conv). This reduces the overall number of parameters while maintaining the receptive field. is a local feature, is the activation function.

[0149] In one embodiment, if Figure 5 As shown, Figure 5 FIG. 1 is a schematic diagram of the structure of an interactive decoder according to an embodiment of the present invention. Figure 5 Specifically, first we start from the global features Extract channel attention map Local features Provide semantic guidance while learning from local features Extracting spatial attention maps Global features Provide detailed guidance, and finally concatenate the interactive features (Concat) and then perform convolution fusion (Conv-BN-Norm) to obtain interactive fusion features .

[0150] In one embodiment, if Figure 6 As shown, Figure 6 This is a flow chart showing a method for monocular image depth estimation based on a diffusion model and target hints according to an embodiment of the present invention. Figure 6In the trained depth estimation model, the input image is loaded and fed into the trained depth estimation model. The target detector is used to extract the object category and detection box information in the image. Then, the category and detection box are encoded into text prompts (i.e., target prompt information) based on the target prompt module. In addition, the input image is mapped to the shallow space using the image encoder to obtain a shallow space representation. Then, the shallow space representation and the text prompt are denoised through the denoising network and multi-scale features are extracted. Finally, the interactive decoder is used to restore the depth information of the multi-scale features and predict the depth map.

[0151] In conjunction with the above embodiments, in one embodiment, the present invention further provides a method for monocular image depth estimation based on a diffusion model and target cues. In this method, the image to be detected is an image captured in real time by a vehicle-mounted camera, and the depth map includes the depth value of each pixel in the image; and after the above step S18, steps S81 and S82 may also be included:

[0152] Step S81: Based on the depth map, a first area consisting of pixels having depth values ​​less than a preset safety distance is compared with a preset area.

[0153] In this embodiment, after obtaining the depth map corresponding to the image to be detected, the depth map includes: the depth value of each pixel in the map. Based on the depth map, the first area composed of pixels whose depth values ​​are less than a preset safety distance (such as 1m, which can be freely set) can be compared with the preset area.

[0154] Step S82: When the first area is larger than the preset area, the first area is marked as an obstacle for early warning and display.

[0155] In this embodiment, when it is determined that the first area is greater than or equal to the preset area, the first area can be determined as a potential obstacle, and the first area can be marked as an obstacle for early warning and display, so as to realize obstacle detection for autonomous driving and realize early warning prompts during the autonomous driving process.

[0156] In an optional embodiment, an online monocular depth estimation system is provided. The system is designed to predict the depth of a user-provided image and return a visual result to help the user better understand the spatial structure and object distribution in the image. The specific flow chart of the system is as follows: Figure 7 As shown, Figure 7 This is a training and reasoning flow chart of a monocular depth estimation system according to an embodiment of the present invention. Figure 7The system specifically includes two main processes: model training and real-time inference. Due to the large differences in depth distribution ranges between indoor and outdoor scenes, it is difficult to use a single model to address both scenarios. Therefore, the system performs training and inference for both separately. The system hardware configuration and software versions are shown in Table 1:

[0157] Table 1 System hardware configuration and software version information

[0158]

[0159] Among them, the training phase:

[0160] During training, the NYU Depth V2 dataset was used for indoor scenes. This dataset contains 464 indoor scenes, totaling over 24,000 RGB images and their corresponding depth maps, each with a resolution of 640×480. The KITTI dataset, commonly used in autonomous driving, was used for outdoor scenes. It includes RGB images captured from different viewpoints, LiDAR data, and camera calibration information, totaling approximately 23,000 images of 1238×374 pixels. To increase data diversity and improve model generalization, the system augments the training data using methods such as random cropping, horizontal flipping, contrast adjustment, and hue adjustment. The crop size for the NYU Depth V2 dataset is 576×448, while the crop size for the KITTI dataset is 768×352. The probability of triggering the remaining augmentation methods is set to 50%.

[0161] In the model building part, this system uses Pytorch 2.0.0 to implement the network structure and uses scale-invariant loss L to train the model. The specific calculation method is as follows:

[0162] ;

[0163] in, and Denotes the depth map value corresponding to the i-th pixel in the real depth map and the model predicted depth map, respectively, and N is the total number of image pixels. A single-cycle learning rate strategy with Adam optimizer is used, and the learning rate is in the first half from grow to , and then in the second half from Descend to To prevent overfitting, the weight decay of the diffusion model is set to 0.01. Use the scaling factor set in Stable Diffusion To rescale the latent encoder output, we used 100x100_0 ...

[0164] Reasoning stage:

[0165] During the inference phase, the system receives the user's input image and loads it into memory. It then preprocesses the image and pads it to ensure that the image dimensions are multiples of 64. The model that performed best during the training phase then loads the input image and performs inference on it, generating a predicted depth map. Because the depth map generated by the model inference is a 16-bit single-channel image, it must be converted to a three-channel RGB visualization for user viewing. Finally, the depth image and the converted visualization are output for display.

[0166] In addition, in another embodiment, an autonomous driving obstacle detection system is proposed. The autonomous driving obstacle detection system is designed to detect obstacles in front of the vehicle in real time through monocular depth estimation technology and provide distance prompts to assist the driver or the autonomous driving system in making safe decisions. The specific flow chart of the system is as follows Figure 8 As shown, Figure 8 This is a training and inference flowchart for an autonomous driving obstacle detection system, illustrating an embodiment of the present invention. To adapt to real-world road conditions, the system uses independently collected datasets for training and testing, ensuring the model's adaptability to diverse road scenarios. Before training and inference, the onboard camera must be calibrated to obtain internal and external camera parameters for image correction and eliminate the effects of lens distortion.

[0167] The process and parameter settings of the training phase are the same as those of the previous embodiment, and the two differ only in the data set used. In order to build a high-quality training data set, this system uses on-board cameras and LiDAR devices to collect data in actual road environments. The collected data include: ① RGB images: The resolution is 1920×1080, covering a variety of lighting and weather conditions such as day, night, rainy days, and foggy days. ② Depth information: High-precision depth maps are obtained through LiDAR devices to supervise the training of the model. The collected data set totals about 5,000 images, covering a variety of scenes such as urban roads, highways, and rural roads.

[0168] During the inference phase, the system first loads the model weights that performed best during the training phase. It then receives real-time images captured by the vehicle's onboard cameras and loads them into memory for inference, generating a predicted depth map. Based on this depth map, the system further performs obstacle detection. Distance information for each area in front of the vehicle is calculated based on the depth map. Using threshold segmentation and connected region analysis, it identifies areas in the depth map that are less than a preset safety distance, marks them as potential obstacles, and issues a warning to the driver or the autonomous driving system.

[0169] In one embodiment, the present invention has a more accurate depth prediction effect than other methods. Table 2 shows the performance comparison results of the present invention and other methods on the KITTI dataset.

[0170] Table 2 Quantitative comparison results on the KITTI dataset

[0171]

[0172] Compared to other monocular depth estimation methods, our method demonstrates significant advantages on the KITTI dataset, achieving leading results in both precision and accuracy. Specifically, our method achieves a root mean square error (RMSE) of 1.967 and an absolute error (REL) of 0.047, both of which are the best among all methods, and also ranks first in accuracy. The calculation formulas for each metric are shown below:

[0173] ;

[0174] Among them, in the above formula, Indicates the The predicted depth of each pixel, Indicates the The true depth of pixels, Represents the total number of pixels in the image. The accuracy is calculated as follows: and The maximum value between the two, the statistical result falls within the given threshold range The percentage of inliers in the total number of pixels. During the evaluation process, the threshold is usually set to 1.25, and n is usually 1, 2, or 3.

[0175] In addition, the present invention is significantly superior to other methods in terms of the accuracy and continuity of prediction results. Figure 9 Traffic signs and Figure 10 The outlines of objects such as plants and computers in the image are clearer, which is attributed to the use of object information as a hint in this method. Figure 9This is a visualization diagram of prediction results of different methods shown in an embodiment of the present invention. Figure 10 This is a visualization diagram showing the impact of detected targets on the depth prediction results according to an embodiment of the present invention.

[0176] Leveraging the rich prior knowledge provided by the diffusion model, this method demonstrates excellent generalization capabilities and outperforms other methods across various performance metrics. The specific comparison results are shown in Table 3. All methods were trained on the NYU Depth V2 indoor dataset and then evaluated on the unseen SUN RGB-D dataset.

[0177] Table 3 Comparison of generalization capabilities of different methods

[0178]

[0179] It should be noted that for the sake of simplicity, the method embodiments are described as a series of actions. However, those skilled in the art should be aware that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in this specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

[0180] Based on the same inventive concept, an embodiment of the present invention provides a monocular image depth estimation device based on a diffusion model and target hints. Figure 11 , Figure 11 This is a structural block diagram of a monocular image depth estimation device based on a diffusion model and target hints provided by an embodiment of the present invention. Figure 11 As shown, the monocular image depth estimation device based on diffusion model and target prompt of this embodiment may include:

[0181] An image input module, configured to input a sample detection image into a depth estimation model to be trained, wherein the depth estimation model to be trained comprises at least: a pre-trained image encoder, a pre-trained target detection network, a target prompt module to be trained, a denoising network to be trained, and an interactive decoder to be trained;

[0182] An image encoding module, configured to process the sample detection image using the pre-trained image encoder to obtain a shallow spatial representation corresponding to the sample detection image;

[0183] An object detection module is used to process the sample detection image through the pre-trained object detection network to extract the object detection result of the sample detection image;

[0184] a prompt acquisition module, configured to process the target detection result through the target prompt module to be trained to obtain target prompt information corresponding to the sample detection image;

[0185] A multi-scale feature extraction module, configured to input the shallow spatial representation and the target prompt information into the denoising network to be trained to extract multi-scale features;

[0186] A depth map output module, configured to input the multi-scale features into the interactive decoder to be trained to obtain a sample depth map;

[0187] A model training module is used to train the target prompt module to be trained, the denoising network to be trained, and the interactive decoder to be trained based on the label depth map corresponding to the sample detection image and the sample depth map, until a trained depth estimation model is obtained;

[0188] The depth estimation module is used to input the image to be detected into the trained depth estimation model to obtain a depth map corresponding to the image to be detected.

[0189] Optionally, the target detection result includes: category label information and bounding box information of the object; the prompt acquisition module includes:

[0190] a label prompt module, configured to process the category label information in the target prompt module to be trained to obtain label prompt information corresponding to the sample detection image;

[0191] A bounding box prompt module is used to process the bounding box information to obtain position prompt information and size prompt information corresponding to the sample detection image;

[0192] The prompt generation module is used to connect the label prompt information, the position prompt information and the size prompt information to obtain the target prompt information through convolution fusion.

[0193] Optionally, the label prompt module includes:

[0194] A first retrieval module is configured to retrieve a target label feature vector corresponding to the category label information from a label prompt set based on the category label information;

[0195] A first generating module is used to obtain the label prompt information based on the target label feature vector;

[0196] The label prompt set includes: multiple label feature vectors, each label feature vector corresponds to different object label information; the label feature vector is a text feature obtained by encoding the object label information in the training data set corresponding to the target detection network.

[0197] Optionally, the bounding box information includes: position information and size information corresponding to the object; the bounding box prompt module includes:

[0198] a second retrieval module, configured to determine a target cell index based on the position information, and retrieve a target position feature vector corresponding to the position information from a position prompt set based on the target cell index;

[0199] A second generating module is used to obtain the position prompt information based on the target position feature vector;

[0200] a third retrieval module, configured to determine a target size index based on the size information, and retrieve a target size feature vector corresponding to the size information from a size prompt set based on the target size index;

[0201] A third generating module is configured to obtain the size prompt information based on the target size feature vector;

[0202] The position hint set includes: a plurality of position feature vectors, each position feature vector corresponding to a different cell index; the position feature vector is a position feature corresponding to a cell after the sample detection image is divided into a plurality of cells;

[0203] The size prompt set includes: multiple size feature vectors, each size feature vector corresponds to a different size index; the size feature vector is the size feature corresponding to a level after the height and width of the object are divided into multiple levels.

[0204] Optionally, the multi-scale feature extraction module includes:

[0205] a single-step denoising module, configured to perform single-step denoising on the shallow spatial representation and the target prompt information in the denoising network to be trained;

[0206] A feature interaction module is used to interact the shallow spatial representation and the target prompt information through a cross-attention mechanism to extract the multi-scale features.

[0207] Optionally, the multi-scale feature includes at least: a first resolution feature, a second resolution feature, a third resolution feature, and a fourth resolution feature with resolutions from small to large;

[0208] Depth map output module, including:

[0209] a first interaction module, configured to use the first resolution feature as a global feature and the second resolution feature as a local feature, interact with the global feature and the local feature to obtain a first interactive fusion feature;

[0210] a second interaction module, configured to use the first interactive fusion feature as the global feature and the third resolution feature as the local feature, and to interact between the global feature and the local feature to obtain a second interactive fusion feature;

[0211] The third interaction module is configured to use the second interactive fusion feature as the global feature and the fourth resolution feature as the local feature, and to interact between the global feature and the local feature to obtain the sample depth map.

[0212] Optionally, the first interaction module, the second interaction module, or the third interaction module includes:

[0213] A first processing module is configured to extract a channel attention map from the global feature, and perform element-by-element multiplication of the channel attention map with the local feature to obtain a first fusion feature;

[0214] A second processing module is configured to extract a spatial attention map from the local features, and perform element-by-element multiplication of the spatial attention map with the global features to obtain a second fused feature;

[0215] a third processing module, configured to perform element-by-element addition of the first fused feature and the local feature to obtain a third fused feature;

[0216] a fourth processing module, configured to perform element-by-element addition of the second fused feature and the global feature to obtain a fourth fused feature;

[0217] The fifth processing module is used to splice and fuse the fourth fusion feature and the third fusion feature.

[0218] Optionally, the image to be detected is an image captured in real time by a vehicle-mounted camera, and the depth map includes a depth value of each pixel in the image; the device further includes:

[0219] A depth comparison module is configured to, after obtaining a depth map corresponding to the image to be detected, compare the size of a first area consisting of pixels having depth values ​​less than a preset safety distance with a preset area based on the depth map;

[0220] The warning module is used to mark the first area as an obstacle, issue a warning and display it when the first area is larger than the preset area.

[0221] Based on the same inventive concept, another embodiment of the present invention provides an electronic device, which includes a memory, a processor, and a computer program stored in the memory and run on the processor. When executed, the processor implements the steps in the monocular image depth estimation method based on diffusion model and target prompts described in any of the above embodiments of the present invention.

[0222] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0223] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0224] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, apparatus, or computer program products. Thus, embodiments of the present invention may take the form of a fully hardware embodiment, a fully software embodiment, or an embodiment combining software and hardware. Furthermore, embodiments of the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0225] The embodiments of the present invention are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the process in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0226] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0227] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0228] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they become aware of the basic creative concepts. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.

[0229] Finally, it should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that includes a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or terminal device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or terminal device that includes the element.

[0230] The above is a detailed introduction to the monocular image depth estimation method, device and equipment based on the diffusion model and target prompt provided by the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the ideas of the present invention, there will be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A monocular image depth estimation method based on diffusion model and target hint, characterized in that: The method comprises: Inputting the sample detection image into a depth estimation model to be trained, wherein the depth estimation model to be trained includes at least: a pre-trained image encoder, a pre-trained object detection network, a target prompt module to be trained, a denoising network to be trained, and an interactive decoder to be trained; Processing the sample detection image by the pre-trained image encoder to obtain a shallow spatial representation corresponding to the sample detection image; Processing the sample detection image through the pre-trained target detection network to extract the target detection result of the sample detection image; Processing the target detection result by the target prompt module to be trained to obtain target prompt information corresponding to the sample detection image; Inputting the shallow spatial representation and the target prompt information into the denoising network to be trained to extract multi-scale features; Inputting the multi-scale features into the interactive decoder to be trained to obtain a sample depth map; Based on the label depth map corresponding to the sample detection image and the sample depth map, the target prompt module to be trained, the denoising network to be trained, and the interactive decoder to be trained are trained until a trained depth estimation model is obtained; Inputting the image to be detected into the trained depth estimation model to obtain a depth map corresponding to the image to be detected; The multi-scale features include at least: a first resolution feature, a second resolution feature, a third resolution feature, and a fourth resolution feature in increasing order of resolution; Inputting the multi-scale features into the interactive decoder to be trained to obtain a sample depth map, including: Taking the first resolution feature as a global feature and the second resolution feature as a local feature, performing interaction between the global feature and the local feature to obtain a first interactive fusion feature; Using the first interactive fusion feature as the global feature and the third resolution feature as the local feature, interacting between the global feature and the local feature to obtain a second interactive fusion feature; Using the second interactive fusion feature as the global feature, using the fourth resolution feature as the local feature, and performing interaction between the global feature and the local feature to obtain the sample depth map; Interacting the global feature with the local feature includes: Extracting a channel attention map from the global feature, and multiplying the channel attention map by the local feature element by element to obtain a first fusion feature; Extracting a spatial attention map from the local features, and multiplying the spatial attention map by the global features element-by-element to obtain a second fused feature; Adding the first fused feature and the local feature element by element to obtain a third fused feature; Adding the second fused feature to the global feature element by element to obtain a fourth fused feature; The fourth fusion feature and the third fusion feature are spliced ​​and fused.

2. The monocular image depth estimation method based on diffusion model and target hint according to claim 1, characterized in that: The target detection result includes: category label information and bounding box information of the object; the target detection result is processed by the target prompt module to be trained to obtain target prompt information corresponding to the sample detection image, including: In the target prompt module to be trained, the category label information is processed to obtain label prompt information corresponding to the sample detection image; Processing the bounding box information to obtain position prompt information and size prompt information corresponding to the sample detection image; The label prompt information, the position prompt information and the size prompt information are connected and fused through convolution to obtain the target prompt information.

3. The monocular image depth estimation method based on diffusion model and target hint according to claim 2, characterized in that: Processing the category label information to obtain label prompt information corresponding to the sample detection image includes: Based on the category label information, retrieving a target label feature vector corresponding to the category label information from a label prompt set; Based on the target label feature vector, obtaining the label prompt information; The label prompt set includes: multiple label feature vectors, each label feature vector corresponds to different object label information; the label feature vector is a text feature obtained by encoding the object label information in the training data set corresponding to the target detection network.

4. The monocular image depth estimation method based on diffusion model and target hint according to claim 2, characterized in that: The bounding box information includes: position information and size information corresponding to the object; processing the bounding box information to obtain position prompt information and size prompt information corresponding to the sample detection image includes: determining a target cell index based on the position information, and retrieving a target position feature vector corresponding to the position information from a position prompt set based on the target cell index; Obtaining the location prompt information based on the target location feature vector; determining a target size index based on the size information, and retrieving a target size feature vector corresponding to the size information from a size prompt set based on the target size index; Obtaining the size prompt information based on the target size feature vector; The position hint set includes: a plurality of position feature vectors, each position feature vector corresponding to a different cell index; the position feature vector is a position feature corresponding to a cell after the sample detection image is divided into a plurality of cells; The size prompt set includes: multiple size feature vectors, each size feature vector corresponds to a different size index; the size feature vector is the size feature corresponding to a level after the height and width of the object are divided into multiple levels.

5. The monocular image depth estimation method based on diffusion model and target hint according to claim 1, characterized in that: Inputting the shallow spatial representation and the target prompt information into the denoising network to be trained to extract multi-scale features, including: In the denoising network to be trained, performing single-step denoising on the shallow spatial representation and the target prompt information; The shallow spatial representation and the target prompt information are interacted through a cross-attention mechanism to extract the multi-scale features.

6. The monocular image depth estimation method based on diffusion model and target hint according to any one of claims 1 to 5, characterized in that: The image to be detected is an image captured in real time by a vehicle-mounted camera, and the depth map includes the depth value of each pixel in the image; After obtaining the depth map corresponding to the image to be detected, the method further includes: Based on the depth map, a first area consisting of pixels having depth values ​​less than a preset safety distance is compared with a preset area; In the case that the first area is larger than the preset area, the first area is marked as an obstacle for early warning and displayed.

7. A monocular image depth estimation device based on diffusion model and target hint, characterized in that: The device comprises: An image input module, configured to input a sample detection image into a depth estimation model to be trained, wherein the depth estimation model to be trained comprises at least: a pre-trained image encoder, a pre-trained target detection network, a target prompt module to be trained, a denoising network to be trained, and an interactive decoder to be trained; An image encoding module, configured to process the sample detection image using the pre-trained image encoder to obtain a shallow spatial representation corresponding to the sample detection image; An object detection module is used to process the sample detection image through the pre-trained object detection network to extract the object detection result of the sample detection image; a prompt acquisition module, configured to process the target detection result through the target prompt module to be trained to obtain target prompt information corresponding to the sample detection image; A multi-scale feature extraction module, configured to input the shallow spatial representation and the target prompt information into the denoising network to be trained to extract multi-scale features; A depth map output module, configured to input the multi-scale features into the interactive decoder to be trained to obtain a sample depth map; A model training module is used to train the target prompt module to be trained, the denoising network to be trained, and the interactive decoder to be trained based on the label depth map corresponding to the sample detection image and the sample depth map, until a trained depth estimation model is obtained; A depth estimation module is used to input the image to be detected into the trained depth estimation model to obtain a depth map corresponding to the image to be detected; The multi-scale features include at least: a first resolution feature, a second resolution feature, a third resolution feature, and a fourth resolution feature in increasing order of resolution; Depth map output module, including: a first interaction module, configured to use the first resolution feature as a global feature and the second resolution feature as a local feature, interact with the global feature and the local feature to obtain a first interactive fusion feature; a second interaction module, configured to use the first interactive fusion feature as the global feature and the third resolution feature as the local feature, and to interact between the global feature and the local feature to obtain a second interactive fusion feature; a third interaction module, configured to use the second interactive fusion feature as the global feature and the fourth resolution feature as the local feature, and interact between the global feature and the local feature to obtain the sample depth map; The first interaction module, the second interaction module or the third interaction module includes: A first processing module is configured to extract a channel attention map from the global feature, and perform element-by-element multiplication of the channel attention map with the local feature to obtain a first fusion feature; A second processing module is configured to extract a spatial attention map from the local features, and perform element-by-element multiplication of the spatial attention map with the global features to obtain a second fused feature; a third processing module, configured to perform element-by-element addition of the first fused feature and the local feature to obtain a third fused feature; a fourth processing module, configured to perform element-by-element addition of the second fused feature and the global feature to obtain a fourth fused feature; The fifth processing module is used to splice and fuse the fourth fusion feature and the third fusion feature.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the computer program is executed by the processor, the method for monocular image depth estimation based on diffusion model and target prompting is implemented.

Images