A monocular depth estimation method, device, electronic equipment and storage medium

Abstract

Embodiments of the present application disclose a monocular depth estimation method and device, electronic equipment and storage medium, relating to the technical field of multi-modal computer vision and natural language processing, wherein the method comprises: extracting multi-scale visual features by using a visual encoder, and combining images and text descriptions to generate scene semantic descriptions; extracting semantic categories in the scene semantic descriptions by using a large language model, and mapping the semantic categories to high-dimensional feature vectors as semantic centers; calculating visual similarity between each pixel and a visual center, and weighting and fusing the visual similarity and semantic matching degrees of the semantic centers to obtain pixel grouping; mapping object spatial relationships in an RGB image to a structured constraint matrix according to the scene semantic descriptions; and generating a depth map by dynamically fusing visual, semantic and geometric information by using a cross-modal attention mechanism. The present application solves the problems of insufficient semantic understanding, lack of structural constraints and shallow multi-modal fusion in the prior art, and significantly improves the accuracy of depth estimation.

Description

Technical Field

[0001] This invention relates to the field of multimodal computer vision and natural language processing, and in particular to a monocular depth estimation method, apparatus, electronic device and storage medium. Background Technology

[0002] With the rapid development of computer vision technology, monocular depth estimation, as one of the core tasks of 3D scene understanding, has shown broad application prospects in fields such as autonomous driving, robot navigation, and augmented reality (AR).

[0003] However, traditional monocular depth estimation methods mainly rely on the similarity calculation of visual features themselves, such as combining an encoder-decoder structure with semantic segmentation to assist the task, or using adaptive binning strategies to optimize the depth discretization process. Although these methods have improved depth prediction to some extent, they still face key problems such as insufficient semantic understanding, lack of structural constraints, and shallow multimodal fusion when dealing with complex, varied, and ambiguous scenes.

[0004] Specifically, traditional methods are often limited by the limitations of visual features and cannot acquire external knowledge beyond the image content. This leads to semantic confusion and depth misjudgment when dealing with regions that have similar visual features but different semantics (such as the sky and distant mountains, the ground and the wall).

[0005] Meanwhile, existing methods typically rely on predefined rules or limited geometric assumptions regarding geometric constraints, making them ill-suited for complex and ever-changing natural scenarios, and lacking the ability to automatically extract structural constraints from natural language. Furthermore, most multimodal fusion methods remain at the shallow level of feature concatenation or attention weighting, failing to establish a deep connection between linguistic semantics and pixel-level depth distribution.

[0006] Therefore, there is an urgent need for a monocular depth estimation method that can integrate the powerful semantic understanding capabilities of large language models with pixel clustering mechanisms to improve the accuracy, consistency, and generalization ability of depth estimation. Summary of the Invention

[0007] The embodiments of this invention provide a monocular depth estimation method to address the problems of insufficient semantic understanding, lack of structural constraints, and shallow multimodal fusion in existing technologies, which lead to low depth estimation accuracy, weak generalization ability, and difficulty in handling complex scenes. The technical solution is as follows:

[0008] According to one aspect of the present invention, a monocular depth estimation method includes: extracting multi-scale visual features from an RGB image using a pre-trained visual encoder to generate multiple sets of feature maps; clustering the feature maps to obtain visual centers; generating a language description for the RGB image and fusing it with the input text description to obtain a scene semantic description; extracting semantic categories from the scene semantic description using a large language model and designed prompting engineering to form semantic prototypes; mapping the semantic prototypes to high-dimensional feature vectors using a learnable semantic embedding network as semantic centers; calculating the visual similarity between each pixel in the RGB image and the visual center, and the semantic matching degree between each pixel and the semantic center, and then weighting and fusing them to obtain pixel groups; obtaining the spatial relationships of objects in the RGB image based on the scene semantic description using a large language model, and mapping them to a structured constraint matrix; dynamically allocating the weights of the visual features, scene semantic description, and constraint matrix according to the pixel groups and scene requirements using a cross-modal attention mechanism; and weighting and upsampling each feature map according to the weights to restore it to the original image resolution to generate a depth map.

[0009] In one embodiment, multi-scale visual features are extracted from an RGB image using a pre-trained visual encoder to generate multiple sets of feature maps through the following steps: multi-scale feature extraction is performed on the input RGB image using a pre-trained visual encoder, generating multiple sets of feature maps with different resolutions and semantic levels at different levels; shallow feature maps include low-level visual information, and deep feature maps include high-level semantic information; the visual encoder includes ResNet and VisionTransformer.

[0010] In one embodiment, generating a language description for the RGB image and fusing it with the input text description to obtain a scene semantic description is achieved through the following steps: obtaining the input text description; automatically generating a natural language description of the RGB image using a pre-trained image description generation model; the natural language description includes the main objects and scene types in the RGB image; and semantically aligning and splicing the natural language description with the text description through a text fusion module to form a scene semantic description.

[0011] In one embodiment, semantic categories are extracted from the scene semantic description through a large language model and a designed prompting engineering to form semantic prototypes. These semantic prototypes are then mapped to high-dimensional feature vectors using a learnable semantic embedding network, serving as semantic centers. This is achieved through the following steps: the scene semantic description is input into a large language model; the designed prompting engineering guides the model to generate multiple semantic categories to form semantic prototypes; and a learnable semantic embedding network is used to map the semantic prototypes to high-dimensional feature vectors, serving as semantic centers. The semantic categories include sky, buildings, roads, vehicles, and pedestrians.

[0012] In one embodiment, the pixel grouping is obtained by weighted fusion after calculating the visual similarity between each pixel in the RGB image and the visual center, and the semantic matching degree between each pixel and the semantic center. This is achieved through the following steps: calculating the distance between the visual features of each pixel in the RGB image and the visual center to obtain the visual similarity; calculating the dot product between each pixel and the semantic center vector to obtain the semantic matching degree; combining the visual similarity and semantic matching degree through a weighted fusion mechanism to generate a comprehensive score; and assigning each pixel to the nearest cluster center according to the comprehensive score to form a semantically aware pixel grouping.

[0013] In one embodiment, the spatial relationships between objects in the RGB image are obtained from the scene semantic description using a large language model and mapped into a structured constraint matrix through the following steps: The spatial relationships between objects in the RGB image are parsed from the scene semantic description using the reasoning capability of the large language model; the spatial relationships are transformed into a graph-structured constraint matrix using a relation extraction network; in the graph structure, nodes represent semantic regions and edges represent spatial constraint relationships; the constraint matrix is ​​transformed into a formal representation and used as a regularization term in the loss function during model training; the spatial relationships include a sofa in front of a coffee table and a window above a wall; the formal representation includes depth order constraints and planar consistency constraints; the relation extraction network includes a parser based on a graph neural network.

[0014] In one embodiment, the method further includes the following steps: obtaining a natural language input request text, identifying the instruction intent in the request text through a semantic parsing module, and dynamically adjusting the weights and parameters in the attention mechanism according to the instruction intent using a lightweight optimization strategy; the request text includes enhancing foreground details and smoothing background areas.

[0015] According to one aspect of the present invention, a monocular depth estimation device is provided, the device comprising: a visual and text feature extraction module, configured to extract multi-scale visual features from an RGB image using a pre-trained visual encoder, generate multiple sets of feature maps, cluster the feature maps to obtain visual centers, generate a linguistic description for the RGB image, and fuse an input text description to obtain a scene semantic description; and a semantic generation and fusion module, configured to extract semantic categories from the scene semantic description through a large language model and designed prompting engineering, construct semantic prototypes, and map the semantic prototypes into high-dimensional feature vectors as semantic centers through a learnable semantic embedding network; pixels The grouping and spatial mapping module is used to calculate the visual similarity between each pixel in the RGB image and the visual center, and the semantic matching degree between each pixel and the semantic center, and then weighted and fused to obtain pixel groups. The spatial relationship of objects in the RGB image is obtained by using a large language model based on the scene semantic description, and mapped into a structured constraint matrix. The depth estimation and image reconstruction module is used to dynamically allocate the weights of the visual features, scene semantic description and constraint matrix according to the pixel groups and scene requirements through a cross-modal attention mechanism. The feature maps are weighted and upsampled and fused according to the weights to restore the original image resolution and generate a depth map.

[0016] According to one aspect of the present invention, an electronic device includes at least one processor and at least one memory, wherein computer-readable instructions are stored on the memory; the computer-readable instructions are executed by one or more of the processors to cause the electronic device to implement the monocular depth estimation method as described above.

[0017] According to one aspect of the invention, a storage medium stores computer-readable instructions thereon, which are executed by one or more processors to implement the monocular depth estimation method as described above.

[0018] The beneficial effects of the technical solution provided by this invention are:

[0019] In the above technical solution, this invention first utilizes a pre-trained visual encoder to extract multi-scale visual features from RGB images, generating feature maps and clustering them to obtain visual centers. Simultaneously, it combines an image description generation model with user text input to form a scene semantic description. Subsequently, a large language model extracts semantic categories to construct semantic prototypes, which are then mapped to high-dimensional feature vectors as semantic centers. In the pixel grouping stage, the similarity between pixels and the visual and semantic centers is calculated and weighted fusion is performed to generate semantically aware pixel groups. Further, the large language model is used to parse the spatial relationships of objects, transforming them into a structured constraint matrix to guide the geometric consistency of depth estimation. Finally, a cross-modal attention mechanism dynamically allocates weights for visual, semantic, and constraint information, performing weighted fusion and upsampling on the feature maps to generate a high-precision depth map. This method significantly improves the accuracy, semantic consistency, and generalization ability of depth estimation, and is particularly suitable for interactive depth modeling tasks in complex scenes. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention, and those skilled in the art can obtain other drawings based on these drawings without creative effort.

[0021] Figure 1 This is a flowchart illustrating a monocular depth estimation method according to an exemplary embodiment;

[0022] Figure 2 This is a flowchart illustrating a monocular depth estimation method in an exemplary embodiment;

[0023] Figure 3 This is a block diagram of a monocular depth estimation device according to an exemplary embodiment;

[0024] Figure 4 This is a hardware structure diagram of an electronic device according to an exemplary embodiment;

[0025] Figure 5 This is a block diagram illustrating an electronic device according to an exemplary embodiment. Detailed Implementation

[0026] Embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0027] Those skilled in the art will understand that, unless specifically stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the term “comprising” as used in this disclosure means the presence of the stated features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. It should be understood that when we say an element is “connected” or “coupled” to another element, it can be directly connected or coupled to the other element, or there may be intermediate elements. Furthermore, “connected” or “coupled” as used herein can include wireless connections or wireless coupling. The term “and / or” as used herein includes all or any units and all combinations of one or more associated listed items.

[0028] This invention provides a monocular depth estimation method that achieves high-precision semantic-aware monocular depth estimation by fusing visual encoding, large language model semantic parsing, and cross-modal attention mechanisms. It effectively solves the problems of poor semantic consistency, lack of geometric constraints, and insufficient depth estimation accuracy in complex scenes using traditional methods. This monocular depth estimation method is applicable to monocular depth estimation devices, which can be electronic devices. The monocular depth estimation method in this invention can be applied to various scenarios, such as zero-shot depth modeling.

[0029] Please see Figure 1 This invention provides a monocular depth estimation method applicable to electronic devices.

[0030] In the following method embodiments, for ease of description, the execution subject of each step of the method is an electronic device, but this does not constitute a specific limitation.

[0031] like Figure 1 As shown, the method may include the following steps:

[0032] Step 110: Use a pre-trained visual encoder to extract multi-scale visual features from the RGB image, generate multiple sets of feature maps, cluster the feature maps to obtain visual centers, generate a language description for the RGB image, and fuse the input text description to obtain a scene semantic description.

[0033] In one possible implementation, a pre-trained visual encoder is used to extract multi-scale features from the input RGB image, generating multiple sets of feature maps with different resolutions and semantic levels at different levels.

[0034] Shallow feature maps include low-level visual information, while deep feature maps include high-level semantic information; visual encoders include ResNet, Vision Transformer, etc., without any specific limitations.

[0035] In one possible implementation, the input text description is obtained, and a pre-trained image description generation model is used to automatically generate a natural language description of the RGB image. The natural language description and the text description are semantically aligned and concatenated by a text fusion module to form a scene semantic description.

[0036] Natural language descriptions include the main objects and scene types in the RGB image; image description generation models include OpenAI GPT-4o, Google Gemini 1.5 Pro, etc., without any specific limitations.

[0037] Specifically, a pre-trained visual encoder (such as ResNet or Vision Transformer) is first used to extract multi-scale features from the input RGB image. This process generates multiple sets of feature maps with different resolutions and semantic levels at different layers. Shallow feature maps contain low-level visual information (such as edges and textures), while deep feature maps are rich in high-level semantic information (such as object categories and scene types).

[0038] Furthermore, pre-trained image description generation models (such as OpenAI GPT-4o or Google Gemini 1.5 Pro) are used to automatically generate natural language descriptions for RGB images, covering the main objects and scene types in the image. If the user provides additional text descriptions, the natural language descriptions are semantically aligned and concatenated with the user's text through a text fusion module to form a more complete scene semantic description.

[0039] In the above process, this embodiment of the invention captures visual and linguistic information from images through multimodal input, providing a rich data foundation for subsequent processing. The multi-scale feature extraction capability of the visual encoder ensures that visual information at different levels is effectively captured, while the image description generation model provides natural language interpretation of the image content, enhancing the system's semantic understanding capability. Through multi-scale visual feature extraction and linguistic description generation and fusion, the system can simultaneously utilize visual and linguistic information, providing a comprehensive foundation for scene understanding and achieving preliminary integration of multimodal data.

[0040] Step 120: Extract semantic categories from the semantic description of the scene through a large language model and the designed prompting engineering to form a semantic prototype. Map the semantic prototype into a high-dimensional feature vector through a learnable semantic embedding network, which serves as the semantic center.

[0041] In one possible implementation, the scene semantic description is input into a large language model. The model is guided to generate multiple semantic categories through a designed prompting engineering to form a semantic prototype. A learnable semantic embedding network is then used to map the semantic prototype into a high-dimensional feature vector, which serves as the semantic center.

[0042] The semantic categories include sky, buildings, roads, vehicles, pedestrians, etc., without any specific restrictions.

[0043] Specifically, the scene semantic description is input into the large language model, and the model is guided to generate multiple semantic categories (such as sky, buildings, roads, vehicles, and pedestrians) through carefully designed prompting engineering. These semantic categories constitute a set of semantic prototypes.

[0044] Furthermore, a learnable semantic embedding network is used to map semantic prototypes into high-dimensional feature vectors, which serve as initial references for cluster centers. These feature vectors have the same dimensional space as the visual features, facilitating subsequent similarity calculations. Clustering is then performed on the visual feature maps to obtain visual centers, which represent the main visual regions in the image.

[0045] In the above process, embodiments of the present invention utilize semantic prototypes extracted from a language model to guide the clustering process of visual features. As an abstract representation of high-level semantic information, the semantic prototype provides semantic guidance for the clustering of visual features, ensuring that the clustering results consider not only visual similarity but also semantic consistency. Through semantic prototype extraction and mapping, as well as visual center clustering, the clustering process can simultaneously consider visual and semantic information, providing semantically perceptive clustering results and achieving a preliminary fusion of visual and semantic information.

[0046] Step 130: Calculate the visual similarity between each pixel in the RGB image and the visual center, and the semantic matching degree between each pixel and the semantic center, and then weight and fuse them to obtain pixel groups. Use a large language model to obtain the spatial relationship of objects in the RGB image based on the semantic description of the scene, and map it into a structured constraint matrix.

[0047] In one possible implementation, the visual similarity is obtained by calculating the distance between the visual features of each pixel in the RGB image and the visual center. The semantic matching degree is obtained by calculating the dot product of each pixel and the semantic center vector. The visual similarity and semantic matching degree are combined through a weighted fusion mechanism to generate a comprehensive score. Each pixel is assigned to the nearest cluster center according to the comprehensive score to form a semantically aware pixel group.

[0048] In one possible implementation, the reasoning ability of a large language model is used to parse the spatial relationships between objects in an RGB image from a scene semantic description. The spatial relationships are then transformed into a constraint matrix of a graph structure through a relation extraction network. The constraint matrix is ​​then transformed into a formal representation and used as a regularization term in the loss function for model training.

[0049] In this graph structure, nodes represent semantic regions, and edges represent spatial constraints. Spatial relationships include the sofa being in front of the coffee table, the window being above the wall, etc. Formal representations include depth order constraints and planar consistency constraints, etc. Relationship extraction networks include parsers based on graph neural networks, etc., none of which are specified here.

[0050] Specifically, the visual similarity is obtained by calculating the distance between the visual features of each pixel in the RGB image and the visual center; simultaneously, the semantic matching degree is obtained by calculating the dot product of each pixel and the semantic center vector. The visual similarity and semantic matching degree are combined through a weighted fusion mechanism to generate a comprehensive score. Each pixel is then assigned to the nearest cluster center according to the comprehensive score, forming semantically aware pixel groups.

[0051] Furthermore, the reasoning capabilities of large language models are utilized to parse the spatial relationships (such as front-back relationships, top-bottom relationships, and occlusion relationships) between objects in RGB images from scene semantic descriptions. These spatial relationships are then transformed into a graph-structured constraint matrix through a relation extraction network, where nodes represent semantic regions and edges represent spatial constraint relationships.

[0052] Specifically, pixel grouping and spatial relationship analysis further refine the semantic and structural information of the image. Pixel grouping considers both visual and semantic factors, ensuring that pixels of the same semantic category can be correctly classified. Spatial relationship analysis utilizes the reasoning ability of a language model to extract spatial relationships between objects from natural language descriptions, providing geometric constraints for depth estimation.

[0053] In the above process, the embodiments of the present invention enable the system to more accurately understand the semantic and structural information of the image through pixel grouping and spatial relationship parsing, providing refined pixel-level semantic grouping and spatial relationship constraints, and realizing in-depth parsing of image content.

[0054] Step 140: The weights of visual features, scene semantic descriptions and constraint matrices are dynamically allocated according to pixel grouping and scene requirements through a cross-modal attention mechanism. The feature maps are then weighted and upsampled and fused according to the weights to restore the original image resolution and generate a depth map.

[0055] In one possible implementation, the requirement text is obtained from natural language input, the instruction intent in the requirement text is identified through a semantic parsing module, and a lightweight optimization strategy is used to dynamically adjust the weights and parameters in the attention mechanism according to the instruction intent.

[0056] The requirements text included things like enhancing foreground details and smoothing background areas, but these were not specified here.

[0057] Specifically, a cross-modal attention mechanism is employed to dynamically allocate weights for visual features, scene semantic descriptions, and constraint matrices based on pixel grouping and scene requirements. This mechanism ensures that the system can automatically select the most relevant information for depth estimation in different scenarios.

[0058] Furthermore, based on the weights assigned by the cross-modal attention mechanism, each feature map is weighted and processed, and the depth map resolution is gradually refined through an upsampling strategy, ultimately generating a dense depth map of the same size as the input image. During the inference phase, users can adjust the depth estimation results through natural language input, such as inputting commands like "enhance foreground details" or "smooth background areas." The system will then adjust model parameters or post-processing strategies in real time, achieving interactive depth estimation.

[0059] Specifically, a cross-modal attention mechanism was used to achieve deep fusion of visual, linguistic, and geometric constraint information. Feature weighting and upsampling fusion strategies ensured the resolution and accuracy of the depth map, while interactive control functions greatly improved the system's usability and user experience.

[0060] In the above process, the embodiments of the present invention enable the system to generate high-precision, semantically consistent depth maps through cross-modal feature fusion and depth map generation, and provide interactive control capabilities, realizing real-time adjustment and optimization of depth estimation results, which significantly improves the practicality of the system and user experience.

[0061] Through the above process, this invention achieves high-precision, semantically consistent depth estimation by employing multimodal input preprocessing, semantic prototype and visual center construction, pixel grouping and spatial relationship analysis, and cross-modal feature fusion and depth map generation. This not only significantly improves the accuracy and consistency of depth estimation but also enhances the system's generalization ability and interactivity, providing a significant breakthrough for the practical application of monocular depth estimation technology.

[0062] In an exemplary embodiment, the process of performing monocular depth estimation by the monocular depth estimation method of the present invention is illustrated.

[0063] like Figure 2 As shown, the following steps may be included:

[0064] Step S1: Input RGB image, extract multi-scale visual features from RGB image through visual encoder, perform text fusion with scene description corresponding to RGB image and user text (if input).

[0065] Specifically, the input RGB image is fed into a pre-trained visual encoder to extract multi-scale visual features, generating multiple sets of feature maps, including shallow feature maps (containing low-level visual information such as edges and textures) and deep feature maps (containing high-level semantic information such as object categories and scene layouts). Simultaneously, a pre-trained image description generation model automatically generates a natural language description of the RGB image, covering basic information such as the main objects and scene types in the image.

[0066] Specifically, clustering is performed based on the extracted multi-scale visual feature maps to obtain visual centers. The scene semantic description formed in step S2 is input into the large language model. Multiple semantic categories (such as sky, buildings, roads, vehicles, pedestrians, etc.) are generated through the designed prompting engineering guidance model to form a semantic prototype set.

[0067] Specifically, the text description input by the user (if any) is obtained, and the automatically generated natural language description is semantically aligned and spliced ​​with the text description input by the user through the text fusion module to form a complete scene semantic description.

[0068] In the above process, the embodiments of the present invention extract multi-scale visual features through a visual encoder, providing a rich visual information foundation for subsequent clustering and depth estimation; generate natural language descriptions of images, providing textual basis for subsequent text fusion and semantic understanding; obtain visual centers through clustering, providing visual references for pixel grouping; extract semantic prototypes, transforming semantic information into feature vectors that can be used for computation; and integrate automatically generated image descriptions and additional information provided by users through text fusion, forming a more comprehensive scene semantic description that better meets user needs, providing richer semantic information for subsequent semantic prototype extraction.

[0069] Step S2: Generate geometric constraints.

[0070] Specifically, the reasoning ability of a large language model is used to parse the spatial relationships between objects in RGB images from scene semantic descriptions (such as a sofa in front of a coffee table, a window above a wall, etc.). These spatial relationships are then transformed into a graph structure constraint matrix through a relation extraction network, where nodes represent semantic regions and edges represent spatial constraint relationships. This matrix is ​​then transformed into a formal representation (such as depth order constraints and planar consistency constraints) and used as a regularization term in the loss function for model training.

[0071] In the above process, the embodiments of the present invention generate geometric constraints, which provide spatial structural constraints for depth estimation and ensure the geometric rationality of the depth map.

[0072] Step S3: Semantic-guided clustering and multimodal feature fusion.

[0073] Specifically, the visual similarity is obtained by calculating the distance between the visual features of each pixel in the RGB image and the visual center; the semantic matching degree is obtained by calculating the dot product of each pixel and the semantic center vector. A weighted fusion mechanism is used to combine the visual similarity and semantic matching degree to generate a comprehensive score. Each pixel is then assigned to the nearest cluster center based on the comprehensive score, forming semantically aware pixel groups. Simultaneously, multimodal feature fusion is performed by combining previously generated geometric constraints and other information.

[0074] In the above process, the embodiments of the present invention consider both visual and semantic factors through pixel grouping, so that pixels of the same semantic category can be correctly classified; through multimodal feature fusion, visual, semantic and geometric constraint information are integrated to provide a comprehensive feature representation for generating accurate depth maps.

[0075] Step S4: Parameter adjustment and feature decoding to generate depth map.

[0076] Specifically, based on the intent of the user's interaction commands, a lightweight optimization strategy is used to dynamically adjust the weights and parameters in the cross-modal attention mechanism. Furthermore, the adjusted multimodal features are input into the feature decoder, and upsampling and other operations are used to restore the original image resolution, generating a continuous depth map.

[0077] In the above process, the embodiments of the present invention optimize the depth estimation results according to user needs by adjusting parameters; and transform the fused features into the final depth map through the feature decoder, thus completing the entire process of monocular depth estimation.

[0078] Step S5: Processing user interaction instructions (if there is user input).

[0079] Specifically, user interaction commands (if any) are obtained, such as "enhance foreground details" or "smooth background areas." Further, the semantic parsing module identifies the intent of the commands, preparing for subsequent parameter adjustments. Simultaneously, combining the previously obtained visual center, semantic center, and geometric constraints, multimodal feature fusion is prepared.

[0080] In the above process, the embodiments of the present invention realize the interactive control of the depth estimation process by obtaining user interaction instructions, thereby meeting the user's personalized needs; by integrating multi-source information, a comprehensive data foundation is provided for multimodal feature fusion.

[0081] Through the above process, this embodiment of the invention achieves monocular depth estimation by integrating a large language model and a pixel clustering mechanism, solving the problems of traditional methods in terms of semantic consistency, geometric constraints, and interactivity. It generates a high-precision, semantically consistent, and interactively adjustable depth map. By integrating the semantic understanding capabilities of the large language model, it extracts scene semantic prototypes and spatial structural relationships, and introduces them as prior knowledge into the pixel clustering and feature decoding process of the depth decoder. Combining visual, semantic, and geometric constraints for multimodal collaborative optimization significantly improves the depth estimation accuracy.

[0082] The following are embodiments of the apparatus of the present invention, which can be used to execute the monocular depth estimation method involved in the present invention. For details not disclosed in the embodiments of the apparatus of the present invention, please refer to the method embodiments of the monocular depth estimation method involved in the present invention.

[0083] Please see Figure 3 This invention provides a monocular depth estimation device 800.

[0084] The monocular depth estimation device 800 includes, but is not limited to: a visual and text feature extraction module 810, a semantic generation and fusion module 830, a pixel grouping and spatial mapping module 850, and a depth estimation and image reconstruction module 870.

[0085] The visual and text feature extraction module 810 is used to extract multi-scale visual features from RGB images using a pre-trained visual encoder, generate multiple sets of feature maps, cluster the feature maps to obtain visual centers, generate language descriptions for RGB images, and fuse the input text descriptions to obtain scene semantic descriptions.

[0086] The semantic generation and fusion module 830 is used to extract semantic categories from scene semantic descriptions through large language models and designed prompt engineering, to form semantic prototypes, and to map the semantic prototypes into high-dimensional feature vectors through a learnable semantic embedding network, which serve as semantic centers.

[0087] The pixel grouping and spatial mapping module 850 is used to calculate the visual similarity between each pixel in the RGB image and the visual center, and the semantic matching degree between each pixel and the semantic center, and then weighted and fused to obtain pixel groups. The spatial relationship of objects in the RGB image is obtained by the large language model based on the scene semantic description, and mapped into a structured constraint matrix.

[0088] The depth estimation and image reconstruction module 870 is used to dynamically allocate the weights of visual features, scene semantic descriptions and constraint matrices according to pixel grouping and scene requirements through a cross-modal attention mechanism. Based on the weights, the feature maps are weighted and upsampled and fused to restore the original image resolution and generate a depth map.

[0089] It should be noted that the monocular depth estimation provided in the above embodiments is only illustrated by the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed. That is, the internal structure of the monocular depth estimation device will be divided into different functional modules to complete all or part of the functions described above.

[0090] Furthermore, the monocular depth estimation device and the monocular depth estimation method provided in the above embodiments belong to the same concept, and the specific way in which each module performs its operation has been described in detail in the method embodiments, and will not be repeated here.

[0091] Figure 4 A schematic diagram of the structure of an electronic device according to an exemplary embodiment is shown.

[0092] It should be noted that this electronic device is merely an example adapted to the present invention and should not be construed as providing any limitation on the scope of use of the present invention. Furthermore, this electronic device should not be interpreted as requiring or depending on having... Figure 4 One or more components of the exemplary electronic device 2000 shown.

[0093] The hardware structure of electronic devices 2000 can vary significantly due to differences in configuration or performance, such as... Figure 4 As shown, the electronic device 2000 includes: a power supply 210, an interface 230, at least one memory 250, and at least one central processing unit (CPU) 270.

[0094] Specifically, power supply 210 is used to provide operating voltage for various hardware devices on electronic device 2000.

[0095] Interface 230 includes at least one wired or wireless network interface 231 for interacting with external devices. Of course, in other examples adapted to this invention, interface 230 may further include at least one serial-to-parallel conversion interface 233, at least one input / output interface 235, and at least one USB interface 237, etc. Figure 4 As shown, this does not constitute a specific limitation.

[0096] The memory 250 serves as a carrier for resource storage and can be a read-only memory, random access memory, disk, or optical disk, etc. The resources stored on it include the operating system 251, application programs 253, and data 255, etc., and the storage method can be temporary storage or permanent storage.

[0097] The operating system 251 is used to manage and control the various hardware devices and application programs 253 on the electronic device 2000, so as to enable the central processing unit 270 to perform calculations and processing on the massive data 255 in the memory 250. It can be Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0098] Application 253 is a computer-readable instruction based on operating system 251 that performs at least one specific task, and may include at least one module ( Figure 4 (Not shown), each module may contain computer-readable instructions for the electronic device 2000. For example, the monocular depth estimation device may be considered as application program 253 deployed on the electronic device 2000.

[0099] Data 255 may be signal information, etc., and is stored in memory 250.

[0100] The central processing unit 270 may include one or more processors and is configured to communicate with the memory 250 via at least one communication bus to read computer-readable instructions stored in the memory 250, thereby performing operations and processing on the massive amounts of data 255 stored in the memory 250. For example, a monocular depth estimation method may be performed by the central processing unit 270 reading a series of computer-readable instructions stored in the memory 250.

[0101] Furthermore, the present invention can also be implemented through hardware circuits or a combination of hardware circuits and software. Therefore, the implementation of the present invention is not limited to any specific hardware circuit, software, or combination thereof.

[0102] Please see Figure 5 This invention provides an electronic device 4000, which may include: a desktop computer, a laptop computer, a server, etc., with sensor recognition capabilities.

[0103] exist Figure 5 In this context, the electronic device 4000 includes at least one processor 4001 and at least one memory 4003.

[0104] The data interaction between the processor 4001 and the memory 4003 can be achieved through at least one communication bus 4002. This communication bus 4002 may include a path for transmitting data between the processor 4001 and the memory 4003. The communication bus 4002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. The communication bus 4002 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 5 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0105] Optionally, the electronic device 4000 may further include a transceiver 4004, which can be used for data interaction between the electronic device and other electronic devices, such as sending and / or receiving data. It should be noted that in practical applications, the transceiver 4004 is not limited to one type, and the structure of the electronic device 4000 does not constitute a limitation on the embodiments of the present invention.

[0106] Processor 4001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this invention. Processor 4001 may also be a combination that implements computing functions, such as including one or more microprocessor combinations, a combination of a DSP and a microprocessor, etc.

[0107] The memory 4003 may be a ROM (Read Only Memory) or other type of static storage device capable of storing static information and instructions, RAM (Random Access Memory) or other type of dynamic storage device capable of storing information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), CD-ROM (Compact Disc Read Only Memory) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program instructions or code in the form of instructions or data structures and accessible by the electronic device 4000, but not limited thereto.

[0108] The memory 4003 stores computer-readable instructions, and the processor 4001 can read the computer-readable instructions stored in the memory 4003 through the communication bus 4002.

[0109] The computer-readable instructions are executed by one or more processors 4001 to implement the monocular depth estimation methods in the above embodiments.

[0110] Furthermore, this embodiment of the invention provides a storage medium storing computer-readable instructions, which are executed by one or more processors to implement the monocular depth estimation method described above.

[0111] This invention provides a computer program product including computer-readable instructions stored in a storage medium. One or more processors of an electronic device read the computer-readable instructions from the storage medium, load and execute the computer-readable instructions, thereby enabling the electronic device to implement the monocular depth estimation method as described above.

[0112] Compared with related technologies, the beneficial effects of the present invention are:

[0113] 1. This invention can significantly improve the accuracy of depth estimation. By integrating the semantic understanding capabilities of large language models, it extracts scene semantic prototypes and spatial structural relationships, and introduces them as prior knowledge into the pixel clustering and feature decoding process of the depth decoder. It combines visual, semantic and geometric constraints to perform multimodal collaborative optimization. Experimental results on multiple mainstream datasets show that, compared with the existing best methods, the mean absolute error (MAE) is reduced by 8-12% and the root mean square error (RMSE) is reduced by 6-10%.

[0114] 2. This invention has stronger semantic consistency; through the semantic prototype-guided clustering mechanism, it establishes for the first time a direct mapping from language semantics to pixel clustering, enabling regions of the same semantic category to maintain better depth consistency, effectively solving the problem of depth discontinuity caused by texture changes in traditional methods, improving the accuracy of semantic boundaries by 15-20%, especially in key areas such as object outlines and occlusion boundaries.

[0115] 3. This invention enhances the geometric structure modeling capability of complex scenes; through a description-driven geometric constraint mechanism, the spatial structure information in the natural language description is automatically parsed into formal geometric constraints (such as depth order constraints and planar consistency constraints), guiding depth prediction to conform to physical laws. The modeling accuracy of geometric laws such as planar consistency in indoor scenes, horizon constraints in outdoor scenes, and front-back relationships between objects is improved by 10-15%, significantly reducing depth prediction results that do not conform to physical laws.

[0116] 4. This invention has excellent generalization ability; with the help of the rich semantic knowledge of the large language model, it performs well in cross-dataset evaluation. Its adaptability to new scene types and new object categories not seen during training is 20-25% higher than that of traditional methods, and it supports zero-shot depth estimation, which can handle specific scenes with scarce labeled data.

[0117] 5. This invention achieves high computational performance; although a multimodal processing module is introduced, through careful architecture design and modular implementation, the inference speed is reduced by only 5-8% compared to the benchmark method, and the memory usage increases by no more than 15%. Deployment experiments on mobile devices show that real-time processing (>30 FPS) can be achieved while maintaining high accuracy.

[0118] 6. This invention supports interactive control; depth estimation results can be adjusted in real time through natural language input (such as "enhance foreground details" or "smooth background area"), with a response time of less than 200ms. This provides strong technical support for interactive applications such as AR / VR, robot navigation, and autonomous driving, and significantly improves user experience and system usability.

[0119] It should be understood that although the steps in the flowcharts of the accompanying figures are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the accompanying figures may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0120] The above description is only a partial embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A monocular depth estimation method, characterized in that, The method includes: A pre-trained visual encoder is used to extract multi-scale visual features from RGB images, generate multiple sets of feature maps, cluster the feature maps to obtain visual centers, generate a language description for the RGB images, and fuse the input text description to obtain a scene semantic description. The semantic categories in the semantic description of the scene are extracted by a large language model and a designed prompting engineering to form a semantic prototype. The semantic prototype is then mapped into a high-dimensional feature vector by a learnable semantic embedding network, which serves as the semantic center. After calculating the visual similarity between each pixel in the RGB image and the visual center, and the semantic matching degree between each pixel and the semantic center, the pixels are weighted and fused to obtain pixel groups. The spatial relationships of objects in the RGB image are obtained by using a large language model based on the semantic description of the scene, and mapped into a structured constraint matrix. The visual features, scene semantic descriptions, and constraint matrices are dynamically weighted according to the pixel grouping and scene requirements through a cross-modal attention mechanism. The feature maps are then weighted and upsampled and fused according to the weights to restore the original image resolution and generate a depth map.

2. The monocular depth estimation method as described in claim 1, characterized in that, The process involves using a pre-trained visual encoder to extract multi-scale visual features from RGB images, generating multiple sets of feature maps, including: A pre-trained visual encoder is used to extract multi-scale features from the input RGB image, generating multiple sets of feature maps with different resolutions and semantic levels at different levels; shallow feature maps include low-level visual information, and deep feature maps include high-level semantic information; the visual encoder includes ResNet and Vision Transformer.

3. The monocular depth estimation method as described in claim 1, characterized in that, The step of generating a language description for the RGB image and fusing it with the input text description to obtain a scene semantic description includes: The input text description is obtained, and a pre-trained image description generation model is used to automatically generate a natural language description of the RGB image; the natural language description includes the object and scene types in the RGB image; The text fusion module performs semantic alignment and splicing of the natural language description and the text description to form a scene semantic description.

4. The monocular depth estimation method as described in claim 1, characterized in that, The process involves extracting semantic categories from the scene semantic description using a large language model and designed prompting engineering to form semantic prototypes. These semantic prototypes are then mapped into high-dimensional feature vectors via a learnable semantic embedding network, serving as semantic centers. This includes: The scene semantic description is input into a large language model. The model generates multiple semantic categories through a designed prompting engineering approach, forming a semantic prototype. A learnable semantic embedding network is used to map the semantic prototype into a high-dimensional feature vector, which serves as the semantic center. The semantic categories include sky, buildings, roads, vehicles, and pedestrians.

5. The monocular depth estimation method as described in claim 1, characterized in that, The step of calculating the visual similarity between each pixel in the RGB image and the visual center, and the semantic matching degree between each pixel and the semantic center, and then weighting and fusing them to obtain pixel groups includes: The visual similarity is obtained by calculating the distance between the visual features of each pixel in the RGB image and the visual center, and the semantic matching degree is obtained by calculating the dot product of each pixel and the semantic center vector. The visual similarity and semantic matching degree are combined by a weighted fusion mechanism to generate a comprehensive score. Each pixel is then assigned to the nearest cluster center according to the comprehensive score to form a semantically aware pixel group.

6. The monocular depth estimation method as described in claim 1, characterized in that, The step of obtaining the spatial relationships of objects in the RGB image based on the scene semantic description using a large language model and mapping them into a structured constraint matrix includes: The spatial relationships between objects in the RGB image are parsed from the semantic description of the scene using the reasoning ability of the large language model. The spatial relationships are then transformed into a constraint matrix of a graph structure through a relation extraction network. In the graph structure, nodes represent semantic regions and edges represent spatial constraint relationships. The constraint matrix is ​​transformed into a formal representation, which is used as a regularization term in the loss function for model training; the spatial relationship includes the sofa being in front of the coffee table and the window being above the wall; the formal representation includes depth order constraints and planar consistency constraints; the relationship extraction network includes a parser based on a graph neural network.

7. The monocular depth estimation method as described in claim 1, characterized in that, The method further includes: The system acquires a natural language input request text, identifies the instruction intent in the request text through a semantic parsing module, and dynamically adjusts the weights and parameters in the attention mechanism according to the instruction intent using a lightweight optimization strategy; the request text includes enhancing foreground details and smoothing background areas.

8. A monocular depth estimation device, characterized in that, The device includes: The visual and text feature extraction module is used to extract multi-scale visual features from RGB images using a pre-trained visual encoder, generate multiple sets of feature maps, cluster the feature maps to obtain visual centers, generate language descriptions for the RGB images, and fuse the input text descriptions to obtain scene semantic descriptions. The semantic generation and fusion module is used to extract semantic categories from the semantic description of the scene through a large language model and designed prompt engineering, to form a semantic prototype, and to map the semantic prototype into a high-dimensional feature vector through a learnable semantic embedding network, which serves as the semantic center. The pixel grouping and spatial mapping module is used to calculate the visual similarity between each pixel in the RGB image and the visual center, and the semantic matching degree between each pixel and the semantic center, and then weighted and fused to obtain pixel groups. The module obtains the spatial relationship of objects in the RGB image based on the scene semantic description through a large language model and maps it into a structured constraint matrix. The depth estimation and image reconstruction module is used to dynamically allocate the weights of the visual features, scene semantic descriptions and constraint matrices according to the pixel grouping and scene requirements through a cross-modal attention mechanism, and to perform weighted upsampling and fusion of each feature map according to the weights to restore the original image resolution and generate a depth map.

9. An electronic device, characterized in that, include: At least one processor and at least one memory, wherein, The memory stores computer-readable instructions; The computer-readable instructions are executed by one or more of the processors, causing the electronic device to implement the monocular depth estimation method as described in any one of claims 1 to 7.

10. A storage medium having computer-readable instructions stored thereon, characterized in that, The computer-readable instructions are executed by one or more processors to implement the monocular depth estimation method as described in any one of claims 1 to 7.

Images