A multi-interpretation degree deep estimation method and system of semantic conditional attention

Abstract

This invention relates to a multi-interpretive depth estimation method and system for semantically conditional attention, belonging to the fields of computer vision and robot perception technology. The method includes the following steps: S1, image and semantic instruction acquisition; S2, semantic instruction encoding; S3, semantically conditional attention encoding; S4, multi-interpretive depth decoding; and S5, geometric self-consistency constraint verification. This multi-interpretive depth estimation method and system for semantically conditional attention redefines depth as a task-related observation rather than a single physical constant, outputting multiple geometrically self-consistent depth interpretations for the same scene according to the intent. This fundamentally solves the deficiency of existing methods that "can only provide one interpretation" in transparent / reflective scenes. For example, a robotic arm can use penetration depth to grasp objects behind glass, and a mobile robot can use surface depth to avoid obstacles. Different interpretations of the same pixel serve different tasks and are all correct.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and robot perception technology, specifically to a multi-interpretation metric depth estimation method and system for semantically conditional attention, which is particularly suitable for scenarios such as embodied intelligence, mobile robots, autonomous driving, and AR / VR that require selecting different depth observations based on downstream task intent. Background Technology

[0002] Whether it's traditional binocular stereo matching, learned stereo matching, monocular depth estimation, or active depth sensing, they all share the implicit premise that each pixel in an image has one and only one correct depth value. This assumption does not hold true in many real-world scenarios—there is one depth from a transparent glass surface to the camera, and another depth from an object seen through the glass to the camera; both are physically correct and serve different tasks (obstacle avoidance requires surface depth, and object recognition requires penetration depth); mirrors themselves have depth, and the virtual image formed in the mirror corresponds to another depth; semi-transparent, fogged, and mesh-like multi-layered scenes also have multiple layers of depth superimposed along the same line of sight. Existing methods, no matter how much accuracy they improve, cannot simultaneously meet the needs of different tasks in these scenarios.

[0003] Existing methods generally degrade significantly in non-Lambertian scenarios such as glass, high reflectivity, and transparency. Advanced learning-based stereo matching exhibits errors orders of magnitude larger in these scenarios compared to conventional scenarios. The infrared light from active depth sensors mostly penetrates glass, making them "invisible" to glass doors and curtain walls, and creating voids in specular reflections. The industry generally attributes this to insufficient training data, but the more fundamental problem is that even with sufficient data, the existence of "multiple correct depths for the same pixel" cannot be expressed using the "single truth value" problem definition. Conditional generation methods that have emerged in recent years typically apply conditions to the output / decoding end rather than changing the model's geometric understanding of the input scene, and the output space is not physically constrained, lacking objective verification. Furthermore, upstream AI systems cannot schedule perception modules to acquire depths with different semantic meanings, making it difficult to establish a closed-loop perception-decision system.

[0004] Therefore, a method is needed that can break through the assumption that "each pixel has only one depth truth value", switch the geometric understanding of the same scene within the network according to semantic instructions, and output multiple geometrically consistent depth interpretations of the same scene on demand, so as to solve the problems that existing methods cannot output multiple interpretation depths according to task intent and that perception modules cannot be scheduled upstream. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides a multi-interpretive metric depth estimation method and system for semantic conditional attention. It has advantages such as multi-interpretive output, semantic layer switching of understanding mode, and scheduling by upstream AI. It solves the problems of unmet multi-task requirements under a single truth assumption, unreliable conditional generation due to lack of physical constraints, and unschedulable perception.

[0006] To achieve the above objectives, the present invention provides the following technical solution: A multi-interpretation metric depth estimation method for semantically conditional attention includes the following steps: S1. Image and semantic instruction acquisition: Acquire an image and a semantic instruction for the same scene, wherein the semantic instruction represents the intention to understand the depth of the scene; S2, Semantic instruction encoding: Encoding the semantic instruction into a semantic embedding; S3. Semantic Conditional Attention Encoding: The image is input into an attention-based encoding network, and the semantic embedding changes the attention calculation path of the encoding network, so that the same image obtains different feature representations under different semantic instructions; S4. Multi-interpretation deep decoding: The feature representation is decoded by the decoding network into a pixel-by-pixel metric depth interpretation corresponding to the semantic instruction; S5. Geometric self-consistency constraint verification: Apply a geometric self-consistency constraint to the metric depth interpretation to make it a metric depth that can be objectively measured with usable geometric precision; In the same scenario, different semantic instructions cause the method to output different depth measurement interpretations, and the different depth measurement interpretations include at least the target surface depth and the target penetration depth.

[0007] Furthermore, the semantic embedding alters the attention computation path by at least one of the following: injecting the semantic embedding as a conditional token into the self-attention sequence to participate in the interaction; generating attention bias, gating, or scaling coefficients from the semantic embedding to modulate the attention weights; modulating normalization or projection parameters from the semantic embedding; the alteration occurs at the attention layer of the encoding network, rather than only at the output of the decoding network.

[0008] Furthermore, the semantic instructions are taken from a predefined finite set of semantics, which includes at least two of the following: measuring the target surface, looking through a transparent object to see objects behind it, and removing reflections or glare.

[0009] Furthermore, the geometric self-consistency constraint includes at least one of multi-view geometric consistency constraint or single-view surface or hierarchical continuity and order relation constraint.

[0010] Furthermore, the depth measurement interpretation also includes de-reflection or de-glare depth interpretation; under the de-reflection semantic instruction, the attention of the encoding network focuses on the geometric consistency of the reflective area and suppresses specular highlights and virtual images, outputting the true surface depth measurement after removing the influence of reflection.

[0011] Furthermore, the method also includes reverting to a preset default depth interpretation and labeling the confidence level when the semantic instruction is missing or exceeds the predefined semantic set.

[0012] Furthermore, the semantic instruction is issued by the upstream world model or the vision-language-action model through a standardized interface, and the method returns a metric depth interpretation corresponding to the semantic instruction.

[0013] Furthermore, the method also accepts camera physical parameters as structured conditional inputs, which are jointly modulated for attention within the coding network via conditional channels different from the semantic embedding.

[0014] This invention also provides a multi-interpretation metric depth estimation system for semantically conditional attention, comprising: The image acquisition module is used to acquire images of the same scene; A semantic instruction input channel is used to receive semantic instructions and encode them into semantic embeddings; The conditional coding network, based on the attention mechanism, changes its attention computation path by the semantic embedding, so that the same image can obtain different feature representations under different semantic instructions; A multi-interpretation deep decoding network is used to decode the feature representation into a pixel-by-pixel metric depth interpretation corresponding to the semantic instruction; For the same scenario, different semantic instructions cause the system to output different geometrically consistent metric depth interpretations, and the different metric depth interpretations include at least the target surface depth and the target penetration depth.

[0015] Furthermore, it also includes a geometrically self-consistent constraint unit, which applies a geometric consistency constraint to each of the metric depth interpretations, making it an objectively measurable metric depth; it also includes a standardized interface for receiving semantic instructions from the upstream decision system; the conditional encoding network and the multi-interpretation depth decoding network are trained, and the training uses multiple sets of metric depth truth pairs corresponding to different semantic intentions in the same scene as supervision signals.

[0016] Compared with existing technologies, this invention provides a multi-interpretation metric depth estimation method and system for semantically conditional attention, which has the following beneficial effects: 1. This semantically conditional attention-based multi-interpretive depth estimation method and system redefines depth as a task-related observation rather than a single physical constant, and outputs multiple geometrically consistent depth interpretations (surface depth, penetration depth, and de-reflection depth) for the same scene according to the intent. This fundamentally solves the problem that existing methods can only provide one interpretation in transparent / reflective scenes. When a robotic arm grasps an object behind glass, it can use the penetration depth, and when a mobile robot avoids obstacles, it can use the surface depth. Different interpretations of the same pixel serve different tasks and are correct in their own way.

[0017] 2. This semantically conditional attention multi-interpretation metric depth estimation method and system enables semantic intent to change the attention computation path within the encoding network, allowing the same input image to generate different feature representations under different semantics, rather than simply switching or changing the generation style at the output end; each interpretation is subject to geometric self-consistency constraints, which is a physically valid metric depth that can be objectively verified with geometric precision, unlike general conditional generation which has an open output space and is evaluated based on subjective quality.

[0018] 3. This semantically conditional attention multi-interpretive metric depth estimation method and system accepts semantic intents from the upstream world model or VLA via a standardized interface and returns corresponding depth interpretations, making the depth perception module a programmable and schedulable perception node. The upstream system can call and switch interpretations multiple times according to the task progress, realizing a complete closed loop of "understanding intent → on-demand perception → serving decision", providing infrastructure for the integrated perception-decision of embodied intelligence systems. Attached Figure Description

[0019] Figure 1 This is a flowchart illustrating the process of a multi-interpretation metric depth estimation method for semantically conditional attention according to the present invention.

[0020] Figure 2 This is a diagram illustrating the overall architecture of a multi-interpretive metric depth estimation system for semantically conditional attention, as described in this invention.

[0021] Figure 3 This is a schematic diagram of the semantic conditional attention principle of the present invention.

[0022] Figure 4 This is a schematic diagram of the depth-related observations of the present invention.

[0023] Figure 5 This is a schematic diagram illustrating the difference between the present invention and general conditional generation.

[0024] Figure 6 This is a schematic diagram of the upstream scheduling interface of the present invention.

[0025] Figure 7 This is a schematic diagram illustrating the training data pairing of the present invention.

[0026] Figure 8 This is a visual illustration of the attention mode switching of the present invention.

[0027] Figure 9 This is a schematic diagram of the dual-navigation system for the glass door of the present invention.

[0028] Figure 10 This is a schematic diagram of the mirror / strong reflection de-reflection method of the present invention.

[0029] Figure 11 This is a schematic diagram illustrating the depth of task-based crawling in this invention. Detailed Implementation

[0030] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0031] Please see Figures 1 to 11 .

[0032] Example 1: System Overall Architecture and Workflow This embodiment provides a multi-interpretation metric depth estimation system for semantically conditional attention.

[0033] System components: Image acquisition module: used to acquire one or more frames of images of the same scene. In one embodiment, it is two RGB images (left and right images), but the present invention is not limited to binoculars, and can also be a single frame image or an image sequence.

[0034] Semantic instruction input channel: Receives semantic instructions, which express "what intention is used to understand the depth of the scene", such as "measuring the target surface", "seeing objects behind through transparent objects", "the real surface after removing reflections", etc. The semantic instructions are encoded into semantic embeddings, which can be achieved by a text encoder or a lookup table embedding of a finite semantic set.

[0035] Conditional coding networks: These are based on attention mechanisms (such as ViT-like self-attention networks). They tokenize images into sequences of visual tokens and compute features in the self-attention layer. The key is that semantic embedding, through a conditional modulation unit, alters the attention computation path of the self-attention layer—that is, it changes the distribution or calculation method of attention weights, causing the network to produce different feature representations of the same input image under different semantic instructions. This conditional intervention occurs within the attention layer of the coding network, not just at the output / decoding end.

[0036] Multi-interpretive deep decoding network: Based on a dense predictive decoding structure (such as DPT), it decodes conditional coding features into pixel-by-pixel metric deep interpretations corresponding to the semantic instruction.

[0037] Geometric self-consistency constraint unit: Constrains / verifies the geometric self-consistency of each depth interpretation under the semantic meaning (e.g., surface interpretation satisfies surface continuity and parallax consistency, penetration interpretation satisfies the geometric relationship of objects behind), so that the output is a physically valid depth measurement that can be objectively measured with geometric precision.

[0038] Upstream scheduling interface: Accepts semantic instructions issued by the upstream world model / VLA in a standardized interface form and returns the corresponding depth interpretation.

[0039] Workflow: The image acquisition module acquires scene images; The semantic instruction input channel receives semantic instructions and encodes them into semantic embeddings; The conditional coding network modulates the attention path based on semantic embeddings and outputs conditional features; The multi-interpretation deep decoding network decodes the corresponding metric depth interpretation; The geometrically self-consistent constraint unit applies a geometric consistency check to the output; The upstream scheduling interface returns a deep explanation and confidence level.

[0040] Example 2: The core mechanism of semantic conditional attention This embodiment details how semantic embedding alters the attention computation path of the encoding network.

[0041] Semantic embedding acquisition: Semantic instructions can come from a predefined finite semantic set (such as "look at the surface", "look through", "remove reflection"), which are mapped into semantic embeddings through table lookup or text encoder. This invention focuses on a scene-related finite semantic set to ensure controllability and verifiability.

[0042] Conditional modulation of the attention path: Semantic embeddings are applied to the self-attention layer of the encoding network via conditional modulation units, and the modulation methods include, but are not limited to, one or more of the following: (1) Inject semantic embeddings as additional tokens / conditional tokens into the self-attention sequence to participate in query-key-value interactions, thereby changing the attention weight distribution of each visual token; (2) Generate attention bias / gating / scaling coefficients from semantic embedding to modulate attention weights or feature channels; (3) Modulate the normalization layer parameters or linear projection parameters by semantic embedding to change the feature extraction method.

[0043] The differences in attention patterns under different semantics are shown in Table 1.

[0044] Table 1: The difference between this invention and output-end switching is that this invention does not perform post-processing switching on the same set of features at the decoding / output end, but rather changes the feature extraction method at the encoding attention layer—the features of the same image entering the decoder are different under different semantics. This is the key difference between this invention and "multiple output heads / post-processing selection".

[0045] Example 3: Geometric self-consistent constraints and their difference from general conditional generation This embodiment details the working principle of geometrically self-consistent constraint elements and their essential differences from general conditional generation methods.

[0046] Implementation of geometric self-consistency constraints: The depth output corresponding to each semantic interpretation is subject to geometric self-consistency constraints. In multi-view (e.g., binocular) embodiments, the depth under this interpretation is required to be consistent with the geometric relationship between the viewpoints—that is, the depth of the left image projected onto the right image should be consistent with the depth of the right image; in single-view embodiments, the interpretation is required to satisfy physical constraints such as the continuity and order of surfaces / layers—the surface depth should satisfy the surface smoothness, and the penetration depth should satisfy the depth order relationship of the objects behind.

[0047] Objective measurement verification: The depth value of each interpretation can be objectively measured with geometric precision (e.g., by comparison with laser ranging values), rather than relying on subjective visual evaluation. Thus, each interpretation is a geometrically valid and objectively measurable depth measurement, and the output space is physically constrained.

[0048] The differences from general conditional generation are shown in Table 2.

[0049] Table 2: Example 4: Dual-interpretation navigation application in a glass door scenario Scenario: A mobile robot approaches a glass door and needs to first detour / slow down to avoid crashing into the door (requiring depth to the glass surface), and then identify whether there is a target / pathway behind the door (requiring penetration depth).

[0050] Operating procedures: (1) The camera captures images of the glass door scene (one frame or two frames from binoculars).

[0051] (2) The upstream navigation system first issues a semantic command "measure the target surface". The semantic embedding changes the encoding attention path through the conditional modulation unit, so that the network focuses on the glass frame contour and surface structure features, decodes and outputs the glass surface measurement depth, and the robot decelerates and detours accordingly.

[0052] (3) Subsequently, the upstream sends a semantic instruction to “penetrate and see the object behind”. The same image is processed with different attention modes under the new semantics - the attention is shifted from the surface border to the geometric relationship of the object behind, and the penetration depth of the object behind the gate is decoded and output to identify the target behind and the passability.

[0053] (4) Both interpretations satisfy the geometric self-consistency constraint: the surface depth satisfies the continuity of the glass plane, and the penetration depth satisfies the depth relationship of the object behind it. They are both usable and do not conflict with each other.

[0054] Results: The same pixel on the glass door can obtain two correct depths: surface and penetration, which serve obstacle avoidance and recognition tasks respectively, solving the problems of traditional sensors "not being able to see the glass" or single interpretation methods "only being able to provide one depth".

[0055] Example 5: Application of de-reflection depth in mirror / highly reflective scenes Scenario: Environments containing mirrored surfaces, polished metal, and highly reflective surfaces, where reflective areas cause conventional methods to produce incorrect depth or voids.

[0056] Operating procedures: (1) Collect images containing reflective areas.

[0057] (2) The upstream sends a semantic instruction of “de-reflection / de-reflection”, and the conditional modulation makes the encoding attention focus on the geometric consistency constraint of the reflective area and suppress the feature response of specular highlights and virtual images.

[0058] (3) Decode the true surface depth after removing the influence of reflection.

[0059] Results: The reflective areas were obtained with geometrically consistent true depth, improving the usability of depth estimation for mirror / high-reflectivity scenes.

[0060] Example 6: As a sensing node that can be scheduled by upstream AI Scenario: In an embodied intelligent system, the world model / VLA serves as the upstream decision-making end, and this module acts as a callable deep perception tool.

[0061] Operating procedures: (1) After understanding the task, the upstream VLA issues a structured request (semantic intent + optional physical parameters) in the form of tool call / API.

[0062] (2) This module performs conditional encoding and decoding based on semantic intent and returns the metric depth + confidence of the corresponding semantic interpretation.

[0063] (3) VLA advances decisions based on the returned depth interpretation and can call and switch interpretations multiple times as needed (e.g., first call "surface depth" for obstacle avoidance planning, and then call "penetration depth" for target recognition).

[0064] Result: The perception module becomes a programmable and schedulable perception node, opening up the perception-decision closed loop and enabling the upstream AI system to actively schedule the perception process according to the task context.

[0065] Example 7: Degradation handling of semantic defaults / outbounds Scenario: The upstream did not provide semantic instructions, or issued unsupported semantics.

[0066] Operating procedures: (1) Module detection of semantic default or out-of-bounds (i.e. semantic instructions are not within the predefined finite semantic set).

[0067] (2) Return to the default interpretation (such as the target surface depth) and report with a lower confidence level.

[0068] Result: The system remains usable even under default / out-of-bounds conditions, ensuring its robustness and engineering availability.

[0069] Comparative Example 1: General Conditional Generation Method A general conditional generation method is adopted—conditions are applied to the decoding / output end, and the output style or content is changed through text conditions, but the attention path of the encoding network is not changed, and no geometric self-consistency constraints are imposed.

[0070] Comparison Results: In the glass door scenario, while general conditional generation methods can produce depth maps with different appearances at the output end, there is a lack of geometric consistency constraints between the outputs, making it impossible to guarantee that each output is a physically valid measurement of depth. When "penetration depth" is required, the output may lack a correspondence with the geometric relationship of the real objects behind, making it unusable for actual obstacle avoidance or grabbing planning.

[0071] Comparative Example 2: Single Truth Depth Estimation Method Using existing depth estimation methods, output one and only one depth value for each pixel.

[0072] Comparison Results: In the glass door scenario, existing methods can only provide either the surface depth or the penetration depth. If the surface depth is output, the robot cannot see the object behind the door; if the penetration depth is output, the robot may directly collide with the glass door. Neither method can simultaneously meet the dual requirements of obstacle avoidance and recognition.

[0073] Comparative Example 3: Multi-task multi-head deep network A deep network employing a shared encoder and multiple branches at the output end is used—different task branches share the same feature, with different decoding heads only set at the output end.

[0074] Comparison Results: Due to the different optimal feature extraction methods corresponding to different semantic intentions, the shared encoder cannot optimize feature representations for the two semantics of "surface" and "penetration" separately, resulting in lower accuracy for each branch compared to the dedicated model. This invention intervenes in the encoding attention layer through semantics, enabling the same image to be processed with different attention modes under different semantics, producing different feature representations and achieving better accuracy.

[0075] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0076] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A multi-interpretation metric depth estimation method for semantically conditional attention, characterized in that, Includes the following steps: S1. Image and semantic instruction acquisition: Acquire an image and a semantic instruction for the same scene, wherein the semantic instruction represents the intention to understand the depth of the scene; S2, Semantic instruction encoding: Encoding the semantic instruction into a semantic embedding; S3. Semantic Conditional Attention Encoding: The image is input into an attention-based encoding network, and the semantic embedding changes the attention calculation path of the encoding network, so that the same image obtains different feature representations under different semantic instructions; S4. Multi-interpretation deep decoding: The feature representation is decoded by the decoding network into a pixel-by-pixel metric depth interpretation corresponding to the semantic instruction; S5. Geometric self-consistency constraint verification: Apply a geometric self-consistency constraint to the metric depth interpretation to make it a metric depth that can be objectively measured with usable geometric precision; In the same scenario, different semantic instructions cause the method to output different depth measurement interpretations, and the different depth measurement interpretations include at least the target surface depth and the target penetration depth.

2. The multi-interpretation metric depth estimation method for semantically conditional attention according to claim 1, characterized in that, The semantic embedding that alters the attention computation path includes at least one of the following: injecting the semantic embedding as a conditional token into the self-attention sequence to participate in the interaction; The semantic embedding generates attention bias, gating, or scaling coefficients to modulate attention weights; The change is determined by the semantic embedding modulation normalization or projection parameters; the change occurs at the attention layer of the encoding network, not just at the output of the decoding network.

3. The multi-interpretation metric depth estimation method for semantically conditional attention according to claim 1, characterized in that, The semantic instructions are taken from a predefined finite set of semantics, which includes at least two of the following: measuring the target surface, looking through a transparent object to see objects behind it, and removing reflections or glare.

4. The multi-interpretation metric depth estimation method for semantically conditional attention according to claim 1, characterized in that, The geometric self-consistency constraint includes at least one of multi-view geometric consistency constraint or single-view surface or hierarchical continuity and order relation constraint.

5. The multi-interpretation metric depth estimation method for semantically conditional attention according to claim 1, characterized in that, The depth measurement interpretation also includes de-reflection or de-glare depth interpretation; under the de-reflection semantic instruction, the attention of the encoding network focuses on the geometric consistency of the reflective area and suppresses specular highlights and virtual images, outputting the true surface depth measurement after removing the influence of reflection.

6. The multi-interpretation metric depth estimation method for semantically conditional attention according to claim 1, characterized in that, The method further includes reverting to a preset default depth interpretation and labeling the confidence level when the semantic instruction is missing or exceeds the predefined semantic set.

7. The multi-interpretation metric depth estimation method for semantically conditional attention according to claim 1, characterized in that, The semantic instruction is issued by the upstream world model or the vision-language-action model through a standardized interface, and the method returns a metric depth interpretation corresponding to the semantic instruction.

8. The multi-interpretation metric depth estimation method for semantically conditional attention according to claim 1, characterized in that, The method also accepts camera physical parameters as structured conditional inputs, which are jointly modulated for attention within the coding network via conditional channels different from the semantic embedding.

9. A multi-interpretation metric depth estimation system for semantically conditional attention, characterized in that, include: The image acquisition module is used to acquire images of the same scene; A semantic instruction input channel is used to receive semantic instructions and encode them into semantic embeddings; The conditional coding network, based on the attention mechanism, changes its attention computation path by the semantic embedding, so that the same image can obtain different feature representations under different semantic instructions; A multi-interpretation deep decoding network is used to decode the feature representation into a pixel-by-pixel metric depth interpretation corresponding to the semantic instruction; In this context, for the same scenario, different semantic instructions cause the system to output different geometrically consistent metric depth interpretations, and the different metric depth interpretations include at least the target surface depth and the target penetration depth.

10. A multi-interpretation metric depth estimation system for semantically conditional attention according to claim 9, characterized in that: It also includes a geometrically self-consistent constraint unit, which applies a geometric consistency constraint to each of the metric depth interpretations, making it an objectively measurable metric depth; It also includes a standardized interface for receiving semantic instructions from upstream decision-making systems; The conditional coding network and the multi-interpretation deep decoding network are trained, and the training uses multiple sets of metric depth truth pairs corresponding to different semantic intentions in the same scene as supervision signals.

Images