Embodied intelligence oriented depth map prediction method and device for hintable interaction

Abstract

The application discloses a depth map prediction method and device for embodied intelligence, comprising: obtaining training images, depth map true values and instance mask true values thereof; performing visual coding on the training images to obtain dense visual embedding; performing prompt coding on the interaction prompt to obtain interaction prompt representation; constructing task labels; generating post-interaction sparse representation and post-interaction dense representation according to the dense visual embedding, the interaction prompt representation and the task labels; performing matching operation on the post-interaction sparse representation and the post-interaction dense representation to generate depth map prediction values; and updating the target promptable depth map prediction model according to the depth map prediction values, the depth map true values and the instance mask true values, combining the to-be-processed images, the interaction prompt and the task labels to obtain target depth map results, which are used to realize the control of the embodied intelligence system. The application can improve the accuracy of the depth map prediction results and can be widely applied to the technical field of image data processing.

Description

Technical Field

[0001] This invention relates to the field of image data processing technology, and in particular to a depth map prediction method and apparatus with prompting interaction for embodied intelligence. Background Technology

[0002] With the rapid development of embodied intelligence and robotics, the vision systems of embodied robots not only need to reliably understand the geometry of complex 3D environments, but also need to output instance-level results with low latency to support real-time decision-making in tasks such as object interaction and target tracking. However, existing 3D vision methods still have problems in embodied scenarios: on the one hand, most geometric perception models often rely on multi-frame aggregation, iterative optimization, or offline inference processes, which are difficult to meet the requirements of "minimum prediction and real-time response" for online interaction; on the other hand, in order to cover multiple capabilities such as depth estimation, instance perception, and tracking, common practices usually adopt expert-style combinations of multi-head decoding or multi-module cascading, resulting in repeated calculations and system redundancy, further increasing deployment costs and inference latency, and limiting the generalization ability and engineering implementation of existing methods in real-world applications of embodied intelligence. Summary of the Invention

[0003] In view of this, the main objective of the embodiments of the present invention is to provide a depth map prediction method and apparatus with prompting interaction for embodied intelligence, in order to solve at least one of the problems of the prior art. The present invention can improve the accuracy of depth map prediction results.

[0004] To achieve the above objectives, one aspect of the present invention provides a depth map prediction method with promptable interaction for embodied intelligence, the method comprising: Acquire training images, ground truth depth maps, and instance mask ground truth values ​​of the training images, and input the training images into an initial cueable depth map prediction model; the initial cueable depth map prediction model includes a visual encoder, a cue encoder, and an interactive decoder; The training images are visually encoded using the visual encoder to obtain dense visual embeddings. The interactive prompts are encoded using the prompt encoder to obtain an interactive prompt representation. Based on the task type of the current depth prediction task, construct task tags; Using the pre-built interaction decoder, post-interaction sparse representations and post-interaction dense representations are generated based on the dense visual embedding, the interaction cue representation, and the task tag. A matching operation is performed on the sparse representation after interaction and the dense representation after interaction to generate the depth map prediction value for the current depth prediction task. Based on the depth map prediction value, the depth map ground value, and the instance mask ground value, the initial cueable depth map prediction model is updated to obtain the target cueable depth map prediction model. The image to be processed, the interactive prompts, and the task tags corresponding to the target depth prediction task are input into the target promptable depth map prediction model to obtain the target depth map result of the target depth prediction task; the target depth map result is used to realize the control of the embodied intelligence system.

[0005] In some embodiments, obtaining the training image, the ground truth depth map value, and the ground truth instance mask value of the training image, and inputting the training image into an initial cueable depth map prediction model, includes the following steps: Generate the synthetic environment and camera trajectory using a pre-set data engine; The offline rendering pipeline renders the compositing environment and the camera trajectory, outputs a color image, and simultaneously exports multi-channel ground truth values ​​that are pixel-level aligned with the color image; the multi-channel ground truth values ​​include the depth map ground truth value, the instance mask ground truth value, the optical flow ground truth value, and the camera pose. Based on the color images and the multi-channel ground truth values, a training dataset is constructed; Color images are selected from the training dataset as the training images; Obtain the ground truth value of the depth map and the ground truth value of the instance mask corresponding to the training image, and input the training image into the initial cueable depth map prediction model.

[0006] In some embodiments, obtaining the training image includes the following steps: Select a frame of the color image from the training dataset to obtain the training image belonging to the monocular scene; Alternatively, from the training dataset, within the same sequence of color images, the color images are sampled at random intervals to obtain the training images belonging to the binocular scene; The random interval is a sampling interval randomly selected within a preset interval range.

[0007] In some embodiments, the step of encoding the interactive prompt using the prompt encoder to obtain an interactive prompt representation includes the following steps: When the interactive prompt is a masked prompt, the masked prompt is input into the prompt encoder to obtain a dense prompt representation; When the interactive prompt is a dot prompt, the dot prompt is input into the prompt encoder to obtain a sparse prompt representation.

[0008] In some embodiments, before generating post-interaction sparse and post-interaction dense representations based on the dense visual embedding, the interaction cue representation, and the task tag using the pre-built interaction decoder, the method further includes the following steps: When the interactive cue representation is a dense cue representation, the dense cue representation is injected into the first dense feature of the dense visual embedding to obtain the updated first dense feature, and the task label is switched to the target tracking label. The task label is then concatenated with the empty cue representation to obtain the sparse input label. When the interactive prompt representation is a sparse prompt representation, the task tag is switched to an instance query tag, and the task tag is concatenated with the sparse prompt representation to obtain a sparse input tag.

[0009] In some embodiments, generating post-interaction sparse and post-interaction dense representations using the pre-built interaction decoder based on the dense visual embedding, the interaction cue representation, and the task tagging includes the following steps: When the training image belongs to a binocular scene, the dense visual embedding is a two-way dense feature. The two-way dense features are input into the dense bidirectional block of the interactive decoder. The two-way dense features are jointly inferred through attention to output an aligned dense representation. The two-way dense features include the first-way dense feature and the second-way dense feature. When the training image belongs to a monocular scene, the dense visual embedding is a single-path dense feature. The single-path dense feature is reused as a second-path dense feature to obtain two-path dense features. The two-path dense features are input into the dense bidirectional block of the interactive decoder. The two-path dense features are jointly inferred through attention to output an aligned dense representation. The single-path dense feature includes the first-path dense feature. The sparse input tag and the aligned dense representation are interactively decoded using the sparse bidirectional block of the interactive decoder to obtain the interactive sparse representation and the interactive dense representation.

[0010] In some embodiments, the step of performing a matching operation between the post-interaction sparse representation and the post-interaction dense representation to generate the depth map prediction value for the current depth prediction task includes the following steps: Obtain the task-related sparse representation from the sparse representation after the interaction; Based on the dense representation after the interaction, obtain the task-related dense representation; The task-related sparse representation and the task-related dense representation are matched along the channel dimension to generate the depth map prediction value.

[0011] In some embodiments, before updating the initial cueable depth map prediction model based on the depth map prediction value, the depth map ground truth value, and the instance mask ground truth value to obtain the target cueable depth map prediction model, the method further includes the following steps: Based on the panoramic depth prediction value and the corresponding depth map ground truth value, the basic regression loss is obtained; Based on the basic regression loss, the pre-trained cueable depth map prediction model is pre-trained in the first stage to obtain the initial cueable depth map prediction model. Feature extraction is performed on the preset network layer of the initial suggestible depth map prediction model to obtain geometric reference features; Wherein, when the task type is panoramic depth prediction, the depth map prediction value includes the panoramic depth prediction value.

[0012] In some embodiments, updating the initial cueable depth map prediction model based on the depth map prediction value, the depth map ground truth value, and the instance mask ground truth value to obtain the target cueable depth map prediction model includes the following steps: Based on the instance depth prediction value or the tracking depth prediction value, and based on the corresponding true value of the depth map, obtain the basic regression loss; Construct pixel weights associated with the truth value of the instance mask; Based on the instance depth prediction value or the tracking depth prediction value, and based on the corresponding depth map ground truth value and the pixel weight, obtain the instance label distribution smoothing loss; Feature extraction is performed on the preset network layer of the initial suggestible depth map prediction model to obtain the current features; Obtain the first gram matrix of the geometric reference feature, and obtain the second gram matrix of the current feature; Obtain the Frobenius norm between the first Gram matrix and the second Gram matrix to obtain the Gram anchoring regularization loss; The joint loss is obtained based on the basic regression loss, the instance label distribution smoothing loss, and the Gram anchoring regularization loss; Based on the joint loss, the initial cueable depth map prediction model is jointly trained in the second stage to obtain the target cueable depth map prediction model. Wherein, when the task type is instance depth prediction, the depth map prediction value includes the instance depth prediction value; when the task type is tracking depth prediction, the depth map prediction value includes the tracking depth prediction value.

[0013] To achieve the above objectives, another aspect of the present invention proposes a depth map prediction device with promptable interaction for embodied intelligence, the device comprising: The data construction module is used to acquire training images and ground truth values ​​of depth maps and instance masks of the training images, and input the training images into an initial cueable depth map prediction model; the initial cueable depth map prediction model includes a visual encoder, a cue encoder, and an interactive decoder. A visual encoding module is used to perform visual encoding on the training image through the visual encoder to obtain a dense visual embedding; The prompt encoding module is used to encode the interactive prompts through the prompt encoder to obtain an interactive prompt representation; The task tagging setting module is used to predict the task type based on the current depth and construct task tags; An interaction decoding module is used to generate a sparse representation and a dense representation after interaction based on the dense visual embedding, the interaction cue representation, and the task tag, using the pre-built interaction decoder. The first depth map prediction module is used to perform matching operations on the sparse representation after interaction and the dense representation after interaction to generate the depth map prediction value of the current depth prediction task. The loss constraint and iterative training module is used to update the initial cueable depth map prediction model based on the depth map prediction value, the depth map ground truth value, and the instance mask ground truth value, so as to obtain the target cueable depth map prediction model. The second depth map prediction module is used to input the image to be processed, the interactive prompts, and the task markers corresponding to the target depth prediction task into the target promptable depth map prediction model to obtain the target depth map result of the target depth prediction task; the target depth map result is used to realize the control of the embodied intelligence system.

[0014] To achieve the above objectives, another aspect of the present invention provides an electronic device, the electronic device including a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method described above.

[0015] To achieve the above objectives, another aspect of the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the methods described above.

[0016] To achieve the above objectives, another aspect of the present invention provides a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium and execute the computer instructions to cause the computer device to perform the aforementioned method.

[0017] The embodiments of the present invention include at least the following beneficial effects: The present invention provides a depth map prediction method and apparatus for embodied intelligence with promptable interaction. This scheme obtains training images and the ground truth values ​​of the depth maps and instance masks of the training images to provide training data for model training, and inputs the training images into an initial promptable depth map prediction model. The initial promptable depth map prediction model includes a visual encoder, a prompt encoder, and an interaction decoder. The visual encoder performs visual encoding on the training images to obtain dense visual embeddings, providing a solid foundation for subsequent decoding and interaction. The prompt encoder performs prompt encoding on interactive prompts to obtain interactive prompt representations, transforming diverse user interactions (such as dots and masks) into unified features understandable to the model, achieving a natural and flexible human-computer interaction interface. Based on the task type of the current depth prediction task, task tags are constructed, and learnable task tags explicitly indicate the task type, enabling the model to decouple and accurately respond to different tasks. The system enables zero-overhead task switching by implementing task instructions. Through a pre-built interactive decoder, it generates post-interaction sparse and dense representations based on dense visual embeddings, interactive cue representations, and task tags, achieving deep fusion and alignment of visual features, interactive cue, and task instructions. Matching operations are performed on the post-interaction sparse and dense representations to generate the depth map prediction value for the current depth prediction task. A unified dot product matching head replaces multiple independent output heads, simplifying the model structure and improving computational efficiency and consistency. The initial cueable depth map prediction model is updated based on the depth map prediction value, the ground truth depth map value, and the instance mask ground truth value to obtain the target cueable depth map prediction model, improving the model's depth map prediction accuracy. The image to be processed, interactive cue, and the task tag corresponding to the target depth prediction task are input into the target cueable depth map prediction model to obtain a high-precision target depth map result for the target depth prediction task, effectively supporting the control tasks of downstream embodied intelligence systems. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a flowchart of a depth map prediction method with prompting interaction for embodied intelligence provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of the hardware structure of the electronic device provided in an embodiment of the present invention. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with those of this invention; they are merely examples of apparatuses and methods consistent with some aspects of the embodiments of this invention as detailed in the appended claims.

[0021] It should be noted that although functional modules are divided in the system diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the system or the order in the flowchart. The terms "first / S100" and "second / S200" in the specification, claims, and the foregoing drawings may be used herein to describe various concepts, but unless specifically stated otherwise, these concepts are not limited by these terms. These terms are used only to distinguish one concept from another. For example, first information may also be referred to as second information without departing from the scope of the embodiments of the invention, and similarly, second information may also be referred to as first information. Depending on the context, the words "if" or "when" as used herein may be interpreted as "when," "in response to a determination," or "in the event of a determination."

[0022] The terms “at least one,” “multiple,” “each,” “any,” etc., used in this invention, “at least one” includes one, two, or more than two; “multiple” includes two or more than two; “each” refers to each of the corresponding multiple; and “any” refers to any one of the multiple.

[0023] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein is for the purpose of describing embodiments of the invention only and is not intended to limit the invention.

[0024] Before providing a detailed description of the embodiments of the present invention, some of the nouns and terms involved in the embodiments of the present invention will be explained first. The nouns and terms involved in the embodiments of the present invention are subject to the following interpretations.

[0025] Embodied intelligence is a cutting-edge field at the intersection of artificial intelligence and robotics. It emphasizes that intelligent agents achieve autonomous learning and evolution through dynamic interaction between their bodies and the environment. Its core lies in the deep integration of perception, action, and cognition.

[0026] The Movie Render Queue (MRQ) is a powerful tool in Unreal Engine used to generate high-quality offline renders.

[0027] Label Distribution Smoothing (LDS) is a technique used to address the problem of continuous label imbalance, aiming to improve the generalization ability and prediction performance of a model by smoothing the label distribution.

[0028] Among related technologies, there are models that rely on multi-frame aggregation, iterative optimization, or offline inference processes to complete geometry perception. These models are difficult to meet the low-latency response requirements of real-time interactive applications. At the same time, in order to take into account multiple functions such as depth estimation, instance perception, and target tracking, multiple dedicated modules or decoding heads are usually required. This design leads to significant model redundancy and repeated computation, which not only increases deployment costs and inference latency, but also restricts practical application and generalization capabilities in resource-constrained embodied intelligence scenarios.

[0029] In view of this, this invention provides a depth map prediction method and apparatus for embodied intelligence with prompting interaction. This scheme uses a single feedforward neural network to unify the visual needs of various embodied intelligence such as scene geometric understanding, instance interaction and target tracking into a dense prediction form of "depth map", and achieves on-demand output in a single model through task labeling and prompting information. For example, the method supports single-frame or dual-frame corresponding images at the input end, extracting dense visual embeddings through a visual encoder with shared weights; simultaneously, it introduces a cue encoding mechanism, encoding dot cues and mask cues into sparse cue representations and dense cue representations respectively, and constructs task labels covering monocular, binocular, instance query, and target tracking to eliminate output uncertainty under homologous image embedding; subsequently, it adopts a cascaded bidirectional Transformer structure to perform latent geometric alignment of dense features across frames / views, and injects the "task label-cue interaction" condition into the dense representation in the sparse-dense bidirectional interaction; finally, it uses task-related sparse representations to query dense features, and generates the depth map corresponding to the target depth prediction task through a dot-product unified output head, realizing unified modeling and inference without the need for multi-branch prediction heads.

[0030] The depth map prediction method for embodied intelligence with prompting interaction provided in this embodiment of the invention relates to the fields of image data processing and artificial intelligence. This method can be applied to terminals, servers, or software running on either a terminal or server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, or in-vehicle terminal, but is not limited to these. The server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The server can also be a node server in a blockchain network. The software can be an application implementing the depth map prediction method for embodied intelligence with prompting interaction, but is not limited to the above forms.

[0031] Figure 1 This is an optional flowchart of a depth map prediction method with prompting interaction for embodied intelligence provided in an embodiment of the present invention. Figure 1 The method may include, but is not limited to, steps S100 to S800: Step S100: Obtain the training image and the ground truth depth map and instance mask of the training image, and input the training image into the initial cueable depth map prediction model; the initial cueable depth map prediction model includes a visual encoder, a cue encoder and an interactive decoder. Step S200: Visually encode the training images using a visual encoder to obtain dense visual embeddings; Step S300: The interactive prompt is encoded using a prompt encoder to obtain an interactive prompt representation; Step S400: Based on the current depth, predict the task type and construct task tags; Step S500: Using a pre-built interaction decoder, generate post-interaction sparse representation and post-interaction dense representation based on dense visual embedding, interaction cue representation and task labeling. Step S600: Perform a matching operation on the sparse representation after interaction and the dense representation after interaction to generate the depth map prediction value for the current depth prediction task. Step S700: Update the initial cueable depth map prediction model based on the depth map prediction value, the depth map ground truth value, and the instance mask ground truth value to obtain the target cueable depth map prediction model. In step S800, the image to be processed, interactive prompts, and the task label corresponding to the target depth prediction task are input into the target prompt depth map prediction model to obtain the target depth map result of the target depth prediction task; the target depth map result is used to realize the control of the embodied intelligence system.

[0032] In step S100 of some embodiments, a large-scale training data is generated using a synthetic data engine based on Unreal Engine and an offline rendering pipeline. Supervisory signals such as depth, instance mask, optical flow, and camera pose, which are strictly aligned to the pixel level of the rendered image, are simultaneously exported and organized into a training dataset that supports monocular, binocular, and interactive tasks. In the constructed instance-geometry training dataset, training images are selected, and the ground truth depth map value and instance mask value of the training images are obtained. The training images are then input into an initial cueable depth map prediction model to train the initial cueable depth map prediction model. Furthermore, the cueable depth map prediction model used in this embodiment is a single feedforward neural network, including a visual encoder, a cue encoder, and an interactive decoder.

[0033] In some embodiments, step S100 may include, but is not limited to, steps S110 to S150: Step S110: Generate the synthetic environment and camera trajectory using a preset data engine; Step S120: Render the compositing environment and camera trajectory through the offline rendering pipeline, output a color image, and simultaneously export multi-channel ground truth values ​​that are pixel-level aligned with the color image; the multi-channel ground truth values ​​include depth map ground truth values, instance mask ground truth values, optical flow ground truth values, and camera pose. Step S130: Construct a training dataset based on the color images and multi-channel ground truth. Step S140: Select color images from the training dataset as training images; Step S150: Obtain the ground truth depth map value and instance mask value corresponding to the training image, and input the training image into the initial cueable depth map prediction model.

[0034] In steps S110 to S120 of some embodiments, a synthetic data engine for data generation is constructed. Optionally, the data engine is based on Unreal Engine 5 to generate the compositing environment and camera trajectory. Furthermore, it outputs color images and simultaneously exports multi-channel ground truth values ​​via an offline rendering pipeline. Optionally, the offline rendering pipeline utilizes the Movie Render Queue (MRQ) in Unreal Engine to achieve high-quality offline rendering, and while rendering high-fidelity images, it simultaneously exports multiple supervised data channel ground truth values ​​that are pixel-perfectly aligned with the image to support instance-geometric joint learning.

[0035] In step S110 of some embodiments, a compositing environment is created using a preset data engine. Modifiable rendering and sampling parameters are configured within the compositing environment, and the injection of dynamic pedestrians, vehicles, and other moving targets is supported to enhance data diversity. The set of compositing environments is set as follows: ,in, Indicates the first A synthetic environment; An index representing the synthesis environment; Indicates the total number of synthetic environments. It covers indoor / outdoor / urban / forest and dynamic scenes. The camera's intrinsic parameters are... ,frame The camera pose is ,in, Represents the rotation matrix; Representing a frame The translation vector; Represents a three-dimensional special Euclidean group. (in frames) Organize sequence sampling and generate camera trajectories using both UAV-based trajectory and random path methods. Thus, the camera pose sequence is obtained. This improves the coverage of the viewing angle and motion distribution.

[0036] In step S120 of some embodiments, MRQ is used to render each synthetic environment and camera trajectory, outputting a color image, and simultaneously exporting a multi-channel truth set aligned pixel-level with the color image. The multi-channel ground truth set includes at least the depth map ground truth, the instance mask ground truth, the optical flow ground truth, and the camera pose, and can be denoted as: ; In the formula, Representing a frame Color images; Representing a frame A multi-channel truth set; Indicates the rendering function; Indicates the synthetic environment; This refers to the video rendering queue, indicating the high-quality offline rendering pipeline used and its parameter configuration. Representing a frame The instance mask truth value; Representing a frame The ground truth of the depth map; Represents the optical flow truth value, representing the value from the frame. to frame Motion vector for each pixel.

[0037] Since the multi-channel ground truth and the color image are rendered from the same source, they satisfy the geometric constraints of the pinhole camera model. For any pixel... There are pixels The true value of the depth map at that location is .in, Represents pixels Column index in the image coordinate system; Represents pixels Row index in the image coordinate system; Indicates scale components; This indicates a transpose operation. Based on pixel points. and pixels ground truth of depth map at location The three-dimensional point in the camera coordinate system is The expression is as follows: ; This 3D point can change its pose according to adjacent frames. Projected onto the next frame, the theoretical projected pixel coordinates are obtained. , can be represented as .in, Indicates from frame to frame rotation matrix; Indicates from frame to frame The translation vector. The theoretical optical flow defined from this. The true value of optical flow directly output by rendering Completely identical, that is In principle, this ensures pixel-perfect alignment among all truth channels.

[0038] In step S130 of some embodiments, the rendered color image and multi-channel ground truth are used to organize the monocular (single frame) / stereo (dual frame) training samples into a training dataset according to the input format required for model training. For example, the training dataset... Represented as: ; In the formula, Representing a frame Color images; Representing a frame The ground truth of the depth map; Representing a frame The instance mask truth value; Representing a frame The camera pose.

[0039] By rendering over 100 distinct synthetic environments on a large scale, a massive training dataset containing over 10 million unique object instances was ultimately constructed. This training dataset is primarily at 1920×1080 resolution, with some data reaching 3840×2160 (4K) resolution, providing a high-quality and diverse data foundation for model training.

[0040] In steps S140 to S150 of some embodiments, the constructed training dataset is... The color images corresponding to a single frame or two frames are selected as training images. These training images are input into an initial cue-enabled depth map prediction model for training. The ground truth depth map values ​​and instance mask values ​​corresponding to the training images are used for joint loss calculation during model training. Optionally, the color images corresponding to two frames are denoted as... ,and ,in, This represents the first color image in the sequence; This represents the second color image in the sequence; Represents the real number field; Indicates the width of the image; Indicates the height of the image; This represents the number of channels in the image, corresponding to the standard RGB three channels (red, green, and blue). When only a single frame of the training image is input... In a monocular scene, when inputting two-frame training images... The scenario is a binocular scene. To ensure the stability of feedforward inference and the consistency of implementation, the input training images are preprocessed, which may include, but is not limited to, normalization, scaling, and necessary cropping / padding. For example, the short side of the image is scaled to a fixed scale to meet the input requirements of the visual encoder and reduce the distribution drift caused by resolution changes.

[0041] In the step of acquiring training images, a single color image can be selected from the training dataset to obtain a training image belonging to the monocular scene; or, from the training dataset, color images within the same sequence are sampled at random intervals to obtain a training image belonging to the binocular scene; where the random interval is a sampling interval randomly selected within a preset interval range. For example, during the model training phase, training images are acquired by constructing input samples in two ways to support different learning objectives. For monocular scenes, any single color image is directly selected from the training dataset as input; for binocular scenes, samples are randomly selected from the same rendering sequence within a preset interval range. A random sampling interval is selected within the range to obtain the random interval. and at that random interval Sample the color image to obtain two frames. Image pairs This random interval sampling strategy automatically generates training pairs with diverse baseline lengths and motion patterns to simulate different parallax and scene variations. All image pairs are accompanied by pixel-aligned multi-channel ground truth generated by synthetic rendering, providing the model with accurate multi-view geometric supervision signals to learn robust cross-frame correspondences and 3D geometric understanding capabilities.

[0042] In step S200 of some embodiments, dense visual embeddings can be obtained by visually encoding the input training images using a weighted visual encoder. Optionally, in practical applications, a network with robust representation capabilities (e.g., a DINO network) can be selected as the visual encoder to improve feature generalization. For example, a symmetric visual encoder is pre-configured. The two visual encoders share completely. Through these two visual encoders, the input training image is mapped to dense visual features, as expressed by: ; ; In the formula, Indicates dense visual embedding. , For the width of dense visual embedding, For the height of dense visual embedding, The number of channels for dense visual embedding; The first dense feature represents dense visual embedding; This represents the second dense feature path of the dense visual embedding. When the input training image is a stereo scene, the resulting dense visual embedding consists of two dense features; when the input training image is a monocular scene, the resulting dense visual embedding consists of a single dense feature path. Dense features from the same frame can be reused as the second dense feature path, thus unifying the monocular and stereo input formats in subsequent processing without changing the network structure.

[0043] In step S300 of some embodiments, interactive prompts are introduced to achieve unified reasoning for interactive and multi-task scenarios. Interactive prompts can include dot prompts. (queried points) (e.g., the set of coordinates of the target point clicked by the user) and mask hints (queried masks) (e.g., target instance mask or trace initialization mask). An interactive cues are mapped to interactive cue representations using a cue encoder.

[0044] In some embodiments, step S300 may include, but is not limited to, steps S310 to S320: Step S310: When the interactive prompt is a masked prompt, input the masked prompt into the prompt encoder to obtain a dense prompt representation; Step S320: When the interactive prompt is a dot prompt, input the dot prompt into the prompt encoder to obtain a sparse prompt representation.

[0045] In step S310 of some embodiments, when the input interactive prompt is a masked prompt... At that time, the native cue encoder using the SAM model was employed. Map masked hints to dense hint representations The expression is: .

[0046] In step S320 of some embodiments, when the input interactive prompt is a dot prompt... At that time, the native cue encoder using the SAM model was employed. Mapping masked hints to sparse hint representations The expression is: .

[0047] In step S400 of some embodiments, a task tag is introduced. (Task tokens) allow for the decoupling of desired outputs from different tasks within a single interactive decoder. Task tokens are set as a set of learnable sparse embeddings used to explicitly indicate the task type of the current depth prediction task. Optionally, task tokens are categorized into four types based on task type: 1) monocular token; 2) stereo token; 3) instance queried token; and 4) tracked token. The task type must include at least one of panoramic depth prediction, instance depth prediction, and tracking depth prediction.

[0048] Before step S500 in some embodiments, the depth map prediction method for embodied intelligence-oriented prompting interaction further includes: when the interaction prompt representation is a dense prompt representation, injecting the dense prompt representation into the first dense feature of the dense visual embedding to obtain the updated first dense feature, switching the task label to the target tracking label, and concatenating the task label with the empty prompt representation to obtain the sparse input label; when the interaction prompt representation is a sparse prompt representation, switching the task label to the instance query label, and concatenating the task label with the sparse prompt representation to obtain the sparse input label.

[0049] In some alternative embodiments, the task is marked. Sparse cue representation Concatenate to form sparse input tags The calculation formulas used include: This strategy is used to clarify the task type through task labeling in the context of homologous image embedding, and guide the model to focus on specific regions by combining sparse cue representations, thereby significantly reducing the uncertainty of depth map prediction. Furthermore, different tasks can be switched within the same architecture simply by replacing the task label, achieving task switching without additional computational overhead and reducing computational redundancy. Optionally, when the input interactive cue is a masked cue, the cue encoding results in dense cue features. In this case, to construct sparse input labels, an empty cue representation needs to be constructed as a placeholder. The task label is then concatenated with this empty cue representation to form the sparse input label. The formulas used include: Empty prompt representation It can be an all-zero vector or a learnable null embedding. For example, an all-zero vector can be constructed to explicitly represent the null cue representation; or a trainable parameter can be assigned to the null cue, and the value of this parameter can be learned and adjusted autonomously during model training to better suit the implicit requirements of the current deep prediction task for null cue representation.

[0050] In some optional embodiments, the injection method for interactive cues varies depending on the type of interactive cue representation. For masked cue injection used for tracking, when dense cue representations exist... Before entering the interactive decoder, dense cue representations are injected into the dense feature branch, enabling the model to focus on the tracked object region. For example, the dense cue representations are fused with the first-path dense features of the dense visual embedding via Hadamard product, and the task label is switched to the target tracking label to form the conditional features required for tracking depth prediction. The calculation formulas used include: ; In the formula, This represents the Hadamard product operation; This represents the first dense feature after the update; Indicates a target tracking marker.

[0051] Furthermore, when constructing sparse input tokens, an empty cue representation needs to be filled in. As placeholders, thus maintaining the uniformity of the sparse input tag structure, there are sparse input tags at this time. .

[0052] For point hint injection used for instance queries, sparse hint representation exists when only point hints are given. At that time, switch the task tag to the instance query tag. sparse cue representation Concatenate with instance query tags to form sparse input tags Furthermore, the sparse input label is input into the interactive decoder, which relies on the sparse input label to guide the model to focus on the instance region, and the final output is controlled by the task label.

[0053] In step S500 of some embodiments, an interactive decoder with a cascaded bidirectional Transformer decoding structure is constructed. This cascaded bidirectional Transformer decoding structure is composed of dense bidirectional blocks and sparse bidirectional blocks connected in series. The dense bidirectional blocks are responsible for joint inference on the dense visual embeddings in the binocular scene to obtain a dense representation after latent geometric alignment. The sparse bidirectional blocks are responsible for processing the interactive cue representations guided by task markers and interactive cues, injecting sparse input markers into the aligned dense representations to form task-related post-interaction sparse representations and post-interaction dense representations. The dense visual embeddings and the sparse input markers formed by concatenating the interactive cue representations and task markers are used as inputs to the interactive decoder. Through dense-sparse bidirectional interaction, cross-view / cross-frame alignment and task-related information fusion are completed, outputting task-related post-interaction sparse representations and post-interaction dense representations for direct use in subsequent depth map prediction.

[0054] In some embodiments, step S500 may include, but is not limited to, steps S510 to S530: Step S510: When the training image belongs to a binocular scene, the dense visual embedding is divided into two dense features. The two dense features are input into the dense bidirectional block of the interactive decoder. The two dense features are jointly inferred through attention, and the aligned dense representation is output. The two dense features include the first dense feature and the second dense feature. Step S520: When the training image belongs to a monocular scene, the dense visual embedding is a single-path dense feature. The single-path dense feature is reused as a second-path dense feature to obtain two-path dense features. The two-path dense features are input into the dense bidirectional block of the interactive decoder. The two-path dense features are jointly inferred through attention to output an aligned dense representation. The single-path dense feature includes the first-path dense feature. Step S530: Through the sparse bidirectional block of the interactive decoder, interactive decoding is performed on the sparse input tag and the aligned dense representation to obtain the interactive sparse representation and the interactive dense representation.

[0055] In step S510 of some embodiments, when the training image belongs to a binocular scene, dense visual embedding is performed. The two dense feature paths are input into the dense bidirectional block of the interactive decoder. Within the dense bidirectional block, joint inference is performed on the two dense feature paths through attention, outputting an aligned dense representation. The formulas used include: ; In the formula, This indicates a dense alignment characteristic. For the first path, align dense embedding, For second-path alignment and dense embedding; This represents a dense bidirectional block.

[0056] In some alternative embodiments, when dense cue representation exists... Before entering the interactive decoder, the conditional features required for tracking depth prediction are calculated, i.e., the updated first-path dense features. If the current depth prediction task is tracking depth prediction, then in the dense bidirectional block, joint inference is performed on the updated first-path dense features and the original second-path dense features to output the aligned dense representation. The formulas used include: .

[0057] In step S520 of some embodiments, when the training image belongs to a monocular scene, the dense visual embedding is a single-path dense feature, including a first-path dense feature. The first dense feature path is reused as a second dense feature path, resulting in two dense feature paths. This reduces the computational complexity of the dense bidirectional block to self-attention reasoning on a single dense feature path, thus unifying the input formats for monocular and binocular networks without altering the network structure. The two dense feature paths obtained after the reuse operation are then input into the dense bidirectional block of the interactive decoder. Joint reasoning on the two dense feature paths is performed through attention, outputting an aligned dense representation.

[0058] In step S530 of some embodiments, after obtaining the aligned dense representation, a sparse bidirectional block is introduced as the interaction core. Task markers and interaction cue representations are injected into the aligned dense representation to form task-related post-interaction dense and post-interaction sparse representations. During interaction, the sparse bidirectional block guides the aligned dense representation to focus spatially on the task-related activation region, conditioned on the task marker. Optionally, when the input is a dense cue representation, a tracking interaction path is entered in the sparse bidirectional block, and the task marker is switched to a target tracking marker. When the input is a sparse cue representation, an instance query interaction path is entered in the sparse bidirectional block. The sparse input marker formed by concatenating the instance query marker and the sparse cue representation is used to guide the aligned dense representation to focus on the target instance and output the corresponding depth.

[0059] For example, aligned dense features are used as dense input, and sparse input labels are used as sparse input. When the task type is depth prediction tracking, the aligned dense features also include conditional features. Feeding both the dense and sparse inputs into the sparse bidirectional block of the interactive decoder yields task-related post-interaction sparse and dense representations. The calculation formulas used include: ; In the formula, This represents the sparse representation after interaction. This represents a task-relevant sparse representation. The sparse representation of the interaction corresponding to the dot prompt; This represents a dense representation after interaction. For dense embedding after the first interaction, For dense embedding after the second interaction; This indicates an interactive decoder.

[0060] In step S600 of some embodiments, a matching operation is performed using the post-interaction sparse representation and the post-interaction dense representation to directly generate the depth map prediction value for the corresponding depth prediction task. Optionally, the final output head is designed as a dot-product prediction head, instead of using the traditional DPT convolutional prediction head. Using the dot-product prediction head, the depth map prediction value is obtained by performing pixel-by-pixel matching with the task-related sparse representation as the query and the task-related dense representation as the query object. By using a unified dot-product prediction head, multiple task predictions can be covered in the same output format, and redundant branches can be reduced.

[0061] In some embodiments, step S600 may include, but is not limited to, steps S610 to S630: Step S610: Obtain the task-related sparse representation from the sparse representation after interaction; Step S620: Obtain task-related dense representations based on the dense representations after interaction; Step S630: Perform matching operations on the channel dimension for task-related sparse representations and task-related dense representations to generate depth map prediction values.

[0062] In step S610 of some embodiments, from the post-interaction sparse representation Obtaining task-related sparse representations Among them, the task-related sparse representation represents the output intent of the current depth prediction task, while the post-interaction sparse representation represents dense features containing pixel-level geometric information and cross-frame alignment information.

[0063] In step S620 of some embodiments, the dense features after interaction are extracted. Dense Embedding after First-Way Interaction This serves as a dense representation relevant to the task. Alternatively, it can be densely embedded after the first interaction. Dense embedding after interaction with the second path By fusing the data, a task-relevant dense representation is obtained.

[0064] In step S630 of some embodiments, a unified dot-product prediction head is used, with task-related sparse representation as the query and task-related dense representation as the key. The task-related sparse representation and task-related dense representation are then matched by a dot-product operation along the feature channel dimension and restored to a two-dimensional grid to obtain the depth map prediction value. The calculation formula used includes: ; In the formula, This represents the predicted depth map value; This represents a task-related dense representation; This indicates that a dot product operation is performed on the feature channel dimension, and the result is mapped back to the original spatial resolution.

[0065] In some embodiments, by switching task tags and the task-related sparse representation obtained after interaction This allows for task switching. Task markers correspond to four task types: monocular depth tasks output monocular panoramic depth maps; binocular / dual-frame depth tasks output binocular panoramic depth maps; instance query tasks output depth maps corresponding to target instances under point hints; and target tracking tasks output depth maps of tracked targets under mask hints. Since the outputs of different tasks are all expressed in depth map form, a single output header can cover all of these tasks.

[0066] By adopting a unified dot-product prediction head, the model can switch tasks without designing multiple prediction heads or additional branches for different tasks, thereby reducing model complexity and inference overhead. This unified dot-product prediction head, in conjunction with the interactive decoding mechanism, enables the same image to be embedded under different task instructions to generate corresponding depth map predictions, improving deployment efficiency and system consistency.

[0067] In some embodiments, a two-stage curriculum learning strategy is employed to train the cueable depth map prediction model. The first stage focuses on the panoramic depth map prediction task, pre-training the model's geometric capabilities to enable it to learn stable scene geometric representations. The second stage introduces interactive tasks such as instance depth prediction and tracking depth prediction while retaining the geometric capabilities. Simultaneously, instance label distribution smoothing (ILDS) and Gram anchoring constraints are enabled for multi-task joint optimization to reduce task conflicts and improve final performance.

[0068] Before step S700 in some embodiments, the model undergoes a first-stage pre-training primarily for the panoramic depth prediction task, including: obtaining a basic regression loss based on the panoramic depth prediction value and the corresponding ground truth depth map value; performing a first-stage pre-training on the pre-trained cueable depth map prediction model based on the basic regression loss to obtain an initial cueable depth map prediction model; and extracting features from the preset network layers of the initial cueable depth map prediction model to obtain geometric baseline features; wherein, when the task type is panoramic depth prediction, the depth map prediction value includes the panoramic depth prediction value.

[0069] For example, in the first stage of pre-training, the basic regression loss between the panoramic depth prediction value and the ground truth depth map value is obtained. Only the basic regression loss is used to pre-train the cueable depth map prediction model, allowing the model to learn basic and robust scene geometric representations. After pre-training, an initial cueable depth map prediction model with good geometric understanding is obtained. Using this initial cueable depth map prediction model, a batch of representative data is forward-propagated, that is, geometric baseline features are extracted from selected network layers. This can be a set of patches after flattening dense feature maps. Optionally, the selected network layer is fixed, i.e., the deepest feature layer among several layers of features output from the various encoders.

[0070] In step S700 of some embodiments, interactive tasks such as instance depth prediction and tracking depth prediction are introduced while preserving geometric capabilities, and the model undergoes a second stage of joint training. The ILDS loss is used to balance the instance foreground / background label distribution, and Gram Anchoring and a phased training strategy are combined to stabilize geometric learning and jointly optimize the interactive tasks, iteratively obtaining the final target-enabled depth map prediction model.

[0071] In some embodiments, step S700 may include, but is not limited to, steps S710 to S780: Step S710: Based on the instance depth prediction value or the tracking depth prediction value, and based on the corresponding depth map true value, obtain the basic regression loss; Step S720: Construct pixel weights related to the true value of the instance mask; Step S730: Based on the instance depth prediction value or tracking depth prediction value, and based on the corresponding depth map ground truth value and pixel weight, obtain the instance label distribution smoothing loss. Step S740: Extract features from the preset network layer of the initial cueable depth map prediction model to obtain the current features; Step S750: Obtain the first gram matrix of the geometric reference feature and obtain the second gram matrix of the current feature; Step S760: Obtain the Frobenius norm between the first Gram matrix and the second Gram matrix to obtain the Gram anchoring regularization loss; Step S770: Obtain the joint loss based on the basic regression loss, instance label distribution smoothing loss, and Gram anchoring regularization loss; Step S780: Based on the joint loss, perform a second-stage joint training on the initial cueable depth map prediction model to obtain the target cueable depth map prediction model. Specifically, when the task type is instance depth prediction, the depth map prediction value includes the instance depth prediction value; when the task type is tracking depth prediction, the depth map prediction value includes the tracking depth prediction value.

[0072] In step S710 of some embodiments, based on the depth map prediction value output by the initial cueable depth map prediction model and the ground truth depth map value of the training image, a pixel-by-pixel method is used. Norm (absolute error) or Norm (squared error) form, and only in the set of valid pixels. The above describes the construction of the basic regression loss function, using the following calculation formulas: ; In the formula, Indicates the basic regression loss; This represents the predicted depth map value; Represents the ground truth of the depth map; This represents a measure of regression error. Indicates the number of valid pixels; This represents the set of valid pixels. The set of valid pixels is the effective supervision region determined according to the task type. For example, panoramic depth prediction corresponds to all valid pixels, while instance depth prediction or tracking depth prediction corresponds to the target foreground region and its background region. When the task type is panoramic depth prediction, the depth map prediction value... The depth map prediction value is the panoramic depth prediction value; when the task type is instance depth prediction, the depth map prediction value is the instance depth prediction value; when the task type is tracking depth prediction, the depth map prediction value is the tracking depth prediction value.

[0073] In step S720 of some embodiments, based on the instance mask Define pixel weights ,in This indicates that the pixel belongs to the target foreground. This indicates that the pixel belongs to the background. Adaptive weights are assigned to different pixels to balance the contributions of the foreground and background.

[0074] In step S730 of some embodiments, since the instance depth map is usually composed of "instance region depth + large area of ​​background zero / invalid values", the distribution of instance depth labels is severely unbalanced during training: the number of foreground pixels is much smaller than the number of background pixels, and the background is often represented by 0 or invalid values, which easily makes the model dominated by the background during training, resulting in problems such as instance prediction collapse or unstable foreground depth. Therefore, this embodiment of the invention introduces the instance label distribution smoothing loss ILDS, which balances the contributions of the foreground and background through pixel weights, and the following instance label distribution smoothing function can be constructed: ; In the formula, This represents the loss for smoothing the distribution of instance labels. Wherein, pixel weights... By assigning higher weights to foreground pixels and lower weights to background pixels, or by performing smooth normalization based on label distribution statistics, the problem of background dominating training can be alleviated and the quality of instance depth prediction can be improved.

[0075] In steps S740 to S760 of some embodiments, a Gram anchoring regularization term is introduced to characterize the similarity relationship between feature patches using a Gram matrix and constrain its stability during training. For example, geometrically relevant features (which can be a set of patches after flattening a dense feature map) are extracted from the preset network layer of the initial cueable depth map prediction model to obtain the current feature. This preset network layer of the initial cueable depth map prediction model is the same as the preset network layer selected in the first stage of pre-training. The first Gram matrix of the geometric baseline features obtained in the first stage of pre-training is calculated, and the second Gram matrix of the current feature is calculated. Based on the first and second Gram matrices, a Gram anchoring regularization loss function can be constructed, using the following formula: ; ; ; In the formula, Represents the first Gram matrix; Indicates geometric datum features; Represents the second Gram matrix; Indicates the current feature; This indicates Gram anchoring regularization loss; This represents the Frobenius norm.

[0076] Because panoramic geometric deep learning tends to preserve fine-grained local geometric structures, while instance / interaction tasks tend to form representations with stronger instance consistency, conflicts may arise during joint training, leading to the destruction of geometric structures or insufficient instance consistency. To mitigate this conflict, embodiments of the present invention introduce geometric structure preservation constraints, ensuring that the model does not forget its learned geometric similarities when learning instance / interaction capabilities. Optionally, a Gram-anchored regularized loss function is constructed to constrain the inter-patch similarity relationships of the current features during the second-stage training process, maintaining consistency with the inter-patch similarity relationships of the geometric baseline features obtained in the first stage, thereby reducing the destruction of existing geometric structure representations during joint training.

[0077] In step S770 of some embodiments, the total loss function can be obtained based on the basic regression loss function, the instance label distribution smoothing loss function, and the Gram anchoring regularization loss function: ; In the formula, Indicates joint loss; , This represents the weighting coefficient.

[0078] In step S780 of some embodiments, a second-stage joint training is performed on the initial cueable depth map prediction model based on the joint loss. During the training process, stochastic gradient descent or its variants (e.g., AdamW) are used to adjust the model parameters. Perform iterative updates: ; in, The learning rate; This represents the gradient of the joint loss with respect to the model parameters.

[0079] By repeatedly executing the training loop of sampling, forward inference, loss calculation, backpropagation, and parameter update until the validation set index converges or the preset number of iterations is reached, the final target-enabled depth map prediction model is obtained.

[0080] In step S800 of some embodiments, the target-enabled depth map prediction model is invoked, a single or double frame image to be processed and interactive prompts are input, and the task label corresponding to the target depth prediction task is set. The target depth map result (panoramic / instance / tracking) of the target depth prediction task is output to support embodied intelligence scene perception, target interaction and decision-making.

[0081] For example, during the model deployment or invocation phase, the task type of the current target depth prediction task is determined based on the target application, and a task tag corresponding to that target depth prediction task is set. The task type includes at least one of panoramic depth prediction, instance depth prediction, and tracking depth prediction, and a single-frame image is input according to the task type. or dual-frame image The input image is fed into a visual encoder to extract dense visual embeddings.

[0082] When performing instance depth prediction or tracking depth prediction, receive interactive prompts (dot prompts) provided by the user or upstream modules. Or mask prompt The interactive cues are encoded into interactive cue representations (sparse cue representations) using a cue encoder. Or dense indications When performing panoramic depth prediction, you can retain only the task markers without entering instance-level interactive prompts.

[0083] Embedding dense visuals Task marker Sparse cue representation A common input is used to cascade bidirectional Transformers for forward inference to complete dense-sparse feature interaction, cross-view / cross-frame information alignment, and task-related representation fusion, resulting in sparse representations and dense representations after interaction.

[0084] Based on the decoded post-interaction sparse and dense representations, a unified matching operation is performed to output the target depth map result compared to the target depth prediction task. Specifically, when the task is panoramic depth prediction, the system outputs the overall scene depth map; when the task is instance depth prediction, it outputs the instance depth map of the prompted target; and when the task is tracking depth prediction, it outputs the tracking depth map of the target in the current frame. The obtained target depth map results are output to the downstream perception or control modules of the embodied intelligence system for target localization, scene modeling, interactive operation, continuous tracking, grasping decision-making, or navigation planning.

[0085] In some embodiments, taking the generation of instance-geometric alignment data based on a synthetic environment and the training of a unified depth model as an example, the depth map prediction method for embodied intelligence with promptable interaction of the present invention includes the following steps: 1) First, a compositing environment and offline rendering pipeline for data generation are set up. Typical environments are prepared in Unreal Engine, and camera parameters and sampling strategies are configured. Camera pose sequences are generated according to preset trajectories or random paths. Dynamic targets such as pedestrians and vehicles are injected as needed to increase scene diversity. Then, MRL is started for offline rendering to obtain color image sequences. Simultaneously, supervision channels such as depth, instance segmentation, optical flow, and camera pose, strictly aligned to the image pixels, are exported. Finally, these are organized into training dataset entries in single-frame or double-frame format, providing a strongly aligned "instance-geometry" data foundation for subsequent model training.

[0086] 2) The system uses the training dataset The image is selected as the input, either a single frame or two frames. After normalization and scaling, it is fed into a shared-weight visual encoder to extract dense visual embeddings. Interactive prompts are also introduced. The input point will be suggested. Or mask prompt Sparse cue representation is obtained through the cue encoder. Or dense indications .

[0087] The system then builds task tags. This is used to indicate whether the current depth prediction task belongs to monocular depth, dual-frame depth, instance query, or target tracking. Furthermore, if the encoder output is... Then and Concatenate the sparse input tokens; if the encoder output is... Then With empty hint representation The sparse input tags are then concatenated with dense visual embeddings. These are organized together as decoder input, thereby clarifying the type of depth map that the model currently needs to output and preparing for subsequent interactive decoding.

[0088] 3) The system embeds dense visual information. Task marker Sparse cue representation The input is cascaded bidirectional Transformers for interactive decoding. Dense bidirectional blocks are first processed... Cross-view joint inference yields aligned dense representations; in the monocular case, features from the same frame can be reused to maintain structural consistency.

[0089] Subsequently, the sparse bidirectional block realizes sparse-dense bidirectional interaction under the conditions of task labeling and sparse cue representation, so that the dense features focus on the region related to the current depth prediction task, and outputs the task-related sparse representation and dense representation after interaction, which can be directly used for subsequent depth map generation.

[0090] 4) The system uses a unified dot-product prediction head to generate depth maps. Specifically, it employs task-relevant sparse representations. As a query, it is characterized by task-related dense representation. As the queried feature map, pixel-by-pixel matching is performed to obtain the depth map prediction value. By switching task markers and combining them with point hints or mask hints, the system can output panoramic depth, instance depth, or tracking depth under the same output head, achieving a unified output interface without the need for additional multi-head branches.

[0091] 5) During training, the system calculates the basic depth-supervised loss and introduces an imbalanced instance label distribution smoothing loss for instance / tracking tasks to stabilize foreground learning. Simultaneously, regularization constraints are added to maintain the stability of geometric representations, reducing interference from instance representations on geometric details. The training strategy can be phased, first learning panoramic geometric capabilities, then incorporating instance query and tracking tasks for joint optimization, until the metrics converge or the stopping condition is met, resulting in the final target-indicating depth map prediction model.

[0092] 6) After model training is complete, the system enters the online inference and application phase. For the scene to be processed, a single-frame or dual-frame image is input, and the corresponding task marker is selected according to the task requirements. When performing instance depth or tracking depth prediction, point cues or mask cues can be further input as interaction conditions. The system then uses the trained unified depth prediction model for forward computation, outputting the target depth map result corresponding to the target depth prediction task: outputting the overall scene depth for panoramic tasks, the instance depth of a specified target for instance tasks, and the tracking depth of the target in the current frame for tracking tasks. The obtained target depth map result can be further provided to downstream modules of the embodied intelligence system for scene perception, target localization, interactive operations, and decision control.

[0093] Table 1 shows the improvement effect of the depth map prediction method with promptable interaction for embodied intelligence on a certain dataset.

[0094] Table 1

[0095] Experimental results demonstrate that the depth map prediction method for embodied intelligence with prompting interaction provided in this invention has significant advantages and practical value. Experimental data shows that compared to existing combined "segmentation model + depth model" schemes, the method of this invention exhibits superior performance in terms of interactive instance perception accuracy and end-to-end inference efficiency. Firstly, in terms of interactive instance perception, this invention achieves higher mIoU and F-measure metrics in point-cue interactive instance segmentation tasks, indicating that the unified modeling approach of task labeling and prompt encoding can more accurately focus on target instances and output stable instance-related prediction results. Secondly, in terms of inference efficiency, this invention employs a single feedforward network and a unified dot product prediction head, avoiding redundant computation caused by multi-module cascading. The end-to-end latency is significantly lower than that of combined schemes, making it more suitable for the real-time requirements of embodied intelligence systems. Overall, this invention, through its "unified model, unified output" design, improves interactive instance capabilities while reducing system redundancy and deployment costs, thus demonstrating good generalization ability in practical application scenarios such as robot navigation, grasping, and dynamic target tracking.

[0096] This invention also provides a depth map prediction device with promptable interaction for embodied intelligence, which can implement the above-mentioned depth map prediction method with promptable interaction for embodied intelligence. The device includes: The data construction module is used to obtain the training images, as well as the ground truth depth maps and instance mask ground truth values ​​of the training images, and input the training images into the initial cueable depth map prediction model; the initial cueable depth map prediction model includes a visual encoder, a cue encoder, and an interactive decoder; The visual encoding module is used to visually encode the training images using a visual encoder to obtain dense visual embeddings. The prompt encoding module is used to encode interactive prompts using a prompt encoder to obtain an interactive prompt representation. The task tagging setting module is used to predict the task type based on the current depth and construct task tags; The interaction decoding module is used to generate post-interaction sparse representations and post-interaction dense representations based on dense visual embeddings, interaction cue representations, and task tags using a pre-built interaction decoder. The first depth map prediction module is used to perform matching operations on the sparse representation and the dense representation after interaction to generate the depth map prediction value for the current depth prediction task. The loss constraint and iterative training module is used to update the initial cueable depth map prediction model based on the depth map prediction value, the depth map ground truth value, and the instance mask ground truth value, so as to obtain the target cueable depth map prediction model. The second depth map prediction module is used to input the image to be processed, interactive prompts, and the task label corresponding to the target depth prediction task into the target prompt depth map prediction model to obtain the target depth map result of the target depth prediction task; the target depth map result is used to realize the control of the embodied intelligence system.

[0097] It is understood that the content of the above method embodiments is applicable to the present device embodiments. The specific functions implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0098] This invention also provides an electronic device, which includes a processor and a memory. The memory stores a computer program, and the processor executes the computer program to implement the above-described method. This electronic device can be any smart terminal, including a tablet computer, an in-vehicle computer, or similar device.

[0099] It is understood that the content of the above method embodiments is applicable to this device embodiment. The specific functions implemented by this device embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0100] refer to Figure 2 , Figure 2 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes: The processor 901 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present invention. The memory 902 can be implemented as a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 902 can store the operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 902 and is called and executed by the processor 901. The 903 input / output interface is used to implement information input and output. The communication interface 904 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.). Bus 905 transmits information between various components of the device (e.g., processor 901, memory 902, input / output interface 903, and communication interface 904); The processor 901, memory 902, input / output interface 903, and communication interface 904 are connected to each other within the device via bus 905.

[0101] This invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method.

[0102] It is understood that the content of the above method embodiments is applicable to this storage medium embodiment. The specific functions implemented in this storage medium embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0103] This invention also provides a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium and execute the computer instructions to cause the computer device to perform the aforementioned method.

[0104] In summary, in practical applications, the depth map prediction method and apparatus for embodied intelligence with prompting interaction according to embodiments of the present invention firstly, in the data construction stage, utilizes a synthetic data engine based on Unreal Engine and an offline rendering pipeline to generate large-scale training data, simultaneously exporting supervision signals such as depth, instance segmentation, optical flow, and camera pose that are strictly aligned with the rendered images at the pixel level, and organizing them into a training sample set that supports single-frame, dual-frame, and interactive tasks. Subsequently, in the input and prompting encoding stage, the system selects single-frame or dual-frame corresponding images from the dataset as input, extracts dense visual embeddings through a shared-weight visual encoder, and simultaneously introduces prompting information, encoding point prompts and mask prompts into sparse prompting representations and dense prompting representations, respectively, and constructs task markers covering monocular depth, dual-frame depth, instance query depth, and target tracking depth to indicate the current output target. Subsequently, in the interactive decoding stage, dense visual embeddings are fed into a cascaded bidirectional Transformer for processing: dense bidirectional blocks are used to complete potential geometric alignment and joint inference across frames / views, while sparse bidirectional blocks are used to achieve sparse-dense bidirectional interaction under task labeling and interactive cue representations, thus obtaining task-related post-interaction dense and post-interaction sparse representations. Finally, in the unified output stage, task-related sparse representations are used as queries to perform dot product matching on task-related dense representations to generate depth map predictions. By switching task labels and combining dot cues or mask cues, panoramic depth, instance depth, or tracking depth results can be obtained under the same model and the same output header. In the training stage, the system can combine a loss term for uneven distribution of instance depth labels with regularization constraints for stable geometric representations, and adopt a phased training strategy to iteratively optimize model parameters to obtain a unified model that combines geometric quality and interactive capabilities.

[0105] In the depth map prediction method and apparatus for embodied intelligence with prompting interaction of the present invention, the design and implementation of each step have the following advantages: 1. The instance-geometric data construction step alleviates the problem of scarce strong supervision of real-world geometry-instance by synchronously generating multimodal, strongly aligned supervision signals in a synthetic environment, providing a high-quality data foundation for unified learning.

[0106] 2. The input and prompting encoding steps, through the collaborative design of a shared visual encoder, prompting encoding, and task labeling, clarify the output intent of embedding the same source image under different tasks, thereby reducing output uncertainty and supporting interactive use.

[0107] 3. The cascaded bidirectional Transformer interactive decoding steps use a "dense alignment first, sparse injection later" structure to unify cross-frame / cross-view geometric alignment and prompt condition interaction within the same decoding framework, balancing geometric consistency and task controllability.

[0108] 4. The unified depth map output process replaces multi-head convolutional prediction with dot-product prediction heads, avoiding multi-branch redundancy and improving inference efficiency, enabling the use of a single output interface to cover multiple tasks.

[0109] 5. Loss Constraints and Iterative Training Steps: By addressing the imbalance in instance depth label distribution, stabilizing geometric representations, and employing a curriculum-based training strategy, the training stability and final generalization performance are further improved.

[0110] In some alternative embodiments, the functions / operations mentioned in the block diagrams may not occur in the order shown in the operation diagrams. For example, depending on the functions / operations involved, two consecutively shown blocks may actually be executed substantially simultaneously, or the blocks may sometimes be executed in reverse order. Furthermore, the embodiments presented and described in the flowcharts of this invention are provided by way of example to provide a more comprehensive understanding of the technology. The disclosed methods are not limited to the operations and logic flows presented herein. Alternative embodiments are contemplated in which the order of various operations is altered and sub-operations described as part of a larger operation are executed independently.

[0111] Furthermore, although the invention has been described in the context of functional modules, it should be understood that, unless otherwise stated, one or more of the described functions and / or features may be integrated into a single physical device and / or software module, or one or more functions and / or features may be implemented in a separate physical device or software module. It is also understood that a detailed discussion of the actual implementation of each module is unnecessary for understanding the invention. Rather, given the properties, functions, and internal relationships of the various functional modules in the apparatus disclosed herein, the actual implementation of the module will be understood within the scope of conventional skill of an engineer. Therefore, those skilled in the art can implement the invention as set forth in the claims using ordinary techniques without excessive experimentation. It is also understood that the specific concepts disclosed are merely illustrative and not intended to limit the scope of the invention, which is determined by the full scope of the appended claims and their equivalents.

[0112] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, essentially, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0113] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-including system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0114] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0115] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0116] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.

[0117] The above is a detailed description of the preferred embodiments of the present invention. However, the present invention is not limited to the embodiments described. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention. All such equivalent modifications or substitutions are included within the scope defined by the claims of the present invention.

Claims

1. A depth map prediction method with promptable interaction for embodied intelligence, characterized in that, Includes the following steps: The process of acquiring training images, their ground truth depth maps, and instance mask ground truth values, and inputting the training images into an initial cueable depth map prediction model, includes the following steps: generating a synthetic environment and camera trajectory using a preset data engine; rendering the synthetic environment and camera trajectory using an offline rendering pipeline to output a color image, and simultaneously exporting multi-channel ground truth values ​​pixel-aligned with the color image; the multi-channel ground truth values ​​include the depth map ground truth value, the instance mask ground truth value, the optical flow ground truth value, and the camera pose; constructing a training dataset based on the color image and the multi-channel ground truth values; selecting a color image from the training dataset as the training image; acquiring the depth map ground truth value and instance mask ground truth value corresponding to the training image, and inputting the training image into the initial cueable depth map prediction model; the initial cueable depth map prediction model includes a visual encoder, a cue encoder, and an interaction decoder. The acquisition of training images includes the following steps: selecting a frame of the color image from the training dataset to obtain the training image belonging to a monocular scene; or, sampling the color images in the same sequence from the training dataset at random intervals to obtain the training image belonging to a binocular scene; wherein the random interval is a sampling interval randomly selected within a preset interval range. The training images are visually encoded using the visual encoder to obtain dense visual embeddings. The interactive prompt is encoded using the prompt encoder to obtain an interactive prompt representation, including the following steps: when the interactive prompt is a masked prompt, the masked prompt is input into the prompt encoder to obtain a dense prompt representation; when the interactive prompt is a dotted prompt, the dotted prompt is input into the prompt encoder to obtain a sparse prompt representation. Based on the task type of the current depth prediction task, construct task tags; Using the pre-built interaction decoder, post-interaction sparse representations and post-interaction dense representations are generated based on the dense visual embedding, the interaction cue representation, and the task tag. A matching operation is performed on the sparse representation after the interaction and the dense representation after the interaction to generate the depth map prediction value of the current depth prediction task. Based on the depth map prediction value, the depth map ground value, and the instance mask ground value, the initial cueable depth map prediction model is updated to obtain the target cueable depth map prediction model. The image to be processed, the interactive prompts, and the task tags corresponding to the target depth prediction task are input into the target promptable depth map prediction model to obtain the target depth map result of the target depth prediction task; the target depth map result is used to realize the control of the embodied intelligence system.

2. The method according to claim 1, characterized in that, Before generating post-interaction sparse and post-interaction dense representations based on the dense visual embedding, the interaction cue representation, and the task tag using the pre-built interaction decoder, the method further includes the following steps: When the interactive cue representation is a dense cue representation, the dense cue representation is injected into the first dense feature of the dense visual embedding to obtain the updated first dense feature, and the task label is switched to the target tracking label. The task label is then concatenated with the empty cue representation to obtain the sparse input label. When the interactive prompt representation is a sparse prompt representation, the task tag is switched to an instance query tag, and the task tag is concatenated with the sparse prompt representation to obtain a sparse input tag.

3. The method according to claim 2, characterized in that, The step of generating post-interaction sparse and post-interaction dense representations using the pre-built interaction decoder based on the dense visual embedding, the interaction cue representation, and the task tagging includes the following steps: When the training image belongs to a binocular scene, the dense visual embedding is a two-way dense feature. The two-way dense features are input into the dense bidirectional block of the interactive decoder. The two-way dense features are jointly inferred through attention to output an aligned dense representation. The two-way dense features include the first-way dense feature and the second-way dense feature. When the training image belongs to a monocular scene, the dense visual embedding is a single-path dense feature. The single-path dense feature is reused as a second-path dense feature to obtain two-path dense features. The two-path dense features are input into the dense bidirectional block of the interactive decoder. The two-path dense features are jointly inferred through attention to output an aligned dense representation. The single-path dense feature includes the first-path dense feature. The sparse input tag and the aligned dense representation are interactively decoded using the sparse bidirectional block of the interactive decoder to obtain the interactive sparse representation and the interactive dense representation.

4. The method according to claim 1, characterized in that, The step of performing a matching operation between the sparse representation and the dense representation after the interaction to generate the depth map prediction value for the current depth prediction task includes the following steps: Obtain the task-related sparse representation from the sparse representation after the interaction; Based on the dense representation after the interaction, obtain the task-related dense representation; The task-related sparse representation and the task-related dense representation are matched along the channel dimension to generate the depth map prediction value.

5. The method according to claim 1, characterized in that, Before updating the initial clueable depth map prediction model based on the depth map prediction value, the depth map ground truth value, and the instance mask ground truth value to obtain the target clueable depth map prediction model, the method further includes the following steps: Based on the panoramic depth prediction value and the corresponding depth map ground truth value, the basic regression loss is obtained; Based on the basic regression loss, the pre-trained cueable depth map prediction model is pre-trained in the first stage to obtain the initial cueable depth map prediction model. Feature extraction is performed on the preset network layer of the initial suggestible depth map prediction model to obtain geometric reference features; Wherein, when the task type is panoramic depth prediction, the depth map prediction value includes the panoramic depth prediction value.

6. The method according to claim 5, characterized in that, The step of updating the initial clueable depth map prediction model based on the depth map prediction value, the ground truth depth map value, and the instance mask ground truth value to obtain the target clueable depth map prediction model includes the following steps: Based on the instance depth prediction value or the tracking depth prediction value, and based on the corresponding true value of the depth map, obtain the basic regression loss; Construct pixel weights associated with the truth value of the instance mask; Based on the instance depth prediction value or the tracking depth prediction value, and based on the corresponding depth map ground truth value and the pixel weight, obtain the instance label distribution smoothing loss; Feature extraction is performed on the preset network layer of the initial suggestible depth map prediction model to obtain the current features; Obtain the first gram matrix of the geometric reference feature, and obtain the second gram matrix of the current feature; Obtain the Frobenius norm between the first Gram matrix and the second Gram matrix to obtain the Gram anchoring regularization loss; The joint loss is obtained based on the basic regression loss, the instance label distribution smoothing loss, and the Gram anchoring regularization loss; Based on the joint loss, the initial cueable depth map prediction model is jointly trained in the second stage to obtain the target cueable depth map prediction model. Wherein, when the task type is instance depth prediction, the depth map prediction value includes the instance depth prediction value; when the task type is tracking depth prediction, the depth map prediction value includes the tracking depth prediction value.

7. A depth map prediction device with promptable interaction for embodied intelligence, used to implement the method as described in any one of claims 1 to 6, characterized in that, include: The data construction module is used to acquire training images and ground truth values ​​of depth maps and instance masks of the training images, and input the training images into an initial cueable depth map prediction model; the initial cueable depth map prediction model includes a visual encoder, a cue encoder, and an interactive decoder; The visual encoding module is used to perform visual encoding on the training image through the visual encoder to obtain dense visual embedding; The prompt encoding module is used to encode the interactive prompts through the prompt encoder to obtain an interactive prompt representation; The task tagging setting module is used to predict the task type based on the current depth and construct task tags; An interaction decoding module is used to generate a sparse representation and a dense representation after interaction based on the dense visual embedding, the interaction cue representation, and the task tag, using the pre-built interaction decoder. The first depth map prediction module is used to perform matching operations on the sparse representation after interaction and the dense representation after interaction to generate the depth map prediction value of the current depth prediction task. The loss constraint and iterative training module is used to update the initial cueable depth map prediction model based on the depth map prediction value, the depth map ground truth value, and the instance mask ground truth value, so as to obtain the target cueable depth map prediction model. The second depth map prediction module is used to input the image to be processed, the interactive prompts, and the task markers corresponding to the target depth prediction task into the target promptable depth map prediction model to obtain the target depth map result of the target depth prediction task; the target depth map result is used to realize the control of the embodied intelligence system.

Images